Sharing our latest work presented at the ICML workshop “Identifying and Understanding Deep Learning Phenomena”:

Layer rotation: a surprisingly powerful indicator of generalization in deep networks? (arxiv link)

​

We’re pretty excited about it: we really believe layer rotation (the metric we study) is somehow related to a fundamental aspect of deep learning, and that it is worth much more investigation. For the moment, our work demonstrates that layer rotation’s relation with generalization exhibits a remarkable

• consistency : a rule of thumb that is widely applicable, explaining differences of up to 30% test accuracy,
• simplicity : a network-independent optimum w.r.t. generalization, and
• explanatory power: novel insights around widely used techniques (weight decay, adaptive gradient methods, learning rate warmups,…).

We also provide preliminary evidence that layer rotations correlate with the degree to which intermediate features are learned during the training procedure.

​

Since we also provide tools to monitor and control layer rotation during training, our work could also greatly reduce the current hyperparameter tuning struggle. Code available! Here and here.

​

Looking forward to your feedback!

​

Abstract:

Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer’s weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final and initial weights of each layer consistently translate into better generalization performance of the final model. Interestingly, this relation admits a network independent optimum: training procedures during which all layers’ weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 30% test accuracy. Moreover, we show that layer rotations are easily monitored and controlled (helpful for hyperparameter tuning) and potentially provide a unified framework to explain the impact of learning rate tuning, weight decay, learning rate warmups and adaptive gradient methods on generalization and training speed. In an attempt to explain the surprising properties of layer rotation, we show on a 1-layer MLP trained on MNIST that layer rotation correlates with the degree to which features of intermediate layers have been trained.