On the Impact of Stable Ranks in Deep Nets
This work addresses the role of stable ranks in DNN dynamics and training, offering insights for improving network efficiency and performance, though it is incremental in building on prior connections.
The paper investigates how stable ranks of layer weights affect deep neural networks, showing that stable ranks act as linear factors with exponential depthwise accumulation and that stable rank initialization can improve convergence speed.
A recent line of work has established intriguing connections between the generalization/compression properties of a deep neural network (DNN) model and the so-called layer weights' stable ranks. Intuitively, the latter are indicators of the effective number of parameters in the net. In this work, we address some natural questions regarding the space of DNNs conditioned on the layers' stable rank, where we study feed-forward dynamics, initialization, training and expressivity. To this end, we first propose a random DNN model with a new sampling scheme based on stable rank. Then, we show how feed-forward maps are affected by the constraint and how training evolves in the overparametrized regime (via Neural Tangent Kernels). Our results imply that stable ranks appear layerwise essentially as linear factors whose effect accumulates exponentially depthwise. Moreover, we provide empirical analysis suggesting that stable rank initialization alone can lead to convergence speed ups.