Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Nets
This work addresses the problem of understanding optimization dynamics for neural network pruning, offering incremental theoretical insights into weight normalization methods.
The paper analyzes the inductive bias of gradient descent for weight-normalized smooth homogeneous neural networks, showing that exponential weight normalization (EWN) leads to asymptotic relative sparsity in weights and provides finite-time convergence rates for loss, with experimental validation on synthetic data.
We analyze the inductive bias of gradient descent for weight normalized smooth homogeneous neural nets, when trained on exponential or cross-entropy loss. We analyse both standard weight normalization (SWN) and exponential weight normalization (EWN), and show that the gradient flow path with EWN is equivalent to gradient flow on standard networks with an adaptive learning rate. We extend these results to gradient descent, and establish asymptotic relations between weights and gradients for both SWN and EWN. We also show that EWN causes weights to be updated in a way that prefers asymptotic relative sparsity. For EWN, we provide a finite-time convergence rate of the loss with gradient flow and a tight asymptotic convergence rate with gradient descent. We demonstrate our results for SWN and EWN on synthetic data sets. Experimental results on simple datasets support our claim on sparse EWN solutions, even with SGD. This demonstrates its potential applications in learning neural networks amenable to pruning.