The Expressive Power of Tuning Only the Normalization Layers
This work addresses a theoretical problem for researchers in deep learning, providing foundational insights into the capabilities of normalization layer tuning, though it is incremental as it builds on prior empirical findings.
The paper investigates the expressive power of tuning only normalization layers in frozen neural networks, showing that for random ReLU networks, this approach can reconstruct any target network up to a size scaling factor of O(sqrt(width)), even with sparsification under overparameterization.
Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks. Recent studies on fine-tuning large pretrained models indicate that just tuning the parameters of these affine transforms can achieve high accuracy for downstream tasks. These findings open the questions about the expressive power of tuning the normalization layers of frozen networks. In this work, we take the first step towards this question and show that for random ReLU networks, fine-tuning only its normalization layers can reconstruct any target network that is $O(\sqrt{\text{width}})$ times smaller. We show that this holds even for randomly sparsified networks, under sufficient overparameterization, in agreement with prior empirical work.