Counterbalancing Teacher: Regularizing Batch Normalized Models for Robustness
This addresses robustness issues in deep learning for practitioners, though it is incremental as it builds on existing regularization and teacher-student methods.
The paper tackles the problem that batch normalization (BN) incentivizes models to rely on low-variance features specific to training data, hurting out-of-domain generalization. It proposes Counterbalancing Teacher (CT), a method using a frozen model without BN as a teacher to enforce robust representations, outperforming baselines on benchmarks like CIFAR-10-C and CIFAR-100-C.
Batch normalization (BN) is a ubiquitous technique for training deep neural networks that accelerates their convergence to reach higher accuracy. However, we demonstrate that BN comes with a fundamental drawback: it incentivizes the model to rely on low-variance features that are highly specific to the training (in-domain) data, hurting generalization performance on out-of-domain examples. In this work, we investigate this phenomenon by first showing that removing BN layers across a wide range of architectures leads to lower out-of-domain and corruption errors at the cost of higher in-domain errors. We then propose Counterbalancing Teacher (CT), a method which leverages a frozen copy of the same model without BN as a teacher to enforce the student network's learning of robust representations by substantially adapting its weights through a consistency loss function. This regularization signal helps CT perform well in unforeseen data shifts, even without information from the target domain as in prior works. We theoretically show in an overparameterized linear regression setting why normalization leads to a model's reliance on such in-domain features, and empirically demonstrate the efficacy of CT by outperforming several baselines on robustness benchmarks such as CIFAR-10-C, CIFAR-100-C, and VLCS.