Group Orthogonalization Regularization For Vision Models Adaptation and Robustness
This incremental method addresses efficiency and robustness issues for vision models, benefiting researchers and practitioners in computer vision.
The paper tackles parameter redundancy in deep neural networks by proposing a group orthogonality regularization technique, which improves performance on downstream tasks for diffusion models and vision transformers and enhances robustness during adversarial training.
As neural networks become deeper, the redundancy within their parameters increases. This phenomenon has led to several methods that attempt to reduce the correlation between convolutional filters. We propose a computationally efficient regularization technique that encourages orthonormality between groups of filters within the same layer. Our experiments show that when incorporated into recent adaptation methods for diffusion models and vision transformers (ViTs), this regularization improves performance on downstream tasks. We further show improved robustness when group orthogonality is enforced during adversarial training. Our code is available at https://github.com/YoavKurtz/GOR.