Robust low-rank training via approximate orthonormal constraints
This addresses the robustness issue in resource-efficient deep learning for practitioners using low-rank methods, though it is incremental as it builds on existing low-rank techniques.
The paper tackles the problem of low-rank training methods compromising adversarial robustness in deep learning by introducing a robust low-rank training algorithm that enforces approximate orthonormal constraints to ensure well-conditioning, resulting in reduced training and inference costs while maintaining accuracy and improving robustness, as shown through extensive numerical evidence.
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.