TAOTF: A Two-stage Approximately Orthogonal Training Framework in Deep Neural Networks
This work addresses robustness issues in noisy data scenarios for deep learning models, particularly CNNs and ViTs, but appears incremental as it builds on existing orthogonal constraint methods.
The authors tackled the problem of poor robustness to noisy data in deep neural networks with orthogonal constraints by proposing a two-stage approximately orthogonal training framework, which achieved stable and superior performance compared to existing methods on natural and medical image datasets.
The orthogonality constraints, including the hard and soft ones, have been used to normalize the weight matrices of Deep Neural Network (DNN) models, especially the Convolutional Neural Network (CNN) and Vision Transformer (ViT), to reduce model parameter redundancy and improve training stability. However, the robustness to noisy data of these models with constraints is not always satisfactory. In this work, we propose a novel two-stage approximately orthogonal training framework (TAOTF) to find a trade-off between the orthogonal solution space and the main task solution space to solve this problem in noisy data scenarios. In the first stage, we propose a novel algorithm called polar decomposition-based orthogonal initialization (PDOI) to find a good initialization for the orthogonal optimization. In the second stage, unlike other existing methods, we apply soft orthogonal constraints for all layers of DNN model. We evaluate the proposed model-agnostic framework both on the natural image and medical image datasets, which show that our method achieves stable and superior performances to existing methods.