Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization
This addresses a critical issue for practitioners using DCCA-based methods in multi-view representation learning, preventing performance degradation and enabling wider adoption, though it is incremental as it builds on existing DCCA frameworks.
The paper tackles the problem of model collapse in Deep Canonical Correlation Analysis (DCCA) methods, where performance drops drastically during training, and proposes NR-DCCA with noise regularization to prevent this, achieving stable and consistent outperformance over baselines on synthetic and real-world datasets.
Multi-View Representation Learning (MVRL) aims to learn a unified representation of an object from multi-view data. Deep Canonical Correlation Analysis (DCCA) and its variants share simple formulations and demonstrate state-of-the-art performance. However, with extensive experiments, we observe the issue of model collapse, {\em i.e.}, the performance of DCCA-based methods will drop drastically when training proceeds. The model collapse issue could significantly hinder the wide adoption of DCCA-based methods because it is challenging to decide when to early stop. To this end, we develop NR-DCCA, which is equipped with a novel noise regularization approach to prevent model collapse. Theoretical analysis shows that the Correlation Invariant Property is the key to preventing model collapse, and our noise regularization forces the neural network to possess such a property. A framework to construct synthetic data with different common and complementary information is also developed to compare MVRL methods comprehensively. The developed NR-DCCA outperforms baselines stably and consistently in both synthetic and real-world datasets, and the proposed noise regularization approach can also be generalized to other DCCA-based methods such as DGCCA.