Scalable and Effective Deep CCA via Soft Decorrelation
This work addresses a bottleneck in multi-view learning for researchers and practitioners by improving deep CCA scalability and performance, though it is incremental as it builds on existing deep CCA frameworks.
The paper tackles the computational inefficiency and sub-optimal solutions in deep CCA models by replacing exact decorrelation with a soft decorrelation loss, resulting in a more effective and efficient method that outperforms existing models in experiments.
Recently the widely used multi-view learning model, Canonical Correlation Analysis (CCA) has been generalised to the non-linear setting via deep neural networks. Existing deep CCA models typically first decorrelate the feature dimensions of each view before the different views are maximally correlated in a common latent space. This feature decorrelation is achieved by enforcing an exact decorrelation constraint; these models are thus computationally expensive due to the matrix inversion or SVD operations required for exact decorrelation at each training iteration. Furthermore, the decorrelation step is often separated from the gradient descent based optimisation, resulting in sub-optimal solutions. We propose a novel deep CCA model Soft CCA to overcome these problems. Specifically, exact decorrelation is replaced by soft decorrelation via a mini-batch based Stochastic Decorrelation Loss (SDL) to be optimised jointly with the other training objectives. Extensive experiments show that the proposed soft CCA is more effective and efficient than existing deep CCA models. In addition, our SDL loss can be applied to other deep models beyond multi-view learning, and obtains superior performance compared to existing decorrelation losses.