Effective Combination of Language and Vision Through Model Composition and the R-CCA Method
This work addresses the challenge of multimodal representation learning for researchers in natural language processing and computer vision, though it appears incremental as it builds on existing methods like CCA.
The paper tackles the problem of integrating textual and visual information in vector space models for word meaning representation by proposing the Residual CCA (R-CCA) method and sequential modeling with composition of techniques like PCA and CCA, resulting in models that outperform recent multimodal alternatives on five standard semantic benchmarks.
We address the problem of integrating textual and visual information in vector space models for word meaning representation. We first present the Residual CCA (R-CCA) method, that complements the standard CCA method by representing, for each modality, the difference between the original signal and the signal projected to the shared, max correlation, space. We then show that constructing visual and textual representations and then post-processing them through composition of common modeling motifs such as PCA, CCA, R-CCA and linear interpolation (a.k.a sequential modeling) yields high quality models. On five standard semantic benchmarks our sequential models outperform recent multimodal representation learning alternatives, including ones that rely on joint representation learning. For two of these benchmarks our R-CCA method is part of the Best configuration our algorithm yields.