Cross-model convolutional neural network for multiple modality data representation
This addresses the challenge of integrating multiple data types for machine learning applications, but it appears incremental as it builds on existing CNN and regularization techniques.
The paper tackles the problem of representing data from different modalities in a common space using a novel CNN-based method, achieving improved performance on benchmark datasets.
A novel data representation method of convolutional neural net- work (CNN) is proposed in this paper to represent data of different modalities. We learn a CNN model for the data of each modality to map the data of differ- ent modalities to a common space, and regularize the new representations in the common space by a cross-model relevance matrix. We further impose that the class label of data points can also be predicted from the CNN representa- tions in the common space. The learning problem is modeled as a minimiza- tion problem, which is solved by an augmented Lagrange method (ALM) with updating rules of Alternating direction method of multipliers (ADMM). The experiments over benchmark of sequence data of multiple modalities show its advantage.