Improving Multimodal Accuracy Through Modality Pre-training and Attention
This work addresses the problem of inefficient multimodal training for researchers and practitioners by offering a method to simplify architectures and reduce training costs, though it is incremental as it builds on existing pre-training and attention techniques.
The paper tackled the challenge of training multimodal networks by addressing differing convergence rates among modalities through independent pre-training of modality-specific sub-networks and adding an attention mechanism to prioritize important modalities in ambiguous scenarios, resulting in a simple network achieving similar performance to more complex architectures on tasks like sentiment analysis, emotion recognition, and speaker trait recognition.
Training a multimodal network is challenging and it requires complex architectures to achieve reasonable performance. We show that one reason for this phenomena is the difference between the convergence rate of various modalities. We address this by pre-training modality-specific sub-networks in multimodal architectures independently before end-to-end training of the entire network. Furthermore, we show that the addition of an attention mechanism between sub-networks after pre-training helps identify the most important modality during ambiguous scenarios boosting the performance. We demonstrate that by performing these two tricks a simple network can achieve similar performance to a complicated architecture that is significantly more expensive to train on multiple tasks including sentiment analysis, emotion recognition, and speaker trait recognition.