Improving Unimodal Inference with Multimodal Transformers
This work addresses the challenge of boosting unimodal inference for researchers and practitioners in multimodal AI, though it appears incremental as it builds on existing multimodal transformer methods.
The paper tackles the problem of enhancing unimodal model performance by using multimodal training, achieving improved results over conventionally trained unimodal models across tasks like hand gesture recognition, emotion recognition, and sentiment analysis.
This paper proposes an approach for improving performance of unimodal models with multimodal training. Our approach involves a multi-branch architecture that incorporates unimodal models with a multimodal transformer-based branch. By co-training these branches, the stronger multimodal branch can transfer its knowledge to the weaker unimodal branches through a multi-task objective, thereby improving the performance of the resulting unimodal models. We evaluate our approach on tasks of dynamic hand gesture recognition based on RGB and Depth, audiovisual emotion recognition based on speech and facial video, and audio-video-text based sentiment analysis. Our approach outperforms the conventionally trained unimodal counterparts. Interestingly, we also observe that optimization of the unimodal branches improves the multimodal branch, compared to a similar multimodal model trained from scratch.