Integrating Multimodal Information in Large Pretrained Transformers
This addresses the challenge of adapting language models to multimodal contexts in NLP, enabling improved sentiment analysis in face-to-face communication, though it is incremental as it builds on existing Transformer architectures.
The paper tackled the problem of fine-tuning large pre-trained Transformers like BERT and XLNet for multimodal language tasks by proposing a Multimodal Adaptation Gate (MAG) that integrates visual and acoustic data, resulting in MAG-XLNet achieving human-level performance on the CMU-MOSI dataset for sentiment analysis.
Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straightforward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.