CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets
This addresses the challenge of adapting large pretrained models to new multimodal tasks with limited data and resources, though it appears incremental as it builds on existing adapter techniques.
The paper tackles the problem of efficient multimodal learning with inhomogeneous datasets by introducing CM3T, a model-agnostic plugin architecture that adapts transformer-based models using adapter blocks. It achieves comparable or better results than state-of-the-art methods while using only 12.8% trainable parameters for video input and 22.3% for two additional modalities.
Challenges in cross-learning involve inhomogeneous or even inadequate amount of training data and lack of resources for retraining large pretrained models. Inspired by transfer learning techniques in NLP, adapters and prefix tuning, this paper presents a new model-agnostic plugin architecture for cross-learning, called CM3T, that adapts transformer-based models to new or missing information. We introduce two adapter blocks: multi-head vision adapters for transfer learning and cross-attention adapters for multimodal learning. Training becomes substantially efficient as the backbone and other plugins do not need to be finetuned along with these additions. Comparative and ablation studies on three datasets Epic-Kitchens-100, MPIIGroupInteraction and UDIVA v0.5 show efficacy of this framework on different recording settings and tasks. With only 12.8% trainable parameters compared to the backbone to process video input and only 22.3% trainable parameters for two additional modalities, we achieve comparable and even better results than the state-of-the-art. CM3T has no specific requirements for training or pretraining and is a step towards bridging the gap between a general model and specific practical applications of video classification.