Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AI
This addresses the challenge of deploying multimodal LLMs on edge devices for embodied AI by reducing computational costs while maintaining or improving performance, though it is incremental in optimizing existing methods.
The paper tackles the problem of efficiently adapting multimodal LLMs to diverse input modalities in resource-constrained embodied AI by proposing mPnP-LLM, which enables elastic modality adaptation at runtime, achieving up to 3.7x FLOPs reduction and 30% GPU memory reduction while maintaining accuracy, and improving task accuracy by up to 4% under the same compute budget.
Large Language Models (LLMs) are capable of reasoning over diverse input data modalities through pre-trained encoders. However, the growing diversity of input data modalities prevents incorporating all modalities into LLMs, especially when LLMs are deployed on resource-constrained edge devices for embodied AI applications. Instead, a better option is to adaptively involve only the useful modalities at runtime, depending on the current environmental contexts and task requirements. For such modality adaptation, existing work adopts fixed connections between encoders and the LLM's input layer, leading to high training cost at runtime and ineffective cross-modal interaction. In this paper, we address these limitations by presenting mPnP-LLM, a new technique that allows fully elastic, automated and prompt runtime modality adaptation, by connecting unimodal encoders to a flexible set of last LLM blocks and making such latent connections fully trainable at runtime. Experiments over the nuScenes-QA dataset show that mPnP-LLM can achieve up to 3.7x FLOPs reduction and 30% GPU memory usage reduction, while retaining on-par accuracy with the existing schemes. Under the same compute budget, mPnP-LLM improves the task accuracy by up to 4% compared to the best existing scheme.