MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learning
This work addresses parameter and memory efficiency for audio-visual learning tasks, representing an incremental improvement over existing adaptation methods.
The paper tackled the problem of inefficient adaptation in audio-visual learning by proposing MoLT, a framework that replaces heavy sequential adaptation with a parallel, lightweight scheme using layer-wise tokens from late transformer layers, resulting in outperforming existing methods on benchmarks like Audio-Visual Question Answering, Audio-Visual Segmentation, and Audio-Visual Event Localization.
In this paper, we propose Mixture of Layer-Wise Tokens (MoLT), a parameter- and memory-efficient adaptation framework for audio-visual learning. The key idea of MoLT is to replace conventional, computationally heavy sequential adaptation at every transformer layer with a parallel, lightweight scheme that extracts and fuses layer-wise tokens only from the late layers. We adopt two types of adapters to distill modality-specific information and cross-modal interaction into compact latent tokens in a layer-wise manner. A token fusion module then dynamically fuses these layer-wise tokens by taking into account their relative significance. To prevent the redundancy of latent tokens, we apply an orthogonality regularization between latent tokens during training. Through the systematic analysis of the position of adaptation in the pre-trained transformers, we extract latent tokens only from the late layers of the transformers. This strategic adaptation approach avoids error propagation from the volatile early-layer features, thereby maximizing the adaptation performance while maintaining parameter and memory efficiency. Through extensive experiments, we demonstrate that MoLT outperforms existing methods on diverse audio-visual benchmarks, including Audio-Visual Question Answering, Audio-Visual Segmentation, and Audio-Visual Event Localization.