FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference
This work addresses efficiency issues for users of AV-LLMs, though it is incremental as it adapts existing token pruning techniques to a new multimodal context.
The paper tackles the problem of high computational cost in audio-visual large language models (AV-LLMs) by introducing FastAV, a token pruning framework that reduces FLOPs by over 40% while maintaining or improving performance.
In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance.