Segmentwise Pruning in Audio-Language Models
This work addresses efficiency issues for audio-language model practitioners, but it is incremental as it adapts existing token pruning methods from vision-language to audio domains.
The paper tackles the high computational cost of audio-language models due to long audio sequences by proposing a lightweight token pruning method that considers the time dimension, achieving a maximum relative decrease of 2% in CIDEr on Clotho v2 and 4% in accuracy on MMAU while retaining only a quarter of the initial tokens.
Recent audio-language models have shown impressive performance across a wide range of audio tasks and are increasingly capable of handling long audio inputs. However, the computing costs in these models heavily depend on sequence length, which can become very large given the nature of audio data. In the vision-language domain, token pruning methods have proven effective in reducing token counts while preserving strong performance on standard benchmarks. In this work, we investigate the relevance and effectiveness of such token selection strategies in the context of audio-language models. We also improve them by proposing a lightweight strategy that takes the time dimension into account. While retaining only a quarter of the initial tokens, our approach results in a relative maximum decrease of 2% in CIDEr on Clotho v2 and a relative maximum decrease of 4% in accuracy on MMAU.