CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning
For audio captioning, this work reduces inference cost by eliminating the encoder while maintaining competitive performance, though the gap to the upper bound indicates room for improvement.
CARD removes the audio encoder at inference for audio captioning by distilling a teacher model's representations across components, achieving CIDEr-D of 55.4 on AudioCaps (vs. 66.4 upper bound) and outperforming LLM-only distillation by +12.18.
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.