AC/DC: LLM-based Audio Comprehension via Dialogue Continuation
This addresses audio comprehension for applications needing flexible instruction-following, but it is incremental as it builds on existing LLM dialogue methods.
The researchers tackled the problem of audio comprehension by training a model to produce dialogue responses instead of direct captions, which mitigates caption variation and enables zero-shot instruction-following without multitask tuning. Their model demonstrated this capability on AudioCaps, WavCaps, and Clotho datasets with AudioBench tests.
We propose an instruction-following audio comprehension model that leverages the dialogue continuation ability of large language models (LLMs). Instead of directly generating target captions in training data, the proposed method trains a model to produce responses as if the input caption triggered a dialogue. This dialogue continuation training mitigates the caption variation problem. Learning to continue a dialogue effectively captures the caption's meaning beyond its surface-level words. As a result, our model enables zero-shot instruction-following capability without multitask instruction tuning, even trained solely on audio captioning datasets. Experiments on AudioCaps, WavCaps, and Clotho datasets with AudioBench audio-scene question-answering tests demonstrate our model's ability to follow various unseen instructions.