CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation
This addresses the need for cost-effective and accurate evaluation metrics in audio captioning, offering a reference-free solution that outperforms existing baselines.
The paper tackles the problem of robust evaluation in audio captioning by proposing CAF-Score, a reference-free metric that calibrates CLAP's semantic alignment with LALMs' fine-grained comprehension, achieving the highest correlation with human judgments on the BRACE benchmark.
While Large Audio-Language Models (LALMs) have advanced audio captioning, robust evaluation remains difficult. Reference-based metrics are expensive and often fail to assess acoustic fidelity, while Contrastive Language-Audio Pretraining (CLAP)-based approaches frequently overlook syntactic errors and fine-grained details. We propose CAF-Score, a reference-free metric that calibrates CLAP's coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs. By combining contrastive audio-text embeddings with LALM reasoning, CAF-Score effectively detects syntactic inconsistencies and subtle hallucinations. Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments, even outperforming reference-based baselines in challenging scenarios. These results highlight the efficacy of CAF-Score for reference-free audio captioning evaluation. Code and results are available at https://github.com/inseong00/CAF-Score.