Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning
This work addresses the problem of audio captioning for applications requiring high-quality audio descriptions, such as assistive technologies or multimedia analysis, and presents an incremental improvement over existing methods.
This work tackles the problem of exposure bias in audio captioning, resulting in a framework that can generate high-quality audio captions with increased caption length, lexical diversity, and text-to-audio self-retrieval accuracy. The framework achieves this through the introduction of the unbiased sliced Wasserstein RBF kernel, which reduces approximation error at a rate of $mathcal{O}(L^{-1/2})$.
Teacher-forcing training for audio captioning usually leads to exposure bias due to training and inference mismatch. Prior works propose the contrastive method to deal with caption degeneration. However, the contrastive method ignores the temporal information when measuring similarity across acoustic and linguistic modalities, leading to inferior performance. In this work, we develop the temporal-similarity score by introducing the unbiased sliced Wasserstein RBF (USW-RBF) kernel equipped with rotary positional embedding to account for temporal information across modalities. In contrast to the conventional sliced Wasserstein RBF kernel, we can form an unbiased estimation of USW-RBF kernel via Monte Carlo estimation. Therefore, it is well-suited to stochastic gradient optimization algorithms, and its approximation error decreases at a parametric rate of $\mathcal{O}(L^{-1/2})$ with $L$ Monte Carlo samples. Additionally, we introduce an audio captioning framework based on the unbiased sliced Wasserstein kernel, incorporating stochastic decoding methods to mitigate caption degeneration during the generation process. We conduct extensive quantitative and qualitative experiments on two datasets, AudioCaps and Clotho, to illustrate the capability of generating high-quality audio captions. Experimental results show that our framework is able to increase caption length, lexical diversity, and text-to-audio self-retrieval accuracy.