EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
This work addresses audio captioning for applications like accessibility and media indexing, but it is incremental as it builds on existing models like EnCodec and CLAP.
The authors tackled automated audio captioning by combining neural audio codec and audio-text joint embeddings, achieving performance improvements over baseline models on AudioCaps and Clotho datasets.
We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP employs two acoustic representation models, EnCodec and CLAP, along with a pretrained language model, BART. We also introduce a new training objective called masked codec modeling that improves acoustic awareness of the pretrained language model. Experimental results on AudioCaps and Clotho demonstrate that our model surpasses the performance of baseline models. Source code will be available at https://github.com/jaeyeonkim99/EnCLAP . An online demo is available at https://huggingface.co/spaces/enclap-team/enclap .