FLAP: Fast Language-Audio Pre-training
This addresses the challenge of audio-text alignment for retrieval applications, representing an incremental improvement over existing methods.
The paper tackles the problem of learning aligned audio and language representations efficiently, proposing FLAP which achieves state-of-the-art performance on audio-text retrieval tasks, with 53.0% R@1 on AudioCaps and 25.5% R@1 on Clotho.
We propose Fast Language-Audio Pre-training (FLAP), a self-supervised approach that efficiently and effectively learns aligned audio and language representations through masking, contrastive learning and reconstruction. For efficiency, FLAP randomly drops audio spectrogram tokens, focusing solely on the remaining ones for self-supervision. Through inter-modal contrastive learning, FLAP learns to align paired audio and text representations in a shared latent space. Notably, FLAP leverages multiple augmented views via masking for inter-modal contrast and learns to reconstruct the masked portion of audio tokens. Moreover, FLAP leverages large language models (LLMs) to augment the text inputs, contributing to improved performance. These approaches lead to more robust and informative audio-text representations, enabling FLAP to achieve state-of-the-art (SoTA) performance on audio-text retrieval tasks on AudioCaps (achieving 53.0% R@1) and Clotho (achieving 25.5% R@1).