OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation
This addresses audio separation challenges for real-world applications where mixtures contain variable sources, representing a novel method for a known bottleneck.
The paper tackles the problem of audio separation in real-world scenarios with variable numbers of sources by proposing OpenSep, a framework that uses large language models with textual inversion to automatically separate unseen mixtures, outperforming state-of-the-art baseline methods.
Audio separation in real-world scenarios, where mixtures contain a variable number of sources, presents significant challenges due to limitations of existing models, such as over-separation, under-separation, and dependence on predefined training sources. We propose OpenSep, a novel framework that leverages large language models (LLMs) for automated audio separation, eliminating the need for manual intervention and overcoming source limitations. OpenSep uses textual inversion to generate captions from audio mixtures with off-the-shelf audio captioning models, effectively parsing the sound sources present. It then employs few-shot LLM prompting to extract detailed audio properties of each parsed source, facilitating separation in unseen mixtures. Additionally, we introduce a multi-level extension of the mix-and-separate training framework to enhance modality alignment by separating single source sounds and mixtures simultaneously. Extensive experiments demonstrate OpenSep's superiority in precisely separating new, unseen, and variable sources in challenging mixtures, outperforming SOTA baseline methods. Code is released at https://github.com/tanvir-utexas/OpenSep.git