TimbreCLIP: Connecting Timbre to Text and Images
This work addresses the problem of cross-modal retrieval and generation for audio and text, but it is incremental as it builds on existing CLIP-like frameworks.
The authors tackled the problem of connecting timbre to text and images by developing TimbreCLIP, an audio-text cross-modal embedding trained on single instrument notes, and demonstrated its application in tasks like text-driven audio equalization and timbre to image generation.
We present work in progress on TimbreCLIP, an audio-text cross modal embedding trained on single instrument notes. We evaluate the models with a cross-modal retrieval task on synth patches. Finally, we demonstrate the application of TimbreCLIP on two tasks: text-driven audio equalization and timbre to image generation.