Embedding Alignment in Code Generation for Audio
This addresses a specific bottleneck in creative coding for audio, offering an incremental improvement to enhance musical diversity in code generation.
The paper tackles the problem of limited diversity in LLM-generated code for audio applications by investigating the relationship between code and audio embeddings, finding it's not linear but can be learned through a predictive model to create an embedding alignment map.
LLM-powered code generation has the potential to revolutionize creative coding endeavors, such as live-coding, by enabling users to focus on structural motifs over syntactic details. In such domains, when prompting an LLM, users may benefit from considering multiple varied code candidates to better realize their musical intentions. Code generation models, however, struggle to present unique and diverse code candidates, with no direct insight into the code's audio output. To better establish a relationship between code candidates and produced audio, we investigate the topology of the mapping between code and audio embedding spaces. We find that code and audio embeddings do not exhibit a simple linear relationship, but supplement this with a constructed predictive model that shows an embedding alignment map could be learned. Supplementing the aim for musically diverse output, we present a model that given code predicts output audio embedding, constructing a code-audio embedding alignment map.