Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment
This work addresses the challenge of coherence in text-to-music generation for applications in music composition and AI-assisted creativity, but it is incremental as it builds on an existing model without requiring retraining.
The paper tackles the problem of generating symbolic music that aligns with input captions by introducing inference-time alignment objectives, resulting in significant improvements in both objective and subjective evaluation metrics.
We present Text2midi-InferAlign, a novel technique for improving symbolic music generation at inference time. Our method leverages text-to-audio alignment and music structural alignment rewards during inference to encourage the generated music to be consistent with the input caption. Specifically, we introduce two objectives scores: a text-audio consistency score that measures rhythmic alignment between the generated music and the original text caption, and a harmonic consistency score that penalizes generated music containing notes inconsistent with the key. By optimizing these alignment-based objectives during the generation process, our model produces symbolic music that is more closely tied to the input captions, thereby improving the overall quality and coherence of the generated compositions. Our approach can extend any existing autoregressive model without requiring further training or fine-tuning. We evaluate our work on top of Text2midi - an existing text-to-midi generation model, demonstrating significant improvements in both objective and subjective evaluation metrics.