Insertion Based Sequence Generation with Learnable Order Dynamics
This addresses the problem of flexible sequence generation for domains like graph traversal and molecule design, though it appears incremental as it builds on existing insertion and diffusion models.
The paper tackles the challenge of learning insertion-based sequence generation models, which have larger action spaces than autoregressive models, by incorporating trainable order dynamics into discrete flow matching. The result shows that on de novo small molecule generation, learned order dynamics increases the number of valid molecules generated and improves quality compared to uniform order dynamics.
In many domains generating variable length sequences through insertions provides greater flexibility over autoregressive models. However, the action space of insertion models is much larger than that of autoregressive models (ARMs) making the learning challenging. To address this, we incorporate trainable order dynamics into the target rates for discrete flow matching, and show that with suitable choices of parameterizations, joint training of the target order dynamics and the generator is tractable without the need for numerical simulation. As the generative insertion model, we use a variable length masked diffusion model, which generates by inserting and filling mask tokens. On graph traversal tasks for which a locally optimal insertion order is known, we explore the choices of parameterization empirically and demonstrate the trade-offs between flexibility, training stability and generation quality. On de novo small molecule generation, we find that the learned order dynamics leads to an increase in the number of valid molecules generated and improved quality, when compared to uniform order dynamics.