Gradient Variance Reveals Failure Modes in Flow-Based Generative Models
This addresses a fundamental issue in generative modeling for researchers, revealing that straight-path objectives can conceal memorization, which is incremental as it builds on existing flow-based methods.
The paper tackled the problem of failure modes in flow-based generative models, specifically showing that deterministic training in Rectified Flows leads to memorization of arbitrary training pairings, and it found that injecting small noise restores generalization, as validated on the CelebA dataset.
Rectified Flows learn ODE vector fields whose trajectories are straight between source and target distributions, enabling near one-step inference. We show that this straight-path objective conceals fundamental failure modes: under deterministic training, low gradient variance drives memorization of arbitrary training pairings, even when interpolant lines between pairs intersect. To analyze this mechanism, we study Gaussian-to-Gaussian transport and use the loss gradient variance across stochastic and deterministic regimes to characterize which vector fields optimization favors in each setting. We then show that, in a setting where all interpolating lines intersect, applying Rectified Flow yields the same specific pairings at inference as during training. More generally, we prove that a memorizing vector field exists even when training interpolants intersect, and that optimizing the straight-path objective converges to this ill-defined field. At inference, deterministic integration reproduces the exact training pairings. We validate our findings empirically on the CelebA dataset, confirming that deterministic interpolants induce memorization, while the injection of small noise restores generalization.