Curriculum Sampling: A Two-Phase Curriculum for Efficient Training of Flow Matching
This addresses a specific bottleneck in training diffusion models for researchers and practitioners, offering an incremental improvement in efficiency and quality.
The paper tackled the problem of timestep sampling in Flow Matching models, showing that static middle-biased distributions accelerate early convergence but reduce asymptotic fidelity compared to Uniform sampling, and proposed Curriculum Sampling, a two-phase schedule that improved FID from 3.85 to 3.22 on CIFAR-10 while reducing training steps from 150k to 100k.
Timestep sampling $p(t)$ is a central design choice in Flow Matching models, yet common practice increasingly favors static middle-biased distributions (e.g., Logit-Normal). We show that this choice induces a speed--quality trade-off: middle-biased sampling accelerates early convergence but yields worse asymptotic fidelity than Uniform sampling. By analyzing per-timestep training losses, we identify a U-shaped difficulty profile with persistent errors near the boundary regimes, implying that under-sampling the endpoints leaves fine details unresolved. Guided by this insight, we propose \textbf{Curriculum Sampling}, a two-phase schedule that begins with middle-biased sampling for rapid structure learning and then switches to Uniform sampling for boundary refinement. On CIFAR-10, Curriculum Sampling improves the best FID from $3.85$ (Uniform) to $3.22$ while reaching peak performance at $100$k rather than $150$k training steps. Our results highlight that timestep sampling should be treated as an evolving curriculum rather than a fixed hyperparameter.