Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithms
This addresses a bottleneck in multimodal intelligence by improving inference efficiency for generative models, though it appears incremental as it modifies existing diffusion models.
The paper tackles the stagnation in generative pre-training algorithms by proposing an inference-first perspective, which leads to a modified algorithm that achieves superior sample quality with over 10x greater inference efficiency.
Recent years have seen significant advancements in foundation models through generative pre-training, yet algorithmic innovation in this space has largely stagnated around autoregressive models for discrete signals and diffusion models for continuous signals. This stagnation creates a bottleneck that prevents us from fully unlocking the potential of rich multi-modal data, which in turn limits the progress on multimodal intelligence. We argue that an inference-first perspective, which prioritizes scaling efficiency during inference time across sequence length and refinement steps, can inspire novel generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a concrete example, we demonstrate how addressing limitations in diffusion models' inference process through targeted modifications yields a stable, single-stage algorithm that achieves superior sample quality with over an order of magnitude greater inference efficiency.