ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
This addresses the limitation of single-reward alignment in generative models for users needing flexible trade-offs, offering a novel solution for multi-criteria control.
The paper tackles the problem of aligning generative models with multiple conflicting human preferences by introducing ParetoSlider, a multi-objective RL framework that trains a single diffusion model to approximate the entire Pareto front, enabling inference-time control over trade-offs without retraining. It matches or exceeds baseline performance across three state-of-the-art backbones, providing fine-grained control over goals like prompt adherence versus source fidelity.
Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences, yet most methods rely on a single scalar reward. When multiple criteria matter, the prevailing practice of ``early scalarization'' collapses rewards into a fixed weighted sum. This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals -- such as prompt adherence versus source fidelity in image editing. We introduce ParetoSlider, a multi-objective RL (MORL) framework that trains a single diffusion model to approximate the entire Pareto front. By training the model with continuously varying preference weights as a conditioning signal, we enable users to navigate optimal trade-offs at inference time without retraining or maintaining multiple checkpoints. We evaluate ParetoSlider across three state-of-the-art flow-matching backbones: SD3.5, FluxKontext, and LTX-2. Our single preference-conditioned model matches or exceeds the performance of baselines trained separately for fixed reward trade-offs, while uniquely providing fine-grained control over competing generative goals.