Discriminator-Free Direct Preference Optimization for Video Diffusion
This addresses data inefficiency and evaluation uncertainty in aligning video diffusion models with human preferences, representing an incremental improvement for video generation tasks.
The paper tackled the challenges of applying Direct Preference Optimization (DPO) to video diffusion models by proposing a discriminator-free framework that uses real videos as win cases and edited versions as lose cases, eliminating costly synthetic comparisons and enabling unlimited training data expansion.
Direct Preference Optimization (DPO), which aligns models with human preferences through win/lose data pairs, has achieved remarkable success in language and image generation. However, applying DPO to video diffusion models faces critical challenges: (1) Data inefficiency. Generating thousands of videos per DPO iteration incurs prohibitive costs; (2) Evaluation uncertainty. Human annotations suffer from subjective bias, and automated discriminators fail to detect subtle temporal artifacts like flickering or motion incoherence. To address these, we propose a discriminator-free video DPO framework that: (1) Uses original real videos as win cases and their edited versions (e.g., reversed, shuffled, or noise-corrupted clips) as lose cases; (2) Trains video diffusion models to distinguish and avoid artifacts introduced by editing. This approach eliminates the need for costly synthetic video comparisons, provides unambiguous quality signals, and enables unlimited training data expansion through simple editing operations. We theoretically prove the framework's effectiveness even when real videos and model-generated videos follow different distributions. Experiments on CogVideoX demonstrate the efficiency of the proposed method.