VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
It addresses the problem of aligning AI-generated images and videos with human preferences for users and developers, offering an incremental improvement over existing reward models.
The paper tackles the challenge of aligning visual generative models with human preferences by introducing VisionReward, a framework that learns fine-grained, interpretable preferences for image and video generation, resulting in a 17.2% improvement in preference prediction accuracy and a 31.6% higher pairwise win rate compared to existing methods.
Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward.