Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
For researchers and practitioners using RLHF with generative models, JRM provides a more efficient and accurate reward model that balances semantic understanding and inference speed.
Joint Reward Modeling (JRM) internalizes chain-of-thought reasoning from generative models into a discriminative reward model, achieving state-of-the-art results on MMRB2 and EditReward-Bench while enabling fast inference and improving downstream RL stability.
Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models. For complex tasks such as image editing, reward models are required to capture global semantic consistency and implicit logical constraints beyond local similarity. Existing reward modeling approaches have clear limitations. Discriminative reward models align well with human preferences but struggle with complex semantics due to limited reasoning supervision. Generative reward models offer stronger semantic understanding and reasoning, but they are costly at inference time and difficult to align directly with human preferences. To this end, we propose Joint Reward Modeling (JRM), which jointly optimizes preference learning and language modeling on a shared vision-language backbone. This approach internalizes the semantic and reasoning capabilities of generative models into efficient discriminative representations, enabling fast and accurate evaluation. JRM achieves state-of-the-art results on MMRB2 and EditReward-Bench, and significantly improves stability and performance in downstream online reinforcement learning. These results show that joint training effectively bridges efficiency and semantic understanding in reward modeling.