Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning
This addresses a critical issue in scalable AI alignment for researchers and practitioners, though it is incremental by focusing on logical coherence rather than judge accuracy.
The paper tackles the problem of judgment inconsistencies, particularly preference cycles, in AI feedback for aligning language models, which destabilizes reinforcement learning. It introduces a framework with a novel metric and a signal-purification method, resulting in significantly improved training stability and model performance over baselines.
Aligning language models using LLM judge feedback offers a scalable alternative to human annotation, yet is plagued by judgment inconsistencies that destabilize reinforcement learning. While prior work has focused on judge accuracy, the critical issue of logical coherence particularly preference cycles has been largely unaddressed. To address this gap, this work introduces an end to end framework to systematically detect and resolve these inconsistencies within the reinforcement learning training loop. Our framework features two core contributions: the Conflict Detection Rate (CDR), a novel metric to quantify judgment conflicts, and Deconflicted Graph Rewards (DGR), a signal-purification framework that eliminates cycles before policy optimization. DGR constructs preference graphs from raw judgments, transforms them into conflict-free Directed Acyclic Graphs (DAGs), and generates a logically coherent reward signal compatible with any policy optimizer. Experiments confirm that our framework significantly improves training stability and model performance over strong baselines, establishing logical consistency as a crucial and now-addressable dimension of AI feedback. The code for our method is available at https://github.com/modelscope/RM-Gallery.