CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts
This work addresses the challenge of maintaining reasoning consistency in expert LLMs for domains with limited data, offering a cost-effective alternative to expensive methods like Process Reward Models.
The paper tackles the problem of training expert LLMs in data-scarce domains using multiple-choice questions, where standard reinforcement learning risks degrading reasoning quality like logical consistency. The proposed CLARity framework improves response consistency by 16.5% and accuracy by 7.5% over baselines, as validated by experiments and human evaluations.
Training expert LLMs in domains with scarce data is difficult, often relying on multiple-choice questions (MCQs). However, standard outcome-based reinforcement learning (RL) on MCQs is risky. While it may improve accuracy, we observe it often degrades reasoning quality such as logical consistency. Existing solutions to supervise reasoning, such as large-scale Process Reward Models (PRMs), are prohibitively expensive. To address this, we propose CLARity, a cost-effective RL framework that enhances reasoning quality using only a small, general-purpose LLM. CLARity integrates a consistency-aware reward mechanism with a 2-stage refine-then-monitor training pipeline to enhance reasoning consistency, and a dynamic data reformulation strategy to to better exploit limited data. Experiments demonstrate that CLARity improves response consistency by 16.5% and accuracy by 7.5% over baselines. Human evaluations further confirm holistic improvements in coherence and professionalism. Thus, CLARity offers a generalizable solution that enables smaller models to effectively guide expert models by reasoning consistency.Our code is open sourced at: https://github.com/Infinite-set/CLARity