TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
For LLM alignment researchers, TinyJudge offers a scalable and robust method to enforce unverifiable constraints, reducing reward hacking and computational cost.
TinyJudge addresses reward hacking and high computational overhead in aligning LLMs with unverifiable constraints by using an ensemble of specialized tiny language models (~0.6B) as judges. It achieves ~10% higher average performance and 12% better reward precision than baselines, with a 3× speedup in training time.
Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.g., output length) to unverifiable ones (e.g., tone). Reinforcement learning with verifiable rewards has emerged as a paradigm for IF tasks, leveraging LLM-as-a-judge to assess unverifiable constraints. However, we empirically find that this approach remains a significant bottleneck, suffering from severe reward hacking and higher computational overhead. In this work, we first analyze the generalization capabilities of unverifiable constraints and discover that specific constraints exhibit distinct, high-generalization patterns. Motivated by this, we propose TinyJudge, a framework that employs an ensemble of specialized tiny language models ($\sim0.6B$) to provide rewards for soft constraints. By distilling expertise from frontier models into these tiny models, it achieves high-precision, lightweight evaluation. Extensive evaluations across five benchmarks demonstrate that TinyJudge outperforms the baselines by $\sim10\%$ in average performance and $12\%$ in reward precision. Crucially, it also achieves a $3\times$ speedup in total training time. Our work provides a scalable and robust path for aligning LLMs with unverifiable human instructions.