Online Rubrics Elicitation from Pairwise Comparisons
This addresses the issue of dynamic evaluation criteria for LLM training in open-ended tasks, offering an incremental improvement over existing rubric-based methods.
The paper tackled the problem of static rubrics in LLM training being vulnerable to reward hacking and missing emergent criteria, and introduced Online Rubrics Elicitation, which improved performance by up to 8% over static rubrics on benchmarks like AlpacaEval and GPQA.
Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and reasoning.