Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring
For researchers and practitioners in code generation, this work provides a benchmark and trained models for multi-criteria code reward modeling, addressing a gap in existing RM research.
The authors compiled a benchmark (Themis-CodeRewardBench) to evaluate code reward models across five criteria and eight languages, finding current RMs limited beyond functional correctness. They then trained Themis-RM (600M-32B parameters) on the largest open-source code preference dataset (350k+ pairs), achieving strong cross-lingual transfer and multi-criteria scoring.
Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.