Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
This addresses the challenge of post-training supervision for AI models in tasks lacking ground truth, offering a novel method that is not incremental but introduces a new paradigm for leveraging inference compute.
The paper tackles the problem of generating learning signals for post-training models without ground truth by proposing Compute as Teacher (CaT), which uses inference-time compute to create reference-free supervision from parallel rollouts, resulting in performance improvements such as up to +27% on MATH-500 and +12% on HealthBench for models like Gemma 3 4B.
Where do learning signals come from when there is no ground truth in post-training? We propose turning exploration into supervision through Compute as Teacher (CaT), which converts the model's own exploration at inference-time into reference-free supervision by synthesizing a single reference from a group of parallel rollouts and then optimizing toward it. Concretely, the current policy produces a group of rollouts; a frozen anchor (the initial policy) reconciles omissions and contradictions to estimate a reference, turning extra inference-time compute into a teacher signal. We turn this into rewards in two regimes: (i) verifiable tasks use programmatic equivalence on final answers; (ii) non-verifiable tasks use self-proposed rubrics-binary, auditable criteria scored by an independent LLM judge, with reward given by the fraction satisfied. Unlike selection methods (best-of-N, majority, perplexity, or judge scores), synthesis may disagree with the majority and be correct even when all rollouts are wrong; performance scales with the number of rollouts. As a test-time procedure, CaT improves Gemma 3 4B, Qwen 3 4B, and Llama 3.1 8B (up to +27% on MATH-500; +12% on HealthBench). With reinforcement learning (CaT-RL), we obtain further gains (up to +33% and +30%), with the trained policy surpassing the initial teacher signal.