Peer-Predictive Self-Training for Language Model Reasoning
For researchers working on self-supervised language model improvement, PST offers a novel method to enhance reasoning without external labels, though gains are incremental on small models.
The paper introduces Peer-Predictive Self-Training (PST), a label-free fine-tuning framework where multiple language models improve collaboratively using cross-model aggregated responses as internal training signals. On mathematical reasoning benchmarks, PST improves exact-match accuracy by 2.2 to 4.3 percentage points and reduces the generator-verifier gap by 26 to 40 percent across small models.
Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models improve collaboratively by leveraging a cross-model aggregated response as an internal training signal. Given a prompt question, the models generate responses sequentially; the final aggregated answer, often more reliable than individual responses in practice, serves as an internal target for learning. We measure how informative each intermediate response is about the aggregate using pointwise mutual information (PMI), and use this signal to scale self-training updates. Responses already aligned with the aggregate are updated less, while less informative or misaligned responses are updated more. On mathematical reasoning benchmarks (SimulEq, Math500, and MultiArith), PST improves exact-match accuracy by 2.2 to 4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen-2.5-1.5B, and reduces the average generator-verifier gap (GV-Gap) by 26 to 40 percent, while requiring no external supervision or teacher-student hierarchy and relying solely on cross-model interactions. These results suggest that cross-model generations and peer-predictive feedback can serve as an effective approach for self-supervised training.