Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models
For researchers working on reasoning in small language models, this work provides a method to correct step-level errors without requiring ground truth solutions or preference data, though conceptual errors remain challenging.
Small language models suffer from cascading errors in multi-step physics reasoning. The proposed step-level reward framework identifies the first error, generates structured feedback, and trains the model to revise via policy gradient, achieving accuracy gains of 17-20% over CoT prompting and reducing calculation errors from 56.9% to 23.5%.
Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.