In small language models, physics reasoning fails structurally: an error at any step propagates forward, corrupting every subsequent inference. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required, and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0%. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.
Blogger's Review: This study addresses systemic errors in physics reasoning within small language models through a step-level reward mechanism, demonstrating the effectiveness of structured feedback. The approach optimizes the reasoning process without needing ground truth answers, laying a foundation for the agent's self-correction capabilities. Overall, the accuracy improvements and error reductions present significant practical application value.