Symbolic or Numerical? Understanding Physics Problem Solving in Reasoning LLMs
This addresses the challenge of physics reasoning for LLMs, showing incremental improvements through few-shot prompting.
The study tackled physics problem-solving by applying instruction-tuned reasoning LLMs like Deepseek-R1 to the SciBench benchmark, achieving state-of-the-art accuracy and revealing distinctive symbolic derivation patterns in reasoning.
Navigating the complexities of physics reasoning has long been a difficult task for Large Language Models (LLMs), requiring a synthesis of profound conceptual understanding and adept problem-solving techniques. In this study, we investigate the application of advanced instruction-tuned reasoning models, such as Deepseek-R1, to address a diverse spectrum of physics problems curated from the challenging SciBench benchmark. Our comprehensive experimental evaluation reveals the remarkable capabilities of reasoning models. Not only do they achieve state-of-the-art accuracy in answering intricate physics questions, but they also generate distinctive reasoning patterns that emphasize on symbolic derivation. Furthermore, our findings indicate that even for these highly sophisticated reasoning models, the strategic incorporation of few-shot prompting can still yield measurable improvements in overall accuracy, highlighting the potential for continued performance gains.