Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning
This work addresses the problem of improving agentic scientific reasoning for heliophysics researchers, but it is incremental as it builds on existing methods with a new dataset and benchmarking approach.
The paper tackles the challenge of scientific reasoning in heliophysics using Large Language Models by introducing a new dataset and benchmark, finding that multi-agent workflows based on systems engineering principles outperform direct prompting on deductive reasoning tasks.
Scientific reasoning through Large Language Models in heliophysics involves more than just recalling facts: it requires incorporating physical assumptions, maintaining consistent units, and providing clear scientific formats through coordinated approaches. To address these challenges, we present Reasoning With a Star, a newly contributed heliophysics dataset applicable to reasoning; we also provide an initial benchmarking approach. Our data are constructed from National Aeronautics and Space Administration & University Corporation for Atmospheric Research Living With a Star summer school problem sets and compiled into a readily consumable question-and-answer structure with question contexts, reasoning steps, expected answer type, ground-truth targets, format hints, and metadata. A programmatic grader checks the predictions using unit-aware numerical tolerance, symbolic equivalence, and schema validation. We benchmark a single-shot baseline and four multi-agent patterns, finding that decomposing workflows through systems engineering principles outperforms direct prompting on problems requiring deductive reasoning rather than pure inductive recall.