ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction
This addresses the need for rigorous evaluation of LLMs in scientific domains, though it is incremental as it focuses on a specific task rather than broad reasoning improvements.
The authors tackled the problem of evaluating whether large language models (LLMs) truly understand scientific reasoning by introducing a novel task called Latent Reasoning Chain Extraction (ARCHE), which requires models to decompose arguments into structured reasoning paradigms, and found that on a benchmark of 70 Nature Communications articles, none of 10 leading LLMs could extract a complete and standard reasoning chain, revealing a substantial gap in model abilities.
Large language models (LLMs) are increasingly used in scientific domains. While they can produce reasoning-like content via methods such as chain-of-thought prompting, these outputs are typically unstructured and informal, obscuring whether models truly understand the fundamental reasoning paradigms that underpin scientific inference. To address this, we introduce a novel task named Latent Reasoning Chain Extraction (ARCHE), in which models must decompose complex reasoning arguments into combinations of standard reasoning paradigms in the form of a Reasoning Logic Tree (RLT). In RLT, all reasoning steps are explicitly categorized as one of three variants of Peirce's fundamental inference modes: deduction, induction, or abduction. To facilitate this task, we release ARCHE Bench, a new benchmark derived from 70 Nature Communications articles, including more than 1,900 references and 38,000 viewpoints. We propose two logic-aware evaluation metrics: Entity Coverage (EC) for content completeness and Reasoning Edge Accuracy (REA) for step-by-step logical validity. Evaluations on 10 leading LLMs on ARCHE Bench reveal that models exhibit a trade-off between REA and EC, and none are yet able to extract a complete and standard reasoning chain. These findings highlight a substantial gap between the abilities of current reasoning models and the rigor required for scientific argumentation.