SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
This addresses the need for rigorous evaluation standards for LLMs in scientific discovery, though it is incremental as it builds on existing benchmarks by adding constraint-based auditing.
The authors tackled the problem of evaluating large language models (LLMs) in scientific contexts by introducing SciIF, a benchmark that measures scientific instruction following, which assesses both solution correctness and adherence to constraints like scientific conditions and semantic stability, resulting in a tool for fine-grained diagnosis of reasoning failures.
As large language models (LLMs) transition from general knowledge retrieval to complex scientific discovery, their evaluation standards must also incorporate the rigorous norms of scientific inquiry. Existing benchmarks exhibit a critical blind spot: general instruction-following metrics focus on superficial formatting, while domain-specific scientific benchmarks assess only final-answer correctness, often rewarding models that arrive at the right result with the wrong reasons. To address this gap, we introduce scientific instruction following: the capability to solve problems while strictly adhering to the constraints that establish scientific validity. Specifically, we introduce SciIF, a multi-discipline benchmark that evaluates this capability by pairing university-level problems with a fixed catalog of constraints across three pillars: scientific conditions (e.g., boundary checks and assumptions), semantic stability (e.g., unit and symbol conventions), and specific processes(e.g., required numerical methods). Uniquely, SciIF emphasizes auditability, requiring models to provide explicit evidence of constraint satisfaction rather than implicit compliance. By measuring both solution correctness and multi-constraint adherence, SciIF enables finegrained diagnosis of compositional reasoning failures, ensuring that LLMs can function as reliable agents within the strict logical frameworks of science.