HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination
This addresses the need for better evaluation of LLMs in scientific workflows where multiple explanations are possible, though it is incremental as it focuses on diagnostic tools rather than new model capabilities.
The paper tackles the problem of evaluating LLMs' ability to generate multiple plausible explanations for underdetermined scientific problems, introducing HypoSpace, a diagnostic suite that measures validity, uniqueness, and recovery of hypothesis sets, with results showing high validity but degraded uniqueness and recovery as the hypothesis space grows.
As language models are increasingly used in scientific workflows, evaluating their ability to propose sets of explanations-not just a single correct answer-becomes critical. Many scientific problems are underdetermined: multiple, mechanistically distinct hypotheses are consistent with the same observations. We introduce HypoSpace, a diagnostic suite that treats LLMs as samplers of finite hypothesis sets and measures three complementary indicators: Validity (precision of proposals consistent with observations), Uniqueness (non-redundancy among proposals), and Recovery (coverage of the enumerated admissible set). We instantiate HypoSpace in three structured domains with deterministic validators and exactly enumerated hypothesis spaces: (i) causal graphs from perturbations, (ii) gravity-constrained 3D voxel reconstruction from top-down projections, and (iii) Boolean genetic interactions. Across instruction-tuned and reasoning-focused models, Validity often remains high while Uniqueness and Recovery degrade as the admissible space grows, revealing mode collapse that is invisible to correctness-only metrics. HypoSpace offers a controlled probe-rather than a leaderboard-for methods that explicitly explore and cover admissible explanation spaces. Code is available at: https://github.com/CTT-Pavilion/_HypoSpace.