Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
For researchers evaluating LLM mathematical reasoning, this work provides a benchmark to measure proof quality beyond correctness, revealing that current evaluations miss important aspects of usefulness.
LLMs often produce correct proofs, but correctness alone is insufficient; this work identifies proof quality components (conciseness, computational ease, cognitive simplicity, diversity, adaptivity) and introduces ProofRank, a benchmark from mathematical competitions. Results show substantial quality differences not captured by correctness-only metrics and trade-offs between quality and correctness.
Large language models (LLMs) have become capable mathematical problem-solvers, often producing correct proofs for challenging problems. However, correctness alone is not sufficient: mathematical proofs should also be clear, concise, insightful, and transferable to other problems. While this proof quality is subjective and depends on the reader and context, many of its components are concrete and broadly valued. In this work, we identify such components and introduce ProofRank, a benchmark curated from challenging mathematical competitions. ProofRank evaluates several scalable proxies of proof quality: (i) conciseness, measuring whether proofs avoid unnecessary steps; (ii) computational ease, measuring the extent to which a proof relies on tedious calculations; (iii) cognitive simplicity, measuring how accessible the used proof techniques are; (iv) diversity, measuring how varied a model's proofs for a single problem are; and (v) adaptivity, measuring whether a model can follow a specified proof technique. Across models, we find substantial differences in proof quality that are not captured by correctness-only benchmarks. We also observe significant trade-offs between proof-quality metrics and correctness, suggesting that future evaluations of mathematical reasoning should measure how useful LLM-generated proofs are.