ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models
This addresses the reliability issue in LLMs for mathematical reasoning, which is crucial for applications requiring accurate problem-solving, though it is incremental as it builds on prior work focused on knowledge tasks.
The paper tackles the problem of unreliable responses from Large Language Models (LLMs) on unsolvable mathematical reasoning tasks by developing a benchmark dataset and evaluating reliability across solvable and unsolvable problems. The results show that larger LLMs improve reliability with prompts but still lag on unsolvable problems, while small LLMs see little progress, but an alignment strategy significantly enhances their reliability on in-domain and out-of-domain tasks.
Although demonstrating remarkable performance on reasoning tasks, Large Language Models (LLMs) still tend to fabricate unreliable responses when confronted with problems that are unsolvable or beyond their capability, severely undermining the reliability. Prior studies of LLM reliability have primarily focused on knowledge tasks to identify unanswerable questions, while mathematical reasoning tasks have remained unexplored due to the dearth of unsolvable math problems. To systematically investigate LLM reliability in mathematical reasoning tasks, we formulate the reliability evaluation for both solvable and unsolvable problems. We then develop a ReliableMath dataset which incorporates open-source solvable problems and high-quality unsolvable problems synthesized by our proposed construction workflow with human evaluations. Experiments are conducted on various LLMs with several key findings uncovered. LLMs fail to directly identify unsolvable problems and always generate fabricated responses. When instructing LLMs to indicate unsolvability using a reliable prompt, the reliability of larger-sized LLMs remains on solvable problems, but notably improves on unsolvable problems yet still falls short of solvable problems. However, small LLMs rarely show any progress despite employing reliable prompts. Therefore, we further propose an alignment strategy to enhance small LLMs' reliability, which can significantly improve LLM reliability performances on both in-domain and out-of-domain tasks.