Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis
This provides a scalable and adaptable tool for evaluating LLMs' mathematical capabilities, addressing the limitations of manual benchmark creation and enabling broader domain exploration.
The authors tackled the problem of generating math benchmarks for LLMs by proposing a pipeline that uses AI-generated hypotheses to identify specific math concepts and skills LLMs struggle with, then creates problems targeting these weaknesses, reducing Llama-3.3-70B-Instruct's accuracy to as low as 45% compared to 77% on the original MATH benchmark.
Numerous math benchmarks exist to evaluate LLMs' mathematical capabilities. However, most involve extensive manual effort and are difficult to scale. Consequently, they cannot keep pace with LLM development or easily provide new instances to mitigate overfitting. Some researchers have proposed automatic benchmark generation methods, but few focus on identifying the specific math concepts and skills on which LLMs are error-prone, and most can only generate category-specific benchmarks. To address these limitations, we propose a new math benchmark generation pipeline that uses AI-generated hypotheses to identify the specific math concepts and skills that LLMs struggle with, and then generates new benchmark problems targeting these weaknesses. Experiments show that hypothesis accuracy positively correlates with the difficulty of the generated problems: problems generated from the most accurate hypotheses reduce Llama-3.3-70B-Instruct's accuracy to as low as 45%, compared to 77% on the original MATH benchmark. Furthermore, our pipeline is highly adaptable and can be applied beyond math to explore a wide range of LLM capabilities, making it a valuable tool for investigating how LLMs perform across different domains.