Pipeline for Verifying LLM-Generated Mathematical Solutions
This work addresses the need for more accurate verification methods in benchmarks for large reasoning models, though it is incremental as it builds on existing verification techniques.
The authors tackled the problem of verifying solutions generated by large language models for mathematical problems by introducing a pipeline for automatic and interactive verification, which reduces false positives and can generate correct solutions in formal and informal languages.
With the growing popularity of Large Reasoning Models and their results in solving mathematical problems, it becomes crucial to measure their capabilities. We introduce a pipeline for both automatic and interactive verification as a more accurate alternative to only checking the answer which is currently the most popular approach for benchmarks. The pipeline can also be used as a generator of correct solutions both in formal and informal languages. 3 AI agents, which can be chosen for the benchmark accordingly, are included in the structure. The key idea is the use of prompts to obtain the solution in the specific form which allows for easier verification using proof assistants and possible use of small models ($\le 8B$). Experiments on several datasets suggest low probability of False Positives. The open-source implementation with instructions on setting up a server is available at https://github.com/LogicEnj/lean4_verification_pipeline.