Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks
This addresses the challenge of evaluating LLMs in formal tasks where hand-curated datasets are hard to obtain or update, though it is incremental as it builds on existing benchmarking paradigms.
The paper tackles the problem of scaling objective evaluation of Large Language Models (LLMs) for tasks like truth maintenance and logical reasoning by introducing AutoEval, a benchmark that auto-generates tasks and ground truth, eliminating the need for human labeling. Empirical analysis shows that performance on AutoEval is highly indicative of performance on other benchmarks for translation and reasoning tasks.
This paper presents AutoEval, a novel benchmark for scaling Large Language Model (LLM) assessment in formal tasks with clear notions of correctness, such as truth maintenance in translation and logical reasoning. AutoEval is the first benchmarking paradigm that offers several key advantages necessary for scaling objective evaluation of LLMs without human labeling: (a) ability to evaluate LLMs of increasing sophistication by auto-generating tasks at different levels of difficulty; (b) auto-generation of ground truth that eliminates dependence on expensive and time-consuming human annotation; (c) the use of automatically generated, randomized datasets that mitigate the ability of successive LLMs to overfit to static datasets used in many contemporary benchmarks. Empirical analysis shows that an LLM's performance on AutoEval is highly indicative of its performance on a diverse array of other benchmarks focusing on translation and reasoning tasks, making it a valuable autonomous evaluation paradigm in settings where hand-curated datasets can be hard to obtain and/or update.