TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks
This addresses the need for more efficient and verifiable programmatic solutions in AI tasks like math and image reasoning, though it is incremental as it builds on existing code LMs.
The researchers tackled the problem of language models generating verbose and error-prone programs for tasks like table QA and image reasoning by developing TROVE, a training-free method that induces reusable high-level functions, resulting in simpler solutions with higher accuracy across 11 datasets and 79-98% smaller toolboxes.
Language models (LMs) can solve tasks such as answering questions about tables or images by writing programs. However, using primitive functions often leads to verbose and error-prone programs, and higher-level functions require expert design. To enable better solutions without human labor, we ask code LMs to curate reusable high-level functions, and use them to write solutions. We present TROVE, a training-free method of inducing a verifiable and efficient toolbox of functions, by generating via using, growing, and periodically trimming the toolbox. On 11 datasets from math, table question answering, and image reasoning tasks, TROVE consistently yields simpler solutions with higher accuracy than baselines using CODELLAMA and previous methods using GPT, while using 79-98% smaller toolboxes. TROVE further enables 31% faster and 13% more accurate human verification than baselines. With the same pipeline, it creates diverse functions for varied tasks and datasets, providing insights into their individual characteristics.