TaxCalcBench: Evaluating Frontier Models on the Tax Calculation Task
This work addresses the challenge of applying AI to personal income tax calculation, which is an incremental step in assessing model capabilities for real-world financial tasks.
The authors tackled the problem of evaluating AI models' ability to calculate US personal income taxes by introducing TaxCalcBench, a benchmark for this task, and found that state-of-the-art models succeeded in calculating less than a third of federal income tax returns on a simplified sample set.
Can AI file your taxes? Not yet. Calculating US personal income taxes is a task that requires building an understanding of vast amounts of English text and using that knowledge to carefully compute results. We propose TaxCalcBench, a benchmark for determining models' abilities to calculate personal income tax returns given all of the necessary information. Our experiment shows that state-of-the-art models succeed in calculating less than a third of federal income tax returns even on this simplified sample set. Our analysis concludes that models consistently misuse tax tables, make errors in tax calculation, and incorrectly determine eligibility. Our findings point to the need for additional infrastructure to apply LLMs to the personal income tax calculation task.