BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting
This benchmark addresses the lack of standardized evaluation for LLM-based automated backtesting, a critical bottleneck for democratizing quantitative finance.
BacktestBench introduces the first large-scale benchmark for automated quantitative backtesting, comprising 18,246 annotated QA pairs from over 6 million market records. Evaluation of 23 LLMs with the proposed AutoBacktest baseline identifies key factors for end-to-end performance.
Quantitative backtesting is essential for evaluating trading strategies but remains hampered by high technical barriers and limited scalability. While Large Language Models (LLMs) offer a transformative path to automate this complex, interdisciplinary workflow through advanced code generation, tool usage, and agentic planning, the practical realization is significantly challenged by the current lack of a large-scale benchmark dedicated to automated quantitative backtesting, which hinders progress in this field. To bridge this critical gap, we introduce BacktestBench, the first large-scale benchmark for automated quantitative backtesting. Built from over 6 million real market records, it comprises 18,246 meticulously annotated question-answering pairs across four task categories: metrics calculation, ticker selection, strategy selection, and parameter confirmation. We also propose AutoBacktest, a robust multi-agent baseline that translates natural language strategies into reproducible backtests by coordinating a Summarizer for semantic factor extraction, a Retriever for validated SQL generation, and a Coder for Python backtesting implementation. Our evaluation on 23 mainstream LLMs, complemented by targeted ablations, identifies key factors that influence end-to-end performance and highlights the importance of grounded verification and standardized indicator representations.