KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
For researchers evaluating LLMs in complex, non-stationary environments, this benchmark reveals significant gaps in current models' ability to handle open-ended goals and adapt over time.
KellyBench evaluates frontier language models on long-horizon sequential decision-making in sports betting markets, finding that all models lose money on average (best model: -8% return) and achieve low strategy sophistication scores (e.g., Claude Opus 4.6 scores 26.5% on a human expert rubric).
Language models are saturating benchmarks for procedural tasks with narrow objectives. But they are increasingly being deployed in long-horizon, non-stationary environments with open-ended goals. In this paper we introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets. Agents are placed in a sequential simulation of the 2023-24 English Premier League season and tasked with maximising their long-term bankroll growth. They are given detailed historical data, including advanced statistics, lineups, and public odds. To succeed they must build machine learning models, identify edge in public markets, and adapt as the environment changes over time. We find that all frontier models evaluated lose money on average over the course of the season for five seeds. The best performing model achieves an average return of -8%, and many models experiencing ruin across seeds. To judge strategy sophistication, we use a human expert rubric to grade each model and find their approaches to be unsophisticated compared to human baselines; Claude Opus 4.6 achieves a rubric score of 26.5%, which means there is significant room for improvement. KellyBench is available as an open-access API endpoint at https://openreward.ai/GeneralReasoning/KellyBench.