Factorio Learning Environment
This provides a new benchmark for evaluating LLMs on open-ended tasks, though it is incremental as it builds on existing game-based environments.
The authors tackled the problem of evaluating LLMs on open-ended tasks by introducing the Factorio Learning Environment (FLE), which tests agents in long-term planning and resource optimization, and found that models still lack strong spatial reasoning and fail at complex automation.
Large Language Models (LLMs) are rapidly saturating existing benchmarks, necessitating new open-ended evaluations. We introduce the Factorio Learning Environment (FLE), based on the game of Factorio, that tests agents in long-term planning, program synthesis, and resource optimization. FLE provides exponentially scaling challenges -- from basic automation to complex factories processing millions of resource units per second. We provide two settings: (1) lab-play consisting of eight structured tasks with fixed resources, and (2) open-play with the unbounded task of building the largest factory on an procedurally generated map. We demonstrate across both settings that models still lack strong spatial reasoning. In lab-play, we find that LLMs exhibit promising short-horizon skills, yet are unable to operate effectively in constrained environments, reflecting limitations in error analysis. In open-play, while LLMs discover automation strategies that improve growth (e.g electric-powered drilling), they fail to achieve complex automation (e.g electronic-circuit manufacturing).