Develop AI Agents for System Engineering in Factorio
This addresses the need for better AI agents in system engineering for software, manufacturing, energy, and logistics, but it is incremental as it suggests a direction rather than presenting new results.
The paper tackles the problem that current AI agent evaluations fail to capture skills needed for dynamic system engineering, and proposes using Factorio as a sandbox for training and evaluating agents to develop reasoning and planning abilities for complex projects.
Continuing advances in frontier model research are paving the way for widespread deployment of AI agents. Meanwhile, global interest in building large, complex systems in software, manufacturing, energy and logistics has never been greater. Although AI driven system engineering holds tremendous promise, the static benchmarks dominating agent evaluations today fail to capture the crucial skills required for implementing dynamic systems, such as managing uncertain trade-offs and ensuring proactive adaptability. This position paper advocates for training and evaluating AI agents' system engineering abilities through automation-oriented sandbox games-particularly Factorio. By directing research efforts in this direction, we can equip AI agents with the specialized reasoning and long-horizon planning necessary to design, maintain, and optimize tomorrow's most demanding engineering projects.