AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?
This work addresses a critical gap for AI researchers and software engineers by providing a benchmark to assess LLMs' readiness for full-scale software development, though it is incremental as it focuses on evaluation rather than a new method.
The authors tackled the problem of evaluating whether large language models can construct entire software systems from scratch, rather than just generating isolated functions, by proposing APPFORGE, a benchmark of 101 real-world Android app problems. Their evaluation showed that current models, including the best-performing GPT-5, achieved only 18.8% functionally correct applications, highlighting significant limitations in handling complex software engineering challenges.
Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world applications demand reasoning over the entire software system: developers must orchestrate how different components interact, maintain consistency across states over time, and ensure the application behaves correctly within the lifecycle and framework constraints. Yet, no existing benchmark adequately evaluates whether LLMs can bridge this gap and construct entire software systems from scratch. To address this gap, we propose APPFORGE, a benchmark consisting of 101 software development problems drawn from real-world Android apps. Given a natural language specification detailing the app functionality, a language model is tasked with implementing the functionality into an Android app from scratch. Developing an Android app from scratch requires understanding and coordinating app states, lifecycle management, and asynchronous operations, calling for LLMs to generate context-aware, robust, and maintainable code. To construct APPFORGE, we design a multi-agent system to automatically summarize the main functionalities from app documents and navigate the app to synthesize test cases validating the functional correctness of app implementation. Following rigorous manual verification by Android development experts, APPFORGE incorporates the test cases within an automated evaluation framework that enables reproducible assessment without human intervention, making it easily adoptable for future research. Our evaluation on 12 flagship LLMs show that all evaluated models achieve low effectiveness, with the best-performing model (GPT-5) developing only 18.8% functionally correct applications, highlighting fundamental limitations in current models' ability to handle complex, multi-component software engineering challenges.