MobileDev-Bench: A Comprehensive Benchmark for Evaluating Language Models on Mobile Application Development
This addresses the problem of evaluating AI models for mobile development, which is crucial for developers and researchers, though it is incremental as it extends benchmarking to a new domain.
The authors tackled the lack of benchmarks for evaluating language models in mobile app development by introducing MobileDev-Bench, a comprehensive benchmark with 384 real-world tasks, and found that state-of-the-art models achieved low resolution rates of 3.39%-5.21%, highlighting significant performance gaps.
Large language models (LLMs) have shown strong performance on automated software engineering tasks, yet existing benchmarks focus primarily on general-purpose libraries or web applications, leaving mobile application development largely unexplored despite its strict platform constraints, framework-driven lifecycles, and complex platform API interactions. We introduce MobileDev-Bench, a benchmark comprising 384 real-world issue-resolution tasks collected from 18 production mobile applications spanning Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart). Each task pairs an authentic developer-reported issue with executable test patches, enabling fully automated validation of model-generated fixes within mobile build environments. The benchmark exhibits substantial patch complexity: fixes modify 12.5 files and 324.9 lines on average, and 35.7% of instances require coordinated changes across multiple artifact types, such as source and manifest files. Evaluation of four state-of-the-art code-capable LLMs, GPT- 5.2, Claude Sonnet 4.5, Gemini Flash 2.5, and Qwen3-Coder, yields low end-to-end resolution rates of 3.39%-5.21%, revealing significant performance gaps compared to prior benchmarks. Further analysis reveals systematic failure modes, with fault localization across multi-file and multi-artifact changes emerging as the primary bottleneck.