MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
This work addresses the problem of inaccurate performance evaluation for mobile agents in real-world scenarios, particularly for researchers and developers, though it is incremental as it builds on existing benchmarking approaches.
The authors tackled the mismatch between existing mobile agent benchmarks and real-world usage by proposing MobiFlow, a framework that uses trajectory fusion to evaluate tasks from third-party applications, resulting in higher alignment with human assessments compared to AndroidWorld.
Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile-agent scenarios, however, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage and making it difficult to evaluate model performance accurately. To address these issues, we propose MobiFlow, an evaluation framework built on tasks drawn from arbitrary third-party applications. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiFlow can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiFlow covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments and can guide the training of future GUI-based models under real workloads.