MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
This work is significant for users and developers of GUI agents, as it aims to improve the deployment and user experience by creating a single, more versatile agent capable of operating across mobile, web, and desktop environments, addressing the current limitation of domain-specific agents.
The paper addresses the challenge of consolidating domain-specific GUI agents into a single cross-environment policy. It introduces MAGA, a method that re-allocates training signals based on structured actions, suppressing unnecessary distillation signals and focusing on erroneous actions. MAGA achieves the highest mean success rate, outperforming the strongest baseline by 2.0% at 8B model scale and nearly matching the average performance of its teacher models.
Graphical user interface (GUI) agents based on large language models are increasingly deployed across mobile, web, and desktop environments. However, existing agents are typically domain-specific, limiting the deployment and user experience. This motivates the consolidation of specialized models into a single cross-environment policy. Weight merging directly merges domain-specific experts but can corrupt executable actions under expert disagreement, while on-policy distillation (OPD) avoids conflicting teacher supervision yet still treats all response tokens equally during distillation, ignoring that action tokens are the only interface between the environment and the agent. To address this, We introduce MAGA that re-allocates training signal according to the structured action. Based on the correctness of the generated action, it suppresses unnecessary or invalid distillation signals and focuses learning on erroneous actions. Besides, a training-only hint optimizes the supervision signal provided by domain-specific teachers without changing the student input. Across two model scales, MAGA achieves the highest mean success rate, outperforming the strongest baseline by 2.0% at 8B and achieves almost the same average performance with teachers.