Understanding GUI Agent Localization Biases through Logit Sharpness
This work addresses reliability issues in GUI agents for users of multimodal large language models, offering incremental improvements in interpretability and robustness.
The paper tackles the problem of systematic localization errors in GUI agents by introducing a fine-grained evaluation framework and a new metric, Peak Sharpness Score, to quantify uncertainty, and proposes a training-free technique, Context-Aware Cropping, that improves performance.
Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions. Despite their promising performance, these models frequently exhibit hallucinations-systematic localization errors that compromise reliability. We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics. To better quantify model uncertainty, we introduce the Peak Sharpness Score (PSS), a metric that evaluates the alignment between semantic continuity and logits distribution in coordinate prediction. Building on this insight, we further propose Context-Aware Cropping, a training-free technique that improves model performance by adaptively refining input context. Extensive experiments demonstrate that our framework and methods provide actionable insights and enhance the interpretability and robustness of GUI agent behavior.