MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution
This work addresses the critical privacy problem of GUI agents inadvertently exposing sensitive information in screenshots, which is particularly important for users and organizations deploying such agents in private or workplace settings.
MaskClaw is an edge-side privacy arbitrator for GUI agents that decides whether to allow, mask, or ask before raw screenshots leave a trusted environment. It achieves this by extracting local visual evidence and retrieving user- and task-specific policy memory, outperforming static PII detectors and cloud-side VLM reasoning in preventing over-confirmation, over-masking, or raw screen exposure.
GUI agents rely on screenshots to infer intent and operate across applications, but these screenshots often contain private messages, medical records, payment credentials, and workplace-specific workflows. Privacy decisions in this setting depend on task, recipient, application state, and user role, yet static PII detectors miss these boundaries and cloud-side VLM reasoning can upload the raw screen before deciding what should be protected. We present MaskClaw, an edge-side privacy arbitrator for GUI agents. MaskClaw extracts local visual evidence, retrieves user- and task-specific policy memory, and decides Allow, Mask, or Ask before raw screenshots leave a trusted user- or organization-controlled environment. In five designed skill-evolution scenarios, it turns corrections, cancellations, and edits into reusable privacy skills checked by a sandbox gate. We introduce P-GUI-Evo, a benchmark built from real UI patterns, reconstructed HTML screens, and sanitized labels. Experiments show that pattern matching, cloud reasoning, and routing alone tend to over-confirm, over-mask, or expose raw screenshots under the same protocol. The artifact is available at https://github.com/Theodora-Y/MaskClaw.