EvaluatAR: A Cross-Device Evaluation Framework for Rapid Prototyping of Bystander PETs in AR
For AR researchers and developers, EvaluatAR reduces the high overhead and device-specificity of PET evaluation, enabling reproducible cross-device testing.
EvaluatAR is a cross-device evaluation framework for rapid prototyping of bystander privacy-enhancing technologies (PETs) in AR, enabling controlled replication of experimental conditions via a record-replay workflow. Validated on three AR headsets across implicit and explicit PETs, it reveals device-specific trade-offs and improves over state-of-the-art baselines.
Augmented Reality (AR) headsets continuously sense their surroundings, capturing nearby bystanders and raising privacy risks. Visual bystander privacy-enhancing technologies (PETs) mitigate this risk by detecting bystanders in egocentric scene views and applying privacy transformations (e.g., obfuscation). However, traditional PET evaluation is human-dependent, high-overhead, and device-specific, making it difficult to reproduce across devices. We present EvaluatAR, a cross-device evaluation framework for rapid prototyping at the early stage of PET evaluation. Our framework enables controlled replication of experimental conditions by standardizing PET inputs (sensor data and visual stimuli) and outputs through a record-replay workflow. We validate EvaluatAR through three case studies on HoloLens 2, Magic Leap 2, and Meta Quest 3 across implicit (continuous, context-driven) and explicit (intent-driven) PETs: (1) cross-device replay of inputs to a PET to reveal device-specific privacy-performance trade-offs; (2) generalizability of the same framework workflow across implicit and explicit PET design categories; and (3) replay of privacy-relevant edge cases to diagnose failures and validate PET modifications, yielding an improvement over the state-of-the-art baseline. These results demonstrate EvaluatAR's support for rapid, iterative PET development to advance reproducible cross-device evaluation of bystander PETs at a critical moment in the emergence of ubiquitous AR.