ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality
This addresses safety and performance issues for AR users by detecting harmful virtual content, but it is incremental as it applies existing VLMs to a new AR-specific problem.
The paper tackles the problem of task-detrimental virtual content in Augmented Reality, such as obstruction and information manipulation attacks, by introducing ViDDAR, a system that uses Vision Language Models to detect such content, achieving up to 92.15% accuracy for obstruction detection with 533 ms latency and 82.46% accuracy for information manipulation detection with 9.62 s latency.
In Augmented Reality (AR), virtual content enhances user experience by providing additional information. However, improperly positioned or designed virtual content can be detrimental to task performance, as it can impair users' ability to accurately interpret real-world information. In this paper we examine two types of task-detrimental virtual content: obstruction attacks, in which virtual content prevents users from seeing real-world objects, and information manipulation attacks, in which virtual content interferes with users' ability to accurately interpret real-world information. We provide a mathematical framework to characterize these attacks and create a custom open-source dataset for attack evaluation. To address these attacks, we introduce ViDDAR (Vision language model-based Task-Detrimental content Detector for Augmented Reality), a comprehensive full-reference system that leverages Vision Language Models (VLMs) and advanced deep learning techniques to monitor and evaluate virtual content in AR environments, employing a user-edge-cloud architecture to balance performance with low latency. To the best of our knowledge, ViDDAR is the first system to employ VLMs for detecting task-detrimental content in AR settings. Our evaluation results demonstrate that ViDDAR effectively understands complex scenes and detects task-detrimental content, achieving up to 92.15% obstruction detection accuracy with a detection latency of 533 ms, and an 82.46% information manipulation content detection accuracy with a latency of 9.62 s.