Mobile AR Depth Estimation: Challenges & Prospects -- Extended Version
This work addresses the problem of enabling realistic user interactions in mobile AR for developers and users, but it is incremental as it focuses on analyzing existing methods rather than introducing new solutions.
The paper investigates challenges in achieving accurate metric depth estimation for mobile augmented reality using monocular depth estimation models, testing four state-of-the-art models on the ARKitScenes dataset and identifying hardware, data, and model-related challenges while proposing future directions.
Metric depth estimation plays an important role in mobile augmented reality (AR). With accurate metric depth, we can achieve more realistic user interactions such as object placement and occlusion detection. While specialized hardware like LiDAR demonstrates its promise, its restricted availability, i.e., only on selected high-end mobile devices, and performance limitations such as range and sensitivity to the environment, make it less ideal. Monocular depth estimation, on the other hand, relies solely on mobile cameras, which are ubiquitous, making it a promising alternative for mobile AR. In this paper, we investigate the challenges and opportunities of achieving accurate metric depth estimation in mobile AR. We tested four different state-of-the-art monocular depth estimation models on a newly introduced dataset (ARKitScenes) and identified three types of challenges: hard-ware, data, and model related challenges. Furthermore, our research provides promising future directions to explore and solve those challenges. These directions include (i) using more hardware-related information from the mobile device's camera and other available sensors, (ii) capturing high-quality data to reflect real-world AR scenarios, and (iii) designing a model architecture to utilize the new information.