AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Results
This work addresses the problem of optimizing LLM deployment for XR developers and researchers, though it is incremental as it provides a benchmarking framework rather than a new method.
The paper tackles the challenge of selecting appropriate large language models (LLMs) and devices for on-device inference in extended reality (XR) by presenting AIvaluateXR, an evaluation framework that benchmarks 17 LLMs across four XR platforms, measuring metrics like performance consistency, speed, memory usage, and battery consumption, with results showing trade-offs for real-time applications and comparisons to client-server and cloud setups.
The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance the field of human-AI interaction. In the case of direct, on-device model inference, selecting the appropriate model and device for specific tasks remains challenging. In this paper, we present AIvaluateXR, a comprehensive evaluation framework for benchmarking LLMs running on XR devices. To demonstrate the framework, we deploy 17 selected LLMs across four XR platforms: Magic Leap 2, Meta Quest 3, Vivo X100s Pro, and Apple Vision Pro, and conduct an extensive evaluation. Our experimental setup measures four key metrics: performance consistency, processing speed, memory usage, and battery consumption. For each of the 68 model-device pairs, we assess performance under varying string lengths, batch sizes, and thread counts, analyzing the trade-offs for real-time XR applications. We propose a unified evaluation method based on the 3D Pareto Optimality theory to select the optimal device-model pairs from quality and speed objectives. Additionally, we compare the efficiency of on-device LLMs with client-server and cloud-based setups, and evaluate their accuracy on two interactive tasks. We believe our findings offer valuable insight to guide future optimization efforts for LLM deployment on XR devices. Our evaluation method can be used as standard groundwork for further research and development in this emerging field. The source code and supplementary materials are available at: www.nanovis.org/AIvaluateXR.html