XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse
This work targets the development of ML systems for XR and metaverse use cases, providing a foundational tool for benchmarking, but it is incremental as it builds on existing ML benchmarking concepts.
The paper introduces XRBench, a benchmark suite for evaluating machine learning hardware performance in extended reality (XR) applications, addressing real-time multi-task multi-model workloads with unique constraints like heterogeneity and concurrency.
Real-time multi-task multi-model (MTMM) workloads, a new form of deep learning inference workloads, are emerging for applications areas like extended reality (XR) to support metaverse use cases. These workloads combine user interactivity with computationally complex machine learning (ML) activities. Compared to standard ML applications, these ML workloads present unique difficulties and constraints. Real-time MTMM workloads impose heterogeneity and concurrency requirements on future ML systems and devices, necessitating the development of new capabilities. This paper begins with a discussion of the various characteristics of these real-time MTMM ML workloads and presents an ontology for evaluating the performance of future ML hardware for XR systems. Next, we present XRBENCH, a collection of MTMM ML tasks, models, and usage scenarios that execute these models in three representative ways: cascaded, concurrent, and cascaded-concurrent for XR use cases. Finally, we emphasize the need for new metrics that capture the requirements properly. We hope that our work will stimulate research and lead to the development of a new generation of ML systems for XR use cases. XRBench is available as an open-source project: https://github.com/XRBench