The Phong Surface: Efficient 3D Model Fitting using Lifted Optimization
This addresses the need for low-latency tracking in mixed reality applications on devices with limited computational budgets, representing an incremental improvement in optimization methods.
The paper tackled the problem of real-time 3D model fitting for hand tracking on resource-constrained hardware like HoloLens 2, introducing the Phong surface to enable lifted optimization and achieving significant efficiency gains over ICP-based methods.
Realtime perceptual and interaction capabilities in mixed reality require a range of 3D tracking problems to be solved at low latency on resource-constrained hardware such as head-mounted devices. Indeed, for devices such as HoloLens 2 where the CPU and GPU are left available for applications, multiple tracking subsystems are required to run on a continuous, real-time basis while sharing a single Digital Signal Processor. To solve model-fitting problems for HoloLens 2 hand tracking, where the computational budget is approximately 100 times smaller than an iPhone 7, we introduce a new surface model: the `Phong surface'. Using ideas from computer graphics, the Phong surface describes the same 3D shape as a triangulated mesh model, but with continuous surface normals which enable the use of lifting-based optimization, providing significant efficiency gains over ICP-based methods. We show that Phong surfaces retain the convergence benefits of smoother surface models, while triangle meshes do not.