Noise-in, Bias-out: Balanced and Real-time MoCap Solving
This work addresses the challenge of making motion capture more accessible and reliable for applications like animation or gaming, though it appears incremental by building on existing methods.
The paper tackles the problem of real-time optical motion capture using sparse, affordable sensors by applying machine learning to solve noisy marker estimates, achieving robust performance through techniques for imbalanced regression and uncertainty-aware inverse kinematics.
Real-time optical Motion Capture (MoCap) systems have not benefited from the advances in modern data-driven modeling. In this work we apply machine learning to solve noisy unstructured marker estimates in real-time and deliver robust marker-based MoCap even when using sparse affordable sensors. To achieve this we focus on a number of challenges related to model training, namely the sourcing of training data and their long-tailed distribution. Leveraging representation learning we design a technique for imbalanced regression that requires no additional data or labels and improves the performance of our model in rare and challenging poses. By relying on a unified representation, we show that training such a model is not bound to high-end MoCap training data acquisition, and exploit the advances in marker-less MoCap to acquire the necessary data. Finally, we take a step towards richer and affordable MoCap by adapting a body model-based inverse kinematics solution to account for measurement and inference uncertainty, further improving performance and robustness. Project page: https://moverseai.github.io/noise-tail