An uncertainty-aware framework for data-efficient multi-view animal pose estimation
This work addresses a practical problem for researchers in animal behavior science by providing a more efficient and uncertainty-aware pose estimation system, though it is incremental as it builds on existing techniques like transformers and Ensemble Kalman Smoothers.
The paper tackles the challenge of accurate multi-view animal pose estimation with limited labeled data and poor uncertainty estimates by introducing a comprehensive framework that combines novel training, post-processing, and model distillation techniques. The result is a system that consistently outperforms existing methods across three animal species, enabling reliable pose estimation for behavioral analysis under data constraints.
Multi-view pose estimation is essential for quantifying animal behavior in scientific research, yet current methods struggle to achieve accurate tracking with limited labeled data and suffer from poor uncertainty estimates. We address these challenges with a comprehensive framework combining novel training and post-processing techniques, and a model distillation procedure that leverages the strengths of these techniques to produce a more efficient and effective pose estimator. Our multi-view transformer (MVT) utilizes pretrained backbones and enables simultaneous processing of information across all views, while a novel patch masking scheme learns robust cross-view correspondences without camera calibration. For calibrated setups, we incorporate geometric consistency through 3D augmentation and a triangulation loss. We extend the existing Ensemble Kalman Smoother (EKS) post-processor to the nonlinear case and enhance uncertainty quantification via a variance inflation technique. Finally, to leverage the scaling properties of the MVT, we design a distillation procedure that exploits improved EKS predictions and uncertainty estimates to generate high-quality pseudo-labels, thereby reducing dependence on manual labels. Our framework components consistently outperform existing methods across three diverse animal species (flies, mice, chickadees), with each component contributing complementary benefits. The result is a practical, uncertainty-aware system for reliable pose estimation that enables downstream behavioral analyses under real-world data constraints.