SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework
This addresses the labor-intensive annotation challenge in neuroscience and neuroethology for studying animal social behaviors, representing an incremental improvement by applying semi-supervised learning to an existing task.
The paper tackles the problem of costly manual annotation for multi-animal pose estimation by proposing a semi-supervised framework that leverages unlabeled video frames to enhance training, achieving superior results compared to state-of-the-art baselines on three animal experiments.
Multi-animal pose estimation is essential for studying animals' social behaviors in neuroscience and neuroethology. Advanced approaches have been proposed to support multi-animal estimation and achieve state-of-the-art performance. However, these models rarely exploit unlabeled data during training even though real world applications have exponentially more unlabeled frames than labeled frames. Manually adding dense annotations for a large number of images or videos is costly and labor-intensive, especially for multiple instances. Given these deficiencies, we propose a novel semi-supervised architecture for multi-animal pose estimation, leveraging the abundant structures pervasive in unlabeled frames in behavior videos to enhance training, which is critical for sparsely-labeled problems. The resulting algorithm will provide superior multi-animal pose estimation results on three animal experiments compared to the state-of-the-art baseline and exhibits more predictive power in sparsely-labeled data regimes.