ZebraPose: Zebra Detection and Pose Estimation using only Synthetic Data
This addresses the challenge of collecting and labeling real-world wild animal data for monitoring tasks, which is costly and impractical, especially for out-of-distribution viewpoints like aerial imagery.
The paper tackles the problem of detecting and estimating 2D poses of wild zebras by generating a fully synthetic dataset using a 3D photorealistic simulator, eliminating the need for real images or complex bridging strategies, and shows that models trained exclusively on this synthetic data generalize well to real images across multiple benchmarks.
Collecting and labeling large real-world wild animal datasets is impractical, costly, error-prone, and labor-intensive. For animal monitoring tasks, as detection, tracking, and pose estimation, out-of-distribution viewpoints (e.g. aerial) are also typically needed but rarely found in publicly available datasets. To solve this, existing approaches synthesize data with simplistic techniques that then necessitate strategies to bridge the synthetic-to-real gap. Therefore, real images, style constraints, complex animal models, or pre-trained networks are often leveraged. In contrast, we generate a fully synthetic dataset using a 3D photorealistic simulator and demonstrate that it can eliminate such needs for detecting and estimating 2D poses of wild zebras. Moreover, existing top-down 2D pose estimation approaches using synthetic data assume reliable detection models. However, these often fail in out-of-distribution scenarios, e.g. those that include wildlife or aerial imagery. Our method overcomes this by enabling the training of both tasks using the same synthetic dataset. Through extensive benchmarks, we show that models trained from scratch exclusively on our synthetic data generalize well to real images. We perform these using multiple real-world and synthetic datasets, pre-trained and randomly initialized backbones, and different image resolutions. Code, results, models, and data can be found athttps://zebrapose.is.tue.mpg.de/.