Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study
This work addresses the bottleneck of biodiversity monitoring for ecologists by providing an efficient, scalable method for species-level clustering, though it is incremental as it applies existing models to a new domain.
This study tackled the problem of manual labeling of animal images in ecological research by investigating whether Vision Transformer models can cluster unlabeled images to species-level, achieving near-perfect clustering with a V-measure of 0.958 using DINOv3 embeddings and supervised methods, and competitive unsupervised performance at 0.943 with only 1.14% outliers.
Manual labeling of animal images remains a significant bottleneck in ecological research, limiting the scale and efficiency of biodiversity monitoring efforts. This study investigates whether state-of-the-art Vision Transformer (ViT) foundation models can reduce thousands of unlabeled animal images directly to species-level clusters. We present a comprehensive benchmarking framework evaluating five ViT models combined with five dimensionality reduction techniques and four clustering algorithms, two supervised and two unsupervised, across 60 species (30 mammals and 30 birds), with each test using a random subset of 200 validated images per species. We investigate when clustering succeeds at species-level, where it fails, and whether clustering within the species-level reveals ecologically meaningful patterns such as sex, age, or phenotypic variation. Our results demonstrate near-perfect species-level clustering (V-measure: 0.958) using DINOv3 embeddings with t-SNE and supervised hierarchical clustering methods. Unsupervised approaches achieve competitive performance (0.943) while requiring no prior species knowledge, rejecting only 1.14% of images as outliers requiring expert review. We further demonstrate robustness to realistic long-tailed distributions of species and show that intentional over-clustering can reliably extract intra-specific variation including age classes, sexual dimorphism, and pelage differences. We introduce an open-source benchmarking toolkit and provide recommendations for ecologists to select appropriate methods for sorting their specific taxonomic groups and data.