LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images
For ML practitioners needing to compare large datasets, LatentDiff provides a scalable and interpretable tool to detect subtle semantic shifts.
LatentDiff enables semantic dataset comparison in latent space, achieving superior accuracy and robustness to sparse distribution shifts where <1% of images differ, at a fraction of the computational cost of caption-based methods.
We present LatentDiff, a scalable framework for semantic dataset comparison that operates directly in the latent space of pretrained vision encoders. By combining sparse autoencoder-based divergence testing with density ratio estimation, LatentDiff identifies interpretable semantic differences between datasets at a fraction of the computational cost of caption-based alternatives. We also introduce Noisy-Diff, a benchmark capturing realistic sparse distribution shifts that cause existing methods to struggle. Experiments demonstrate that LatentDiff achieves superior accuracy while remaining robust to settings where an extremely small fraction of images (from 5% to <1% ) differ semantically.