Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency
This work addresses data heterogeneity and sample imbalance issues in machine learning, offering a method to enhance model robustness in federated and long-tailed settings, though it is incremental as it builds on existing foundation models.
The paper tackles the problem of biased training distributions in deep learning by leveraging the geometric consistency of vision foundation model features across domains to calibrate distributions, achieving improved performance in federated learning and long-tailed recognition benchmarks.
Despite the fast progress of deep learning, one standing challenge is the gap of the observed training samples and the underlying true distribution. There are multiple reasons for the causing of this gap e.g. sampling bias, noise etc. In the era of foundation models, we show that when leveraging the off-the-shelf (vision) foundation models (e.g., CLIP, DINOv2) for feature extraction, the geometric shapes of the resulting feature distributions exhibit remarkable transferability across domains and datasets. To verify its practical usefulness, we embody our geometric knowledge-guided distribution calibration framework in two popular and challenging settings: federated learning and long-tailed recognition. In the federated setting, we devise a technique of acquiring the global geometric shape under privacy constraints, then leverage this knowledge to generate new samples for clients, in the aim of bridging the gap between local and global observations. In long-tailed learning, it utilizes the geometric knowledge transferred from sample-rich categories to recover the true distribution for sample-scarce tail classes. Comprehensive experiments show that our proposed geometric knowledge-guided distribution calibration effectively overcomes information deficits caused by data heterogeneity and sample imbalance, with boosted performance across benchmarks.