Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning
For researchers and practitioners in low-data regimes (e.g., medical imaging), this work provides a theoretical framework and diagnostic tools (truncated Mahalanobis energy, K(N)) to understand and mitigate spectral collapse, though the proposed zeta-based filtering is not empirically validated.
The paper identifies spectral collapse—where finite-sample noise corrupts the embedding covariance and collapses the eigengap—as a fundamental bottleneck in low-data vision learning, and shows that multimodal training stabilizes the eigengap, increasing the number of recoverable modes and improving data efficiency. Across MNIST and neuroimaging, multimodal models maintain more stable modes and better class separation even when unimodal few-shot accuracy is comparable.
Deep vision models degrade sharply in low-data regimes, particularly in medical imaging where labeled samples are scarce. We show this arises not merely from overfitting but from a geometric failure: finite-sample noise corrupts the embedding covariance, collapsing the eigengap and limiting the number of recoverable signal-bearing modes. We develop a spectral theory of finite-sample representation learning that quantifies the recoverable dimension K(N), the number of eigenmodes that can be stably estimated from N samples. Using perturbation theory and concentration bounds, we show that only modes with eigenvalues above the noise floor $\|\hatΣ - Σ\|_{\mathrm{op}} \sim \sqrt{D/N}$ are reliable, yielding a truncated Mahalanobis energy that governs classification performance. Under a power-law spectral model, this energy can be approximated by a truncated Riemann zeta function, linking eigenvalue decay to data efficiency and AUC. Within this framework, multimodal learning acts as spectral stabilization: vision-language models impose low-rank constraints that suppress noise-dominated directions and preserve the eigengap, increasing K(N) under data scarcity. Across MNIST and multi-disease neuroimaging, we show that multimodal training maintains more stable modes and improves class separation, even when unimodal models achieve comparable few-shot accuracy. These results identify spectral collapse as a fundamental bottleneck in low-data learning. We use truncated Mahalanobis energy and K(N) to diagnose encoder quality, and introduce zeta-based spectral filtering as a principled approach to improve data efficiency.