Diagnosing Generalization Failures from Representational Geometry Markers
This work addresses the problem of anticipating unseen failures in AI deployment for researchers and practitioners, offering a novel diagnostic tool that is more robust than existing methods, though it is incremental in building on prior geometry-based analyses.
The paper tackles the challenge of predicting generalization failures in machine learning models by proposing a top-down approach using representational geometry markers, finding that reductions in effective manifold dimensionality and utility in in-distribution data consistently forecast poor out-of-distribution performance across diverse settings, with these geometric patterns predicting OOD transfer performance more reliably than in-distribution accuracy.
Generalization, the ability to perform well beyond the training context, is a hallmark of biological and artificial intelligence, yet anticipating unseen failures remains a central challenge. Conventional approaches often take a ``bottom-up'' mechanistic route by reverse-engineering interpretable features or circuits to build explanatory models. While insightful, these methods often struggle to provide the high-level, predictive signals for anticipating failure in real-world deployment. Here, we propose using a ``top-down'' approach to studying generalization failures inspired by medical biomarkers: identifying system-level measurements that serve as robust indicators of a model's future performance. Rather than mapping out detailed internal mechanisms, we systematically design and test network markers to probe structure, function links, identify prognostic indicators, and validate predictions in real-world settings. In image classification, we find that task-relevant geometric properties of in-distribution (ID) object manifolds consistently forecast poor out-of-distribution (OOD) generalization. In particular, reductions in two geometric measures, effective manifold dimensionality and utility, predict weaker OOD performance across diverse architectures, optimizers, and datasets. We apply this finding to transfer learning with ImageNet-pretrained models. We consistently find that the same geometric patterns predict OOD transfer performance more reliably than ID accuracy. This work demonstrates that representational geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and AI interpretability.