The Intrinsic Dimension of Images and Its Impact on Learning
This work provides empirical evidence for a foundational concept in computer vision, potentially guiding dataset design and model training, though it is incremental in confirming existing intuitions.
The paper investigates the low intrinsic dimension of natural image datasets and its impact on deep learning, finding that lower-dimensional datasets are easier for neural networks to learn and lead to better generalization, with validation using synthetic GAN-generated data.
It is widely believed that natural image data exhibits low-dimensional structure despite the high dimensionality of conventional pixel representations. This idea underlies a common intuition for the remarkable success of deep learning in computer vision. In this work, we apply dimension estimation tools to popular datasets and investigate the role of low-dimensional structure in deep learning. We find that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images. Additionally, we find that low dimensional datasets are easier for neural networks to learn, and models solving these tasks generalize better from training to test data. Along the way, we develop a technique for validating our dimension estimation tools on synthetic data generated by GANs allowing us to actively manipulate the intrinsic dimension by controlling the image generation process. Code for our experiments may be found here https://github.com/ppope/dimensions.