Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks
This addresses the robustness of CNNs to noise in image data, which is incremental as it builds on existing work on network and data ranks.
The paper tackles the problem of how convolutional neural networks (CNNs) trained with gradient descent learn the intrinsic dimension of data despite background noise, showing that the rank of the trained CNN is far less affected by noise compared to the data rank, with theoretical proofs and experimental validation.
Modern neural networks are usually highly over-parameterized. Behind the wide usage of over-parameterized networks is the belief that, if the data are simple, then the trained network will be automatically equivalent to a simple predictor. Following this intuition, many existing works have studied different notions of "ranks" of neural networks and their relation to the rank of data. In this work, we study the rank of convolutional neural networks (CNNs) trained by gradient descent, with a specific focus on the robustness of the rank to image background noises. Specifically, we point out that, when adding background noises to images, the rank of the CNN trained with gradient descent is affected far less compared with the rank of the data. We support our claim with a theoretical case study, where we consider a particular data model to characterize low-rank clean images with added background noises. We prove that CNNs trained by gradient descent can learn the intrinsic dimension of clean images, despite the presence of relatively large background noises. We also conduct experiments on synthetic and real datasets to further validate our claim.