Testing predictions of representation cost theory with CNNs
This work addresses a fundamental issue in model robustness for machine learning researchers, but it is incremental as it applies existing frequency-space representations to a new domain.
The paper tackles the problem of understanding why convolutional neural networks (CNNs) are sensitive to low-frequency signals, showing through theory and experiments that this sensitivity arises from the frequency distribution of natural images, which concentrates power in low-to-mid frequencies.
It is widely acknowledged that trained convolutional neural networks (CNNs) have different levels of sensitivity to signals of different frequency. In particular, a number of empirical studies have documented CNNs sensitivity to low-frequency signals. In this work we show with theory and experiments that this observed sensitivity is a consequence of the frequency distribution of natural images, which is known to have most of its power concentrated in low-to-mid frequencies. Our theoretical analysis relies on representations of the layers of a CNN in frequency space, an idea that has previously been used to accelerate computations and study implicit bias of network training algorithms, but to the best of our knowledge has not been applied in the domain of model robustness.