A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern
This foundational result explains a general property of deep learning models, impacting theoretical understanding for researchers.
The paper proves that deep neural networks universally contain arbitrary low-dimensional patterns in their loss surfaces, a property that holds across datasets and architectures, and predicts empirical observations like ease of finding and transferability.
The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks. In this paper we prove that: (i) this is a general property of deep universal approximators; and (ii) this property holds for arbitrary smooth patterns, for other dimensionalities, for every dataset, and any neural network that is sufficiently deep and wide. Our analysis predicts not only the existence of all such low-dimensional patterns, but also two other properties that were observed empirically: (i) that it is easy to find these patterns; and (ii) that they transfer to other data-sets (e.g. a test-set).