The Physics of Data and Tasks: Theories of Locality and Compositionality in Deep Learning
It addresses a foundational problem for the machine learning community, though it appears incremental as it builds on existing theories without new empirical validation.
The paper tackles the problem of understanding how deep neural networks learn high-dimensional tasks despite statistical intractability, by studying the roles of locality and compositionality in data and representations, but does not report concrete numerical results.
Deep neural networks have achieved remarkable success, yet our understanding of how they learn remains limited. These models can learn high-dimensional tasks, which is generally statistically intractable due to the curse of dimensionality. This apparent paradox suggests that learnable data must have an underlying latent structure. What is the nature of this structure? How do neural networks encode and exploit it, and how does it quantitatively impact performance - for instance, how does generalization improve with the number of training examples? This thesis addresses these questions by studying the roles of locality and compositionality in data, tasks, and deep learning representations.