Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexity
This work addresses the challenge of interpreting feature learning mechanisms in deep learning theory, offering a more accessible approach for researchers, though it is incremental as it builds on existing theories.
The paper tackles the analytical complexity of predicting feature learning in deep networks by proposing a simpler scaling analysis heuristic, which reproduces known scaling exponents and makes novel predictions for complex architectures like three-layer non-linear networks and attention heads.
Two pressing topics in the theory of deep learning are the interpretation of feature learning mechanisms and the determination of implicit bias of networks in the rich regime. Current theories of rich feature learning effects revolve around networks with one or two trainable layers or deep linear networks. Furthermore, even under such limiting settings, predictions often appear in the form of high-dimensional non-linear equations, which require computationally intensive numerical solutions. Given the many details that go into defining a deep learning problem, this analytical complexity is a significant and often unavoidable challenge. Here, we propose a powerful heuristic route for predicting the data and width scales at which various patterns of feature learning emerge. This form of scale analysis is considerably simpler than such exact theories and reproduces the scaling exponents of various known results. In addition, we make novel predictions on complex toy architectures, such as three-layer non-linear networks and attention heads, thus extending the scope of first-principle theories of deep learning.