Notes on Deep Learning Theory
This paper provides an educational resource for students and researchers interested in the theoretical underpinnings of deep learning.
This paper provides lecture notes from Fall 2020 at MIPT and YSDA, covering initialization, loss landscape, generalization, and neural tangent kernel theory. It serves as an introductory resource to these foundational aspects of deep learning theory.
These are the notes for the lectures that I was giving during Fall 2020 at the Moscow Institute of Physics and Technology (MIPT) and at the Yandex School of Data Analysis (YSDA). The notes cover some aspects of initialization, loss landscape, generalization, and a neural tangent kernel theory. While many other topics (e.g. expressivity, a mean-field theory, a double descent phenomenon) are missing in the current version, we plan to add them in future revisions.