Deep Manifold Part 1: Anatomy of Neural Network Manifold
This foundational work aims to provide a theoretical understanding of neural networks for researchers in machine learning, though it appears incremental in building on existing manifold principles.
The paper tackles the problem of understanding the mathematical structure of neural networks by developing a Deep Manifold framework, revealing properties such as near infinite degrees of freedom and exponential learning capacity with depth.
Based on the numerical manifold method principle, we developed a mathematical framework of a neural network manifold: Deep Manifold and discovered that neural networks: 1) is numerical computation combining forward and inverse; 2) have near infinite degrees of freedom; 3) exponential learning capacity with depth; 4) have self-progressing boundary conditions; 5) has training hidden bottleneck. We also define two concepts: neural network learning space and deep manifold space and introduce two concepts: neural network intrinsic pathway and fixed point. We raise three fundamental questions: 1). What is the training completion definition; 2). where is the deep learning convergence point (neural network fixed point); 3). How important is token timestamp in training data given negative time is critical in inverse problem.