Towards Understanding the Optimization Mechanisms in Deep Learning
This provides theoretical insights into mechanisms like over-parameterization for researchers in machine learning, though it is incremental.
The paper tackles the problem of understanding optimization in deep learning by showing that global optimal solutions can be approximated by minimizing gradient norm and structural error, validated through empirical results.
In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model's parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness.