Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Lifting
This work addresses theoretical issues in neural network optimization for researchers, but it is incremental as it builds on prior studies of critical points.
The paper tackles the problem of sample dependence in critical points of neural networks by introducing a sample-independent critical lifting operator, and demonstrates that previous critical embeddings miss some sample-independent points while proving that sample-dependent points and saddles exist for large sample sizes.
This paper investigates the sample dependence of critical points for neural networks. We introduce a sample-independent critical lifting operator that associates a parameter of one network with a set of parameters of another, thus defining sample-dependent and sample-independent lifted critical points. We then show by example that previously studied critical embeddings do not capture all sample-independent lifted critical points. Finally, we demonstrate the existence of sample-dependent lifted critical points for sufficiently large sample sizes and prove that saddles appear among them.