Charged Point Normalization: An Efficient Solution to the Saddle Point Problem
This addresses optimization challenges in deep learning by providing an efficient solution to saddle points, which is an incremental improvement over existing methods.
The paper tackles the saddle point problem in high-dimensional non-convex optimization by introducing Charged Point Normalization (CPN), a dynamic normalization method that forces escape from saddle points without using second-order information, and it drastically improves learning in deep neural networks across various datasets compared to non-CPN networks.
Recently, the problem of local minima in very high dimensional non-convex optimization has been challenged and the problem of saddle points has been introduced. This paper introduces a dynamic type of normalization that forces the system to escape saddle points. Unlike other saddle point escaping algorithms, second order information is not utilized, and the system can be trained with an arbitrary gradient descent learner. The system drastically improves learning in a range of deep neural networks on various data-sets in comparison to non-CPN neural networks.