Data-Dependent Path Normalization in Neural Networks
This work addresses the challenge of improving neural network training efficiency and stability for machine learning practitioners, but it appears incremental as it builds on existing methods.
The authors tackled the problem of unifying neural network normalization and optimization methods by proposing a framework that includes Path-SGD and Batch-Normalization, investigating invariance, data dependence, and connections with natural gradients.
We propose a unified framework for neural net normalization, regularization and optimization, which includes Path-SGD and Batch-Normalization and interpolates between them across two different dimensions. Through this framework we investigate issue of invariance of the optimization, data dependence and the connection with natural gradients.