Frequency Principle in Deep Learning Beyond Gradient-descent-based Training
This work provides empirical evidence for the universality of the F-Principle, which is important for researchers studying the fundamental learning dynamics of deep neural networks.
This paper investigates the Frequency Principle (F-Principle) in deep neural networks, which states that DNNs fit target functions from low to high frequencies. It demonstrates that the F-Principle holds true even with non-gradient-descent-based training methods, including those with and without gradient information.
Frequency perspective recently makes progress in understanding deep learning. It has been widely verified in both empirical and theoretical studies that deep neural networks (DNNs) often fit the target function from low to high frequency, namely Frequency Principle (F-Principle). F-Principle sheds light on the strength and the weakness of DNNs and inspires a series of subsequent works, including theoretical studies, empirical studies and the design of efficient DNN structures etc. Previous works examine the F-Principle in gradient-descent-based training. It remains unclear whether gradient-descent-based training is a necessary condition for the F-Principle. In this paper, we show that the F-Principle exists stably in the training process of DNNs with non-gradient-descent-based training, including optimization algorithms with gradient information, such as conjugate gradient and BFGS, and algorithms without gradient information, such as Powell's method and Particle Swarm Optimization. These empirical studies show the universality of the F-Principle and provide hints for further study of F-Principle.