Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
This work extends a known principle to broader practical settings, potentially improving efficiency in solving differential equations for computational science.
The study demonstrates that the Frequency Principle (F-Principle), where deep neural networks learn from low to high frequencies, holds for general loss functions like mean square error and cross entropy, and applies it to solve differential equations, showing potential for faster convergence compared to conventional methods like the Jacobi method.
Previous studies have shown that deep neural networks (DNNs) with common settings often capture target functions from low to high frequency, which is called Frequency Principle (F-Principle). It has also been shown that F-Principle can provide an understanding to the often observed good generalization ability of DNNs. However, previous studies focused on the loss function of mean square error, while various loss functions are used in practice. In this work, we show that the F-Principle holds for a general loss function (e.g., mean square error, cross entropy, etc.). In addition, DNN's F-Principle may be applied to develop numerical schemes for solving various problems which would benefit from a fast converging of low frequency. As an example of the potential usage of F-Principle, we apply DNN in solving differential equations, in which conventional methods (e.g., Jacobi method) is usually slow in solving problems due to the convergence from high to low frequency.