Deep Learning without Global Optimization by Random Fourier Neural Networks
This addresses the challenge of avoiding computationally expensive global optimization in deep learning, though it appears incremental as it builds on existing neural network architectures with a new training approach.
The paper tackles the problem of training deep neural networks without global optimization by introducing a training algorithm using random complex exponential activation functions and MCMC sampling, achieving the theoretical approximation rate for residual networks and enabling efficient learning of multiscale features without Gibbs phenomena.
We introduce a new training algorithm for deep neural networks that utilize random complex exponential activation functions. Our approach employs a Markov Chain Monte Carlo sampling procedure to iteratively train network layers, avoiding global and gradient-based optimization while maintaining error control. It consistently attains the theoretical approximation rate for residual networks with complex exponential activation functions, determined by network complexity. Additionally, it enables efficient learning of multiscale and high-frequency features, producing interpretable parameter distributions. Despite using sinusoidal basis functions, we do not observe Gibbs phenomena in approximating discontinuous target functions.