Optimal Nonlinearities Improve Generalization Performance of Random Features
This work addresses the challenge of selecting activation functions for better generalization in supervised learning, offering a method to optimize nonlinearities, though it is incremental as it builds on existing random feature models.
The authors tackled the problem of improving generalization performance in random feature models by deriving optimal nonlinear activation functions from an equivalent Gaussian model, achieving better results than ReLU on synthetic and real datasets like CIFAR10 and mitigating the double descent phenomenon.
Random feature model with a nonlinear activation function has been shown to perform asymptotically equivalent to a Gaussian model in terms of training and generalization errors. Analysis of the equivalent model reveals an important yet not fully understood role played by the activation function. To address this issue, we study the "parameters" of the equivalent model to achieve improved generalization performance for a given supervised learning problem. We show that acquired parameters from the Gaussian model enable us to define a set of optimal nonlinearities. We provide two example classes from this set, e.g., second-order polynomial and piecewise linear functions. These functions are optimized to improve generalization performance regardless of the actual form. We experiment with regression and classification problems, including synthetic and real (e.g., CIFAR10) data. Our numerical results validate that the optimized nonlinearities achieve better generalization performance than widely-used nonlinear functions such as ReLU. Furthermore, we illustrate that the proposed nonlinearities also mitigate the so-called double descent phenomenon, which is known as the non-monotonic generalization performance regarding the sample size and the model size.