Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum
This addresses the need for task-specific control over neural network training dynamics, offering a computationally efficient method for practitioners.
The authors tackled the problem of modifying the inductive bias in wide neural networks to improve training efficiency, achieving polynomial to exponential speedups in convergence without altering the final solution.
Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired eigenvalues for which no closed form is known. We leverage the duality between wide neural networks and Neural Tangent Kernels and propose a preconditioned gradient descent method, which alters the trajectory of GD. As a result, this allows for a polynomial and, in some cases, exponential training speedup without changing the final solution. Our method is both computationally efficient and simple to implement.