Obtaining Adjustable Regularization for Free via Iterate Averaging
This provides a method to avoid expensive hyperparameter tuning for regularization in machine learning, addressing a practical bottleneck for researchers and practitioners.
The paper tackles the problem of costly regularization parameter tuning in optimization by proposing an averaging scheme that converts SGD iterates into regularized solutions with adjustable regularization for strongly convex and smooth objectives, extending this to accelerated methods and showing empirical success on neural networks.
Regularization for optimization is a crucial technique to avoid overfitting in machine learning. In order to obtain the best performance, we usually train a model by tuning the regularization parameters. It becomes costly, however, when a single round of training takes significant amount of time. Very recently, Neu and Rosasco show that if we run stochastic gradient descent (SGD) on linear regression problems, then by averaging the SGD iterates properly, we obtain a regularized solution. It left open whether the same phenomenon can be achieved for other optimization problems and algorithms. In this paper, we establish an averaging scheme that provably converts the iterates of SGD on an arbitrary strongly convex and smooth objective function to its regularized counterpart with an adjustable regularization parameter. Our approaches can be used for accelerated and preconditioned optimization methods as well. We further show that the same methods work empirically on more general optimization objectives including neural networks. In sum, we obtain adjustable regularization for free for a large class of optimization problems and resolve an open question raised by Neu and Rosasco.