First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models
This work addresses the challenge of inefficient and intuitive perturbation design in ML training, offering a systematic optimization method that is transferable across various models and tasks, though it appears incremental as it builds on existing perturbation protocols.
The authors tackled the problem of ad hoc design of perturbations in machine learning training by framing training protocols as first-passage processes, enabling rational optimization of perturbations to improve training efficiency and generalization, as demonstrated with a ResNet-18 model on CIFAR-10 and transferability to other datasets and tasks.
Machine learning models have become indispensable tools in applications across the physical sciences. Their training is often time-consuming, vastly exceeding the inference timescales. Several protocols have been developed to perturb the learning process and improve the training, such as shrink and perturb, warm restarts, and stochastic resetting. For classifiers, these perturbations have been shown to result in enhanced speedups or improved generalization. However, the design of such perturbations is usually done ad hoc by intuition and trial and error. To rationally optimize training protocols, we frame them as first-passage processes and consider their response to perturbations. We show that if the unperturbed learning process reaches a quasi-steady state, the response at a single perturbation frequency can predict the behavior at a wide range of frequencies. We employ this approach to a CIFAR-10 classifier using the ResNet-18 model and identify a useful perturbation and frequency among several possibilities. We demonstrate the transferability of the approach to other datasets, architectures, optimizers and even tasks (regression instead of classification). Our work allows optimization of perturbations for improving the training of machine learning models using a first-passage approach.