Improving Model Training by Periodic Sampling over Weight Distributions
This provides a general technique for enhancing training efficiency across various vision problems, though it appears incremental as it builds on existing optimization methods.
The paper tackles the problem of improving convergence in gradient-based optimization methods for vision tasks by introducing periodic sampling of model weights, resulting in faster, more robust training with only slight computational overhead.
In this paper, we explore techniques centered around periodic sampling of model weights that provide convergence improvements on gradient update methods (vanilla \acs{SGD}, Momentum, Adam) for a variety of vision problems (classification, detection, segmentation). Importantly, our algorithms provide better, faster and more robust convergence and training performance with only a slight increase in computation time. Our techniques are independent of the neural network model, gradient optimization methods or existing optimal training policies and converge in a less volatile fashion with performance improvements that are approximately monotonic. We conduct a variety of experiments to quantify these improvements and identify scenarios where these techniques could be more useful.