Adaptive Learning of the Optimal Batch Size of SGD
This work addresses a practical bottleneck in machine learning optimization by providing a method to adaptively set batch sizes, which is incremental but improves upon theoretical formulas that are not directly applicable.
The paper tackles the problem of determining the optimal batch size for Stochastic Gradient Descent (SGD) in practice, by developing an adaptive method that learns this size during training for strongly convex and smooth functions, achieving nearly optimal performance as if the optimal batch size were known in advance.
Recent advances in the theoretical understanding of SGD led to a formula for the optimal batch size minimizing the number of effective data passes, i.e., the number of iterations times the batch size. However, this formula is of no practical value as it depends on the knowledge of the variance of the stochastic gradients evaluated at the optimum. In this paper we design a practical SGD method capable of learning the optimal batch size adaptively throughout its iterations for strongly convex and smooth functions. Our method does this provably, and in our experiments with synthetic and real data robustly exhibits nearly optimal behaviour; that is, it works as if the optimal batch size was known a-priori. Further, we generalize our method to several new batch strategies not considered in the literature before, including a sampling suitable for distributed implementations.