Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates
This addresses the problem of costly benchmarking for researchers and practitioners, but it is incremental as it builds on existing cyclic learning rate techniques.
The paper tackles the computational expense of benchmarking accuracy vs. training time by using a multiplicative cyclic learning rate schedule to construct tradeoff curves in a single training run, generating curves for methods like Blurpool and MixUp to evaluate algorithmic effects on efficiency.
Benchmarking the tradeoff between neural network accuracy and training time is computationally expensive. Here we show how a multiplicative cyclic learning rate schedule can be used to construct a tradeoff curve in a single training run. We generate cyclic tradeoff curves for combinations of training methods such as Blurpool, Channels Last, Label Smoothing and MixUp, and highlight how these cyclic tradeoff curves can be used to evaluate the effects of algorithmic choices on network training efficiency.