An Analysis of Optimizer Choice on Energy Efficiency and Performance in Neural Network Training
This provides actionable insights for practitioners seeking to balance performance and sustainability in machine learning workflows, though it is incremental as it applies existing methods to new data.
This paper empirically analyzed how optimizer choice affects energy efficiency and performance in neural network training, finding substantial trade-offs between training speed, accuracy, and environmental impact across datasets, with AdamW and NAdam identified as consistently efficient choices while SGD performed better on complex datasets despite higher emissions.
As machine learning models grow increasingly complex and computationally demanding, understanding the environmental impact of training decisions becomes critical for sustainable AI development. This paper presents a comprehensive empirical study investigating the relationship between optimizer choice and energy efficiency in neural network training. We conducted 360 controlled experiments across three benchmark datasets (MNIST, CIFAR-10, CIFAR-100) using eight popular optimizers (SGD, Adam, AdamW, RMSprop, Adagrad, Adadelta, Adamax, NAdam) with 15 random seeds each. Using CodeCarbon for precise energy tracking on Apple M1 Pro hardware, we measured training duration, peak memory usage, carbon dioxide emissions, and final model performance. Our findings reveal substantial trade-offs between training speed, accuracy, and environmental impact that vary across datasets and model complexity. We identify AdamW and NAdam as consistently efficient choices, while SGD demonstrates superior performance on complex datasets despite higher emissions. These results provide actionable insights for practitioners seeking to balance performance and sustainability in machine learning workflows.