Why Do We Need Weight Decay in Modern Deep Learning?
This work clarifies a fundamental technique in deep learning, providing insights for practitioners training models from ResNets to LLMs, though it is incremental in refining existing understanding.
The paper investigates the role of weight decay in modern deep learning, showing that it enhances optimization dynamics rather than acting as an explicit regularizer, leading to improved training stability and lower training loss in vision tasks and large language models.
Weight decay is a broadly used technique for training state-of-the-art deep networks from image classification to large language models. Despite its widespread usage and being extensively studied in the classical literature, its role remains poorly understood for deep learning. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For deep networks on vision tasks trained with multipass SGD, we show how weight decay modifies the optimization dynamics enhancing the ever-present implicit regularization of SGD via the loss stabilization mechanism. In contrast, for large language models trained with nearly one-epoch training, we describe how weight decay balances the bias-variance tradeoff in stochastic optimization leading to lower training loss and improved training stability. Overall, we present a unifying perspective from ResNets on vision tasks to LLMs: weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way. The code is available at https://github.com/tml-epfl/why-weight-decay