Can we avoid Double Descent in Deep Neural Networks?
This addresses the problem of resource-efficient model selection for practitioners in deep learning, but it is incremental as it builds on existing observations without definitive solutions.
The paper investigates whether the double descent phenomenon in deep neural networks can be avoided to find optimal model sizes for generalization and efficiency, suggesting that proper conditioning and regularization, such as ℓ₂ regularization, may help mitigate it.
Finding the optimal size of deep learning models is very actual and of broad impact, especially in energy-saving schemes. Very recently, an unexpected phenomenon, the ``double descent'', has caught the attention of the deep learning community. As the model's size grows, the performance gets first worse, and then goes back to improving. It raises serious questions about the optimal model's size to maintain high generalization: the model needs to be sufficiently over-parametrized, but adding too many parameters wastes training resources. Is it possible to find, in an efficient way, the best trade-off? Our work shows that the double descent phenomenon is potentially avoidable with proper conditioning of the learning problem, but a final answer is yet to be found. We empirically observe that there is hope to dodge the double descent in complex scenarios with proper regularization, as a simple $\ell_2$ regularization is already positively contributing to such a perspective.