Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon
This provides a phenomenon-driven approach that can enhance theoretical understanding and practical applications in deep learning, though it is incremental as it builds on known double descent concepts.
The study identified a 'double descent' phenomenon in the output discrepancy between identically-trained neural networks with different initializations, demonstrating its prevalence across tasks, datasets, and architectures, and used it to develop an early stopping criterion and data quality assessment method.
In this paper, we studied two identically-trained neural networks (i.e. networks with the same architecture, trained on the same dataset using the same algorithm, but with different initialization) and found that their outputs discrepancy on the training dataset exhibits a "double descent" phenomenon. We demonstrated through extensive experiments across various tasks, datasets, and network architectures that this phenomenon is prevalent. Leveraging this phenomenon, we proposed a new early stopping criterion and developed a new method for data quality assessment. Our results show that a phenomenon-driven approach can benefit deep learning research both in theoretical understanding and practical applications.