Neural Network Memorization Dissection
This work addresses the problem of understanding memorization in DNNs for machine learning researchers, providing insights into model behavior, but it is incremental as it builds on existing studies of memorization.
The paper investigates the differences between deep neural networks trained with true labels versus random labels, finding that DNNs prioritize simple patterns and exhibit 'One way to Learn and N ways to Memorize'.
Deep neural networks (DNNs) can easily fit a random labeling of the training data with zero training error. What is the difference between DNNs trained with random labels and the ones trained with true labels? Our paper answers this question with two contributions. First, we study the memorization properties of DNNs. Our empirical experiments shed light on how DNNs prioritize the learning of simple input patterns. In the second part, we propose to measure the similarity between what different DNNs have learned and memorized. With the proposed approach, we analyze and compare DNNs trained on data with true labels and random labels. The analysis shows that DNNs have \textit{One way to Learn} and \textit{N ways to Memorize}. We also use gradient information to gain an understanding of the analysis results.