Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation
This work provides insights into memorization during optimization, which is incremental for understanding neural network behavior in machine learning.
The paper studied memorization in over-parameterized deep neural networks using turn-over dropout to estimate influence, finding that optimization handles easy and difficult examples simultaneously with different speeds, and that difficult examples are more informative for real data.
Over-parameterized deep neural networks are able to achieve excellent training accuracy while maintaining a small generalization error. It has also been found that they are able to fit arbitrary labels, and this behaviour is referred to as the phenomenon of memorization. In this work, we study the phenomenon of memorization with turn-over dropout, an efficient method to estimate influence and memorization, for data with true labels (real data) and data with random labels (random data). Our main findings are: (i) For both real data and random data, the optimization of easy examples (e.g., real data) and difficult examples (e.g., random data) are conducted by the network simultaneously, with easy ones at a higher speed; (ii) For real data, a correct difficult example in the training dataset is more informative than an easy one. By showing the existence of memorization on random data and real data, we highlight the consistency between them regarding optimization and we emphasize the implication of memorization during optimization.