Memorizing Long-tail Data Can Help Generalization Through Composition
This addresses generalization challenges in deep learning for scenarios with rare data combinations, though it is incremental as it builds on existing ideas about memorization and composition.
The paper tackles the problem of generalization on rare test examples requiring combinations of long-tailed features, showing theoretically and experimentally that memorizing long-tail data combined with composition enables correct predictions on unseen combinations.
Deep learning has led researchers to rethink the relationship between memorization and generalization. In many settings, memorization does not hurt generalization due to implicit regularization and may help by memorizing long-tailed examples. In this paper, we consider the synergy between memorization and simple composition -- the ability to make correct prediction on a combination of long-tailed features. Theoretically, we show that for a linear setting, memorization together with composition can help the model make correct predictions on rare test examples that require a combination of long-tailed features, even if such combinations were never observed in the training data. Experiments on neural network architecture on simple data show that the theoretical insight extends beyond the linear setting, and we further observe that the composition capability of the model depends on its architecture.