Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate
This addresses the problem of poor generalization in deep neural networks for researchers and practitioners, offering an incremental improvement through a novel regularization method.
The paper tackles the challenge of generalization in machine learning by proposing Learning from Teaching (LoT), a regularization technique that uses auxiliary student learners to help the main model capture more generalizable correlations, resulting in significant benefits across domains like Computer Vision, Natural Language Processing, and Reinforcement Learning.
Generalization remains a central challenge in machine learning. In this work, we propose Learning from Teaching (LoT), a novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability to capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners are trained by the main model and, in turn, provide feedback to help the main model capture more generalizable and imitable correlations. Our experimental results across several domains, including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction of LoT brings significant benefits compared to training models on the original dataset. The results suggest the effectiveness and efficiency of LoT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code is available at https://github.com/jincan333/LoT.