Learning by Teaching, with Application to Neural Architecture Search
This work proposes a novel framework for enhancing model training in machine learning, specifically applied to neural architecture search, but it appears incremental as it adapts a human learning concept to an existing optimization problem.
The paper tackles the problem of improving machine learning model performance by introducing a learning by teaching (LBT) framework, where a teacher model trains a student model and re-learns based on the student's validation performance, achieving demonstrated efficacy on neural architecture search tasks for CIFAR-10, CIFAR-100, and ImageNet datasets.
In human learning, an effective skill in improving learning outcomes is learning by teaching: a learner deepens his/her understanding of a topic by teaching this topic to others. In this paper, we aim to borrow this teaching-driven learning methodology from humans and leverage it to train more performant machine learning models, by proposing a novel ML framework referred to as learning by teaching (LBT). In the LBT framework, a teacher model improves itself by teaching a student model to learn well. Specifically, the teacher creates a pseudo-labeled dataset and uses it to train a student model. Based on how the student performs on a validation dataset, the teacher re-learns its model and re-teaches the student until the student achieves great validation performance. Our framework is based on three-level optimization which contains three stages: teacher learns; teacher teaches student; teacher re-learns based on how well the student performs. A simple but efficient algorithm is developed to solve the three-level optimization problem. We apply LBT to search neural architectures on CIFAR-10, CIFAR-100, and ImageNet. The efficacy of our method is demonstrated in various experiments.