BAM! Born-Again Multi-Task Networks for Natural Language Understanding
This work addresses a specific problem in natural language understanding for researchers and practitioners by improving multi-task learning performance, though it is incremental as it builds on existing distillation techniques.
The paper tackles the challenge of training multi-task neural networks that underperform single-task models by proposing knowledge distillation with teacher annealing, which gradually shifts from distillation to supervised learning, enabling the multi-task model to surpass its single-task teachers on the GLUE benchmark.
It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts. To help address this, we propose using knowledge distillation where single-task models teach a multi-task model. We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers. We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently improves over standard single-task and multi-task training.