GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks
This addresses the need for efficient and effective task selection in multi-task learning for natural language understanding, though it is incremental as it builds on existing gradient and Transformer-based approaches.
The paper tackles the problem of automatically selecting high-quality auxiliary tasks in multi-task learning by proposing GradTS, a gradient-based method using Transformer networks, which improves performance on 8 GLUE tasks by up to 17.93% compared to a baseline and reduces time by 21.32% on average.
A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. This paper presents GradTS, an automatic auxiliary task selection method based on gradient calculation in Transformer-based models. Compared to AUTOSEM, a strong baseline method, GradTS improves the performance of MT-DNN with a bert-base-cased backend model, from 0.33% to 17.93% on 8 natural language understanding (NLU) tasks in the GLUE benchmarks. GradTS is also time-saving since (1) its gradient calculations are based on single-task experiments and (2) the gradients are re-used without additional experiments when the candidate task set changes. On the 8 GLUE classification tasks, for example, GradTS costs on average 21.32% less time than AUTOSEM with comparable GPU consumption. Further, we show the robustness of GradTS across various task settings and model selections, e.g. mixed objectives among candidate tasks. The efficiency and efficacy of GradTS in these case studies illustrate its general applicability in MTL research without requiring manual task filtering or costly parameter tuning.