Dual-Balancing for Multi-Task Learning
This addresses the challenge of task balancing in multi-task learning for researchers and practitioners, but it appears incremental as it builds on existing balancing techniques.
The paper tackled the problem of performance compromises in multi-task learning due to disparities in loss and gradient scales among tasks, and proposed Dual-Balancing Multi-Task Learning (DB-MTL) to balance tasks from both loss and gradient perspectives, achieving consistent improvements over state-of-the-art methods on benchmark datasets.
Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art.