Minimax Multi-Task Learning and a Generalized Loss-Compositional Paradigm for MTL
This work addresses multi-task learning for AI/ML practitioners by proposing a novel paradigm, though it appears incremental as it builds on classical MTL with new formulations.
The paper tackles the problem of multi-task learning (MTL) by introducing a generalized loss-compositional paradigm, with minimax MTL as a key formulation that minimizes the maximum of tasks' empirical risks, and shows theoretically that it avoids worst-case outcomes on new test tasks in learning to learn settings.
Since its inception, the modus operandi of multi-task learning (MTL) has been to minimize the task-wise mean of the empirical risks. We introduce a generalized loss-compositional paradigm for MTL that includes a spectrum of formulations as a subfamily. One endpoint of this spectrum is minimax MTL: a new MTL formulation that minimizes the maximum of the tasks' empirical risks. Via a certain relaxation of minimax MTL, we obtain a continuum of MTL formulations spanning minimax MTL and classical MTL. The full paradigm itself is loss-compositional, operating on the vector of empirical risks. It incorporates minimax MTL, its relaxations, and many new MTL formulations as special cases. We show theoretically that minimax MTL tends to avoid worst case outcomes on newly drawn test tasks in the learning to learn (LTL) test setting. The results of several MTL formulations on synthetic and real problems in the MTL and LTL test settings are encouraging.