Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners
This addresses interference and catastrophic forgetting in multi-task learning for AI systems, though it is incremental as it builds on existing MoE methods.
The paper tackles multi-task learning interference by using sparsely activated Mixture-of-Experts with task-aware gating to specialize weights for shared and task-specific representations, resulting in improved transfer to low-resource tasks, generalization to unseen tasks, and robustness against catastrophic forgetting.
Traditional multi-task learning (MTL) methods use dense networks that use the same set of shared weights across several different tasks. This often creates interference where two or more tasks compete to pull model parameters in different directions. In this work, we study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning by specializing some weights for learning shared representations and using the others for learning task-specific information. To this end, we devise task-aware gating functions to route examples from different tasks to specialized experts which share subsets of network weights conditioned on the task. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model. We demonstrate such sparse networks to improve multi-task learning along three key dimensions: (i) transfer to low-resource tasks from related tasks in the training mixture; (ii) sample-efficient generalization to tasks not seen during training by making use of task-aware routing from seen related tasks; (iii) robustness to the addition of unrelated tasks by avoiding catastrophic forgetting of existing tasks.