Trace Norm Regularised Deep Multi-Task Learning
This work addresses the challenge of optimizing parameter sharing in multi-task learning for neural networks, offering an incremental improvement over methods with fixed sharing strategies.
The authors tackled the problem of learning parameter sharing strategies in deep multi-task learning by proposing a framework that uses tensor trace norm regularization to encourage neural networks to reuse parameters across tasks, resulting in a data-driven approach without predefined sharing.
We propose a framework for training multiple neural networks simultaneously. The parameters from all models are regularised by the tensor trace norm, so that each neural network is encouraged to reuse others' parameters if possible -- this is the main motivation behind multi-task learning. In contrast to many deep multi-task learning models, we do not predefine a parameter sharing strategy by specifying which layers have tied parameters. Instead, our framework considers sharing for all shareable layers, and the sharing strategy is learned in a data-driven way.