Deep Multi-Task Learning via Generalized Tensor Trace Norm
This work addresses a specific problem in deep multi-task learning for researchers and practitioners by improving low-rank structure discovery and automation, though it appears incremental as it builds on existing tensor trace norm methods.
The paper tackles the limitations of existing tensor trace norms in deep multi-task learning, which cannot discover all low-rank structures and require manual tuning, by proposing a Generalized Tensor Trace Norm (GTTN) that combines matrix trace norms of all tensor flattenings and learns combination coefficients automatically, with experiments on real-world datasets showing its effectiveness.
The trace norm is widely used in multi-task learning as it can discover low-rank structures among tasks in terms of model parameters. Nowadays, with the emerging of big datasets and the popularity of deep learning techniques, tensor trace norms have been used for deep multi-task models. However, existing tensor trace norms cannot discover all the low-rank structures and they require users to manually determine the importance of their components. To solve those two issues together, in this paper, we propose a Generalized Tensor Trace Norm (GTTN). The GTTN is defined as a convex combination of matrix trace norms of all possible tensor flattenings and hence it can discover all the possible low-rank structures. In the induced objective function, we will learn combination coefficients in the GTTN to automatically determine the importance. Experiments on real-world datasets demonstrate the effectiveness of the proposed GTTN.