Low-Rank Deep Convolutional Neural Network for Multi-Task Learning
This work addresses multi-task learning problems in computer vision and NLP, but it is incremental as it builds on existing deep learning and regularization techniques.
The authors tackled multi-task learning by proposing a low-rank deep convolutional neural network that explores task relations through nuclear norm regularization and feature selection via sparsity penalty, achieving improved performance on benchmark datasets for face attribute prediction, natural language processing, and economics index predictions.
In this paper, we propose a novel multi-task learning method based on the deep convolutional network. The proposed deep network has four convolutional layers, three max-pooling layers, and two parallel fully connected layers. To adjust the deep network to multi-task learning problem, we propose to learn a low-rank deep network so that the relation among different tasks can be explored. We proposed to minimize the number of independent parameter rows of one fully connected layer to explore the relations among different tasks, which is measured by the nuclear norm of the parameter of one fully connected layer, and seek a low-rank parameter matrix. Meanwhile, we also propose to regularize another fully connected layer by sparsity penalty, so that the useful features learned by the lower layers can be selected. The learning problem is solved by an iterative algorithm based on gradient descent and back-propagation algorithms. The proposed algorithm is evaluated over benchmark data sets of multiple face attribute prediction, multi-task natural language processing, and joint economics index predictions. The evaluation results show the advantage of the low-rank deep CNN model over multi-task problems.