Deep Elastic Networks with Model Selection for Multi-Task Learning
This work addresses the challenge of optimizing model selection for multiple tasks in machine learning, though it appears incremental as it builds on existing multi-task learning and model selection techniques.
The authors tackled the problem of instance-wise dynamic network model selection for multi-task learning by proposing an efficient approach that uses an estimator and selector to choose accurate models per instance, achieving outstanding performance compared to existing methods.
In this work, we consider the problem of instance-wise dynamic network model selection for multi-task learning. To this end, we propose an efficient approach to exploit a compact but accurate model in a backbone architecture for each instance of all tasks. The proposed method consists of an estimator and a selector. The estimator is based on a backbone architecture and structured hierarchically. It can produce multiple different network models of different configurations in a hierarchical structure. The selector chooses a model dynamically from a pool of candidate models given an input instance. The selector is a relatively small-size network consisting of a few layers, which estimates a probability distribution over the candidate models when an input instance of a task is given. Both estimator and selector are jointly trained in a unified learning framework in conjunction with a sampling-based learning strategy, without additional computation steps. We demonstrate the proposed approach for several image classification tasks compared to existing approaches performing model selection or learning multiple tasks. Experimental results show that our approach gives not only outstanding performance compared to other competitors but also the versatility to perform instance-wise model selection for multiple tasks.