Label Budget Allocation in Multi-Task Learning
This addresses the cost-efficiency challenge in multi-task learning for practitioners, though it is incremental as it builds on existing multi-task frameworks.
The paper tackles the problem of allocating a fixed label budget among tasks in multi-task learning to optimize overall performance, proposing a Task-Adaptive Budget Allocation algorithm that outperforms heuristic strategies on datasets like PASCAL VOC and Taskonomy.
The cost of labeling data often limits the performance of machine learning systems. In multi-task learning, related tasks provide information to each other and improve overall performance, but the label cost can vary among tasks. How should the label budget (i.e. the amount of money spent on labeling) be allocated among different tasks to achieve optimal multi-task performance? We are the first to propose and formally define the label budget allocation problem in multi-task learning and to empirically show that different budget allocation strategies make a big difference to its performance. We propose a Task-Adaptive Budget Allocation algorithm to robustly generate the optimal budget allocation adaptive to different multi-task learning settings. Specifically, we estimate and then maximize the extent of new information obtained from the allocated budget as a proxy for multi-task learning performance. Experiments on PASCAL VOC and Taskonomy demonstrate the efficacy of our approach over other widely used heuristic labeling strategies.