Efficient Pareto Manifold Learning with Low-Rank Structure
This work addresses scalability challenges in multi-task learning for researchers and practitioners dealing with large numbers of tasks, representing an incremental improvement over existing methods.
The paper tackles the scalability issue in continuous Pareto front approximations for multi-task learning by proposing a method that integrates a main network with low-rank matrices, which reduces parameters and improves shared feature extraction. Experimental results show that the approach outperforms state-of-the-art baselines, particularly on datasets with many tasks.
Multi-task learning, which optimizes performance across multiple tasks, is inherently a multi-objective optimization problem. Various algorithms are developed to provide discrete trade-off solutions on the Pareto front. Recently, continuous Pareto front approximations using a linear combination of base networks have emerged as a compelling strategy. However, it suffers from scalability issues when the number of tasks is large. To address this issue, we propose a novel approach that integrates a main network with several low-rank matrices to efficiently learn the Pareto manifold. It significantly reduces the number of parameters and facilitates the extraction of shared features. We also introduce orthogonal regularization to further bolster performance. Extensive experimental results demonstrate that the proposed approach outperforms state-of-the-art baselines, especially on datasets with a large number of tasks.