Data-Efficient and Robust Task Selection for Meta-Learning
This addresses the challenge of optimizing meta-learning for real-world applications where tasks vary in importance and noise, offering an incremental improvement without architectural changes.
The paper tackles the problem of inefficient and non-robust task selection in meta-learning by proposing the DERTS algorithm, which selects weighted task subsets to minimize gradient approximation error, resulting in improved performance over existing sampling strategies in limited data and noisy settings.
Meta-learning methods typically learn tasks under the assumption that all tasks are equally important. However, this assumption is often not valid. In real-world applications, tasks can vary both in their importance during different training stages and in whether they contain noisy labeled data or not, making a uniform approach suboptimal. To address these issues, we propose the Data-Efficient and Robust Task Selection (DERTS) algorithm, which can be incorporated into both gradient and metric-based meta-learning algorithms. DERTS selects weighted subsets of tasks from task pools by minimizing the approximation error of the full gradient of task pools in the meta-training stage. The selected tasks are efficient for rapid training and robust towards noisy label scenarios. Unlike existing algorithms, DERTS does not require any architecture modification for training and can handle noisy label data in both the support and query sets. Analysis of DERTS shows that the algorithm follows similar training dynamics as learning on the full task pools. Experiments show that DERTS outperforms existing sampling strategies for meta-learning on both gradient-based and metric-based meta-learning algorithms in limited data budget and noisy task settings.