Maximising the Utility of Validation Sets for Imbalanced Noisy-label Meta-learning
This addresses a scalability and performance bottleneck in meta-learning for imbalanced and noisy-label data, though it is incremental as it builds on existing meta-learning frameworks.
The paper tackles the problem of sub-optimal validation sets in imbalanced noisy-label meta-learning by proposing new criteria for utility and an algorithm to automatically build validation sets, resulting in significant improvements and new state-of-the-art performance on benchmarks.
Meta-learning is an effective method to handle imbalanced and noisy-label learning, but it depends on a validation set containing randomly selected, manually labelled and balanced distributed samples. The random selection and manual labelling and balancing of this validation set is not only sub-optimal for meta-learning, but it also scales poorly with the number of classes. Hence, recent meta-learning papers have proposed ad-hoc heuristics to automatically build and label this validation set, but these heuristics are still sub-optimal for meta-learning. In this paper, we analyse the meta-learning algorithm and propose new criteria to characterise the utility of the validation set, based on: 1) the informativeness of the validation set; 2) the class distribution balance of the set; and 3) the correctness of the labels of the set. Furthermore, we propose a new imbalanced noisy-label meta-learning (INOLML) algorithm that automatically builds a validation set by maximising its utility using the criteria above. Our method shows significant improvements over previous meta-learning approaches and sets the new state-of-the-art on several benchmarks.