Learning by Ignoring, with Application to Domain Adaptation
This work addresses the problem of domain shift in pretraining data for machine learning models, which is relevant for researchers and practitioners working on domain adaptation.
This paper proposes a machine learning framework called Learning by Ignoring (LBI) that automatically identifies and excludes pretraining data examples with large domain shifts from the target distribution. It achieves this by learning an 'ignoring variable' for each example, leading to improved performance on various datasets.
Learning by ignoring, which identifies less important things and excludes them from the learning process, is broadly practiced in human learning and has shown ubiquitous effectiveness. There has been psychological studies showing that learning to ignore certain things is a powerful tool for helping people focus. In this paper, we explore whether this useful human learning methodology can be borrowed to improve machine learning. We propose a novel machine learning framework referred to as learning by ignoring (LBI). Our framework automatically identifies pretraining data examples that have large domain shift from the target distribution by learning an ignoring variable for each example and excludes them from the pretraining process. We formulate LBI as a three-level optimization framework where three learning stages are involved: pretraining by minimizing the losses weighed by ignoring variables; finetuning; updating the ignoring variables by minimizing the validation loss. A gradient-based algorithm is developed to efficiently solve the three-level optimization problem in LBI. Experiments on various datasets demonstrate the effectiveness of our framework.