On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
This addresses the issue of model performance degradation in real-world applications like paraphrase detection and question answering, where data is naturally imbalanced, by proposing an adaptive data collection method.
The paper tackles the problem of poor generalization in pairwise classification tasks due to extreme label imbalance in real-world data, showing that models trained on balanced datasets achieve only 2.4% average precision on imbalanced test data, while using active learning to collect balanced training data improves average precision to 32.5% on QQP and 20.1% on WikiQA.
Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., $99.99\%$ of examples are negatives). In contrast, many recent datasets heuristically choose examples to ensure label balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only $2.4\%$ average precision when evaluated on realistically imbalanced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently retrieve uncertain points from a very large pool of unlabeled utterance pairs. By creating balanced training data with more informative negative examples, active learning greatly improves average precision to $32.5\%$ on QQP and $20.1\%$ on WikiQA.