Training Dynamic based data filtering may not work for NLP datasets
This addresses the problem of poor generalization due to mislabeled data in NLP for researchers and practitioners, but it is incremental as it critiques an existing method.
The paper investigates the use of the Area Under the Margin (AUM) metric to filter mislabeled data in NLP datasets, finding that while it can identify mislabeled samples, it also removes many correctly labeled points and leads to significant loss of relevant language information, as models rely on distributional rather than syntactic or semantic cues.
The recent increase in dataset size has brought about significant advances in natural language understanding. These large datasets are usually collected through automation (search engines or web crawlers) or crowdsourcing which inherently introduces incorrectly labeled data. Training on these datasets leads to memorization and poor generalization. Thus, it is pertinent to develop techniques that help in the identification and isolation of mislabelled data. In this paper, we study the applicability of the Area Under the Margin (AUM) metric to identify and remove/rectify mislabelled examples in NLP datasets. We find that mislabelled samples can be filtered using the AUM metric in NLP datasets but it also removes a significant number of correctly labeled points and leads to the loss of a large amount of relevant language information. We show that models rely on the distributional information instead of relying on syntactic and semantic representations.