DSReg: Using Distant Supervision as a Regularizer
This addresses a general issue in NLP for tasks where distinguishing similar negative examples is challenging, though it appears incremental as it builds on existing multi-task learning and regularization techniques.
The paper tackles the problem of hard-negative examples in NLP tasks by proposing DSReg, a method that uses distant supervision as a regularizer to improve model performance across text classification, sequence labeling, and reading comprehension tasks, achieving enhanced results over baseline models.
In this paper, we aim at tackling a general issue in NLP tasks where some of the negative examples are highly similar to the positive examples, i.e., hard-negative examples. We propose the distant supervision as a regularizer (DSReg) approach to tackle this issue. The original task is converted to a multi-task learning problem, in which distant supervision is used to retrieve hard-negative examples. The obtained hard-negative examples are then used as a regularizer. The original target objective of distinguishing positive examples from negative examples is jointly optimized with the auxiliary task objective of distinguishing softened positive (i.e., hard-negative examples plus positive examples) from easy-negative examples. In the neural context, this can be done by outputting the same representation from the last neural layer to different $softmax$ functions. Using this strategy, we can improve the performance of baseline models in a range of different NLP tasks, including text classification, sequence labeling and reading comprehension.