Data Annealing for Informal Language Understanding Tasks
This addresses the problem of poor performance of pre-trained models on informal language for NLP researchers and practitioners, though it appears incremental as it adapts existing methods to a specific domain.
The paper tackles the performance gap between formal and informal language understanding tasks by proposing a data annealing transfer learning procedure, which gradually increases the proportion of informal text data during training, and it outperforms state-of-the-art models on three common informal language tasks when implemented with BERT.
There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve a comparable result on informal language. We pro-pose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In our data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.