Time Will Change Things: An Empirical Study on Dynamic Language Understanding in Social Media Classification
This addresses the challenge of dynamic language understanding for social media classification, but it is incremental as it builds on existing domain adaptation methods.
The study tackled the problem of deteriorating performance in natural language understanding models due to evolving language features in social media by empirically testing models trained on past data and tested on future data, finding that auto-encoding and pseudo-labeling collaboratively showed the best robustness with substantial results on four tasks.
Language features are ever-evolving in the real-world social media environment. Many trained models in natural language understanding (NLU), ineffective in semantic inference for unseen features, might consequently struggle with the deteriorating performance in dynamicity. To address this challenge, we empirically study social media NLU in a dynamic setup, where models are trained on the past data and test on the future. It better reflects the realistic practice compared to the commonly-adopted static setup of random data split. To further analyze model adaption to the dynamicity, we explore the usefulness of leveraging some unlabeled data created after a model is trained. The performance of unsupervised domain adaption baselines based on auto-encoding and pseudo-labeling and a joint framework coupling them both are examined in the experiments. Substantial results on four social media tasks imply the universally negative effects of evolving environments over classification accuracy, while auto-encoding and pseudo-labeling collaboratively show the best robustness in dynamicity.