The Dark Side of the Language: Pre-trained Transformers in the DarkNet
This work addresses the robustness of pre-trained Transformers for NLP practitioners, revealing limitations in handling truly novel data, though it is incremental as it builds on existing adaptation methods.
The study investigated how pre-trained Transformers perform on unseen sentences from a DarkNet corpus, finding that syntactic and lexical neural networks matched their performance after fine-tuning, and only extreme domain adaptation (retraining on the novel corpus) allowed Transformers to achieve high results, suggesting that large pre-training corpora may provide unexpected advantages by exposing them to many possible sentences.
Pre-trained Transformers are challenging human performances in many NLP tasks. The massive datasets used for pre-training seem to be the key to their success on existing tasks. In this paper, we explore how a range of pre-trained Natural Language Understanding models perform on definitely unseen sentences provided by classification tasks over a DarkNet corpus. Surprisingly, results show that syntactic and lexical neural networks perform on par with pre-trained Transformers even after fine-tuning. Only after what we call extreme domain adaptation, that is, retraining with the masked language model task on all the novel corpus, pre-trained Transformers reach their standard high results. This suggests that huge pre-training corpora may give Transformers unexpected help since they are exposed to many of the possible sentences.