Pre-Training a Language Model Without Human Language
This research addresses the fundamental question of what aspects of pre-training data contribute to transfer learning capabilities for the NLP community, showing that human language itself might not be strictly necessary for some benefits.
This paper investigates the impact of pre-training data characteristics on downstream performance, finding that models pre-trained on unstructured data outperform those trained from scratch. Surprisingly, pre-training on certain non-human language data achieved GLUE performance comparable to models pre-trained on a non-English human language.
In this paper, we study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance. To this end, we pre-train different transformer-based masked language models on several corpora with certain features, and we fine-tune those language models on GLUE benchmarks. We find that models pre-trained on unstructured data beat those trained directly from scratch on downstream tasks. Our results also show that pre-training on structured data does not always make the model acquire ability that can be transferred to natural language downstream tasks. To our great astonishment, we uncover that pre-training on certain non-human language data gives GLUE performance close to performance pre-trained on another non-English language.