Controlled Randomness Improves the Performance of Transformer Models
This addresses the challenge of data scarcity in specific domains for fine-tuning pre-trained models, though it appears incremental as it builds on existing noise injection techniques.
The authors tackled the problem of fine-tuning language models on small downstream datasets by introducing controlled randomness (noise) during training, finding that it improved performance on joint named entity recognition with relation extraction and text summarization tasks.
During the pre-training step of natural language models, the main objective is to learn a general representation of the pre-training dataset, usually requiring large amounts of textual data to capture the complexity and diversity of natural language. Contrasting this, in most cases, the size of the data available to solve the specific downstream task is often dwarfed by the aforementioned pre-training dataset, especially in domains where data is scarce. We introduce controlled randomness, i.e. noise, into the training process to improve fine-tuning language models and explore the performance of targeted noise in addition to the parameters of these models. We find that adding such noise can improve the performance in our two downstream tasks of joint named entity recognition and relation extraction and text summarization.