Training ELECTRA Augmented with Multi-word Selection
This work addresses the efficiency and semantic richness of pre-training for NLP researchers, though it appears incremental as it builds directly on ELECTRA with multi-task enhancements.
The authors tackled the problem of ELECTRA's binary classification task being less semantically informative by proposing a multi-task learning approach that trains the discriminator to detect replaced tokens and select original tokens from candidate sets, achieving improved performance on GLUE and SQuAD datasets.
Pre-trained text encoders such as BERT and its variants have recently achieved state-of-the-art performances on many NLP tasks. While being effective, these pre-training methods typically demand massive computation resources. To accelerate pre-training, ELECTRA trains a discriminator that predicts whether each input token is replaced by a generator. However, this new task, as a binary classification, is less semantically informative. In this study, we present a new text encoder pre-training method that improves ELECTRA based on multi-task learning. Specifically, we train the discriminator to simultaneously detect replaced tokens and select original tokens from candidate sets. We further develop two techniques to effectively combine all pre-training tasks: (1) using attention-based networks for task-specific heads, and (2) sharing bottom layers of the generator and the discriminator. Extensive experiments on GLUE and SQuAD datasets demonstrate both the effectiveness and the efficiency of our proposed method.