Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
This work addresses the challenge of enhancing pretraining efficiency and performance for natural language processing tasks, representing an incremental improvement over existing methods like ELECTRA.
The paper tackles the problem of improving text encoder pretraining by introducing AMOS, a framework that uses multiple auxiliary masked language models to generate varied difficulty training signals, resulting in a 1-point gain on the GLUE benchmark over ELECTRA for BERT base-sized models.
We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-style pretraining, the main encoder is trained as a discriminator to detect replaced tokens generated by auxiliary masked language models (MLMs). Different from ELECTRA which trains one MLM as the generator, we jointly train multiple MLMs of different sizes to provide training signals at various levels of difficulty. To push the discriminator to learn better with challenging replaced tokens, we learn mixture weights over the auxiliary MLMs' outputs to maximize the discriminator loss by backpropagating the gradient from the discriminator via Gumbel-Softmax. For better pretraining efficiency, we propose a way to assemble multiple MLMs into one unified auxiliary model. AMOS outperforms ELECTRA and recent state-of-the-art pretrained models by about 1 point on the GLUE benchmark for BERT base-sized models.