Effective Pre-Training Objectives for Transformer-based Autoencoders
This work addresses efficiency and cost issues in pre-training for NLP researchers and practitioners, but it is incremental as it builds on existing methods like ELECTRA and BERT.
The paper tackles the trade-offs between efficiency, cost, and accuracy in pre-training Transformer encoders by analyzing and combining common objectives, resulting in light token generators that reduce computational cost compared to ELECTRA while maintaining performance.
In this paper, we study trade-offs between efficiency, cost and accuracy when pre-training Transformer encoders with different pre-training objectives. For this purpose, we analyze features of common objectives and combine them to create new effective pre-training approaches. Specifically, we designed light token generators based on a straightforward statistical approach, which can replace ELECTRA computationally heavy generators, thus highly reducing cost. Our experiments also show that (i) there are more efficient alternatives to BERT's MLM, and (ii) it is possible to efficiently pre-train Transformer-based models using lighter generators without a significant drop in performance.