DACBERT: Leveraging Dependency Agreement for Cost-Efficient Bert Pretraining
This work addresses the need for more efficient and interpretable pretraining methods in natural language processing, offering incremental improvements over existing cost-efficient approaches.
The paper tackles the problem of cost-efficient pretraining for BERT models by introducing DACBERT, a novel framework that incorporates dependency agreement to enhance performance and interpretability, resulting in improvements such as a 3.13% increase in RTE task accuracy and a 0.83% boost in average GLUE score compared to Crammed BERT.
Building on the cost-efficient pretraining advancements brought about by Crammed BERT, we enhance its performance and interpretability further by introducing a novel pretrained model Dependency Agreement Crammed BERT (DACBERT) and its two-stage pretraining framework - Dependency Agreement Pretraining. This framework, grounded by linguistic theories, seamlessly weaves syntax and semantic information into the pretraining process. The first stage employs four dedicated submodels to capture representative dependency agreements at the chunk level, effectively converting these agreements into embeddings. The second stage uses these refined embeddings, in tandem with conventional BERT embeddings, to guide the pretraining of the rest of the model. Evaluated on the GLUE benchmark, our DACBERT demonstrates notable improvement across various tasks, surpassing Crammed BERT by 3.13% in the RTE task and by 2.26% in the MRPC task. Furthermore, our method boosts the average GLUE score by 0.83%, underscoring its significant potential. The pretraining process can be efficiently executed on a single GPU within a 24-hour cycle, necessitating no supplementary computational resources or extending the pretraining duration compared with the Crammed BERT. Extensive studies further illuminate our approach's instrumental role in bolstering the interpretability of pretrained language models for natural language understanding tasks.