Transfer Fine-Tuning: A BERT Case Study
This addresses the computational barrier for practical NLP applications by enhancing smaller models, though it is incremental as it builds on existing BERT methods.
The paper tackles the computational inefficiency of large BERT models for semantic equivalence assessment by injecting phrasal paraphrase relations into a smaller BERT model, achieving superior performance compared to a larger BERT model on these tasks, with larger gains on datasets with limited training data.
A semantic equivalence assessment is defined as a task that assesses semantic equivalence in a sentence pair by binary judgment (i.e., paraphrase identification) or grading (i.e., semantic textual similarity measurement). It constitutes a set of tasks crucial for research on natural language understanding. Recently, BERT realized a breakthrough in sentence representation learning (Devlin et al., 2019), which is broadly transferable to various NLP tasks. While BERT's performance improves by increasing its model size, the required computational power is an obstacle preventing practical applications from adopting the technology. Herein, we propose to inject phrasal paraphrase relations into BERT in order to generate suitable representations for semantic equivalence assessment instead of increasing the model size. Experiments on standard natural language understanding tasks confirm that our method effectively improves a smaller BERT model while maintaining the model size. The generated model exhibits superior performance compared to a larger BERT model on semantic equivalence assessment tasks. Furthermore, it achieves larger performance gains on tasks with limited training datasets for fine-tuning, which is a property desirable for transfer learning.