Neural Architecture Search for Sentence Classification with BERT
This work addresses a specific bottleneck in NLP fine-tuning for researchers and practitioners, offering an incremental improvement over standard methods.
The paper tackled the suboptimal practice of using a single output layer for fine-tuning BERT in sentence classification by performing an AutoML search to find better architectures, achieving improved results on GLUE benchmarks with minimal computational overhead.
Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.