BERTVision -- A Parameter-Efficient Approach for Question Answering
This incremental improvement benefits researchers and practitioners by lowering resource requirements for NLP tasks.
The paper tackles the problem of reducing computational cost in BERT fine-tuning for question answering by leveraging hidden state activations, achieving maximal BERT performance with less training time and data.
We present a highly parameter efficient approach for Question Answering that significantly reduces the need for extended BERT fine-tuning. Our method uses information from the hidden state activations of each BERT transformer layer, which is discarded during typical BERT inference. Our best model achieves maximal BERT performance at a fraction of the training time and GPU or TPU expense. Performance is further improved by ensembling our model with BERTs predictions. Furthermore, we find that near optimal performance can be achieved for QA span annotation using less training data. Our experiments show that this approach works well not only for span annotation, but also for classification, suggesting that it may be extensible to a wider range of tasks.