From BERT to T5: A Study of Named Entity Recognition
For NLP practitioners, this provides a comparative analysis of two popular architectures for NER, but the results are incremental and confirm known strengths of encoder-only models.
This study compares BERT and T5 for named entity recognition, finding that BERT with a classification head outperforms T5 with few-shot prompts across 7-class and 3-class tag schemes, with BERT achieving higher F1 scores.
Named entity recognition (NER) has been one of the essential preliminary steps in modern NLP applications. This report focuses on implementing the NER task on finetuning two pretrained models: (i) an encoder-only model (BERT) with a simple classification head, and (ii) a sequence-to-sequence model (T5) with few-shot prompts. Under the original 7-class tag and 3-class simplified tag schemes, BERT is applied a weighted cross-entropy for training loss, and T5 is fine-tuned with two validation strategies. It also conducted an ablation study with different hyperparameters. Moreover, the related analysis provides valuable insights into common errors in BERT and the two models' performance. Based on a bunch of performance metrics, this report aims to compare the above two architectures and explore their abilities in the sequence labelling task, laying the groundwork for further practical use cases.