Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition
This work addresses the need for efficient hyperparameter tuning in NLP tasks, particularly for named entity recognition, and is incremental as it builds on existing fine-tuning methods with adaptive optimizations.
The paper tackles the problem of fine-tuning transformer-based language models for named entity recognition by introducing adaptive fine-tuning, which dynamically adjusts training epochs and uses a custom learning rate schedule, resulting in improved performance, stability, and efficiency, especially on small datasets where it outperforms state-of-the-art methods by a large margin.
The current standard approach for fine-tuning transformer-based language models includes a fixed number of training epochs and a linear learning rate schedule. In order to obtain a near-optimal model for the given downstream task, a search in optimization hyperparameter space is usually required. In particular, the number of training epochs needs to be adjusted to the dataset size. In this paper, we introduce adaptive fine-tuning, which is an alternative approach that uses early stopping and a custom learning rate schedule to dynamically adjust the number of training epochs to the dataset size. For the example use case of named entity recognition, we show that our approach not only makes hyperparameter search with respect to the number of training epochs redundant, but also leads to improved results in terms of performance, stability and efficiency. This holds true especially for small datasets, where we outperform the state-of-the-art fine-tuning method by a large margin.