Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain
This work addresses domain adaptation challenges in NLP for specialized fields like finance, though it is incremental in nature.
The paper tackles the problem of fine-tuning pre-trained language models for specialized domains by incorporating regular expression patterns as domain knowledge features, resulting in improved text classification performance on real production data compared to fine-tuning only on domain-specific text.
A common way to use large pre-trained language models for downstream tasks is to fine tune them using additional layers. This may not work well if downstream domain is a specialized domain whereas the large language model has been pre-trained on a generic corpus. In this paper, we discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning, in addition to domain specific text. Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks as compared to fine tuning only on domain specific text. We also show that the use of attention network for fine tuning improves results compared to simple linear layers.