LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension
This work addresses the problem of automating legal text analysis for legal services by improving NLP tools for mixed-domain and long-range comprehension, though it is incremental as it builds on existing frameworks like Electra and Reformer.
The authors tackled the challenge of processing specialized legal texts containing mixed-domain vocabulary, such as medical terms in personal injury documents, by proposing LegalRelectra, a legal-domain language model trained on mixed legal and medical corpora. They showed that their model outperforms general-domain and single-domain models on mixed-domain text, with improved performance on long passages due to using Reformer within the Electra framework.
The application of Natural Language Processing (NLP) to specialized domains, such as the law, has recently received a surge of interest. As many legal services rely on processing and analyzing large collections of documents, automating such tasks with NLP tools emerges as a key challenge. Many popular language models, such as BERT or RoBERTa, are general-purpose models, which have limitations on processing specialized legal terminology and syntax. In addition, legal documents may contain specialized vocabulary from other domains, such as medical terminology in personal injury text. Here, we propose LegalRelectra, a legal-domain language model that is trained on mixed-domain legal and medical corpora. We show that our model improves over general-domain and single-domain medical and legal language models when processing mixed-domain (personal injury) text. Our training architecture implements the Electra framework, but utilizes Reformer instead of BERT for its generator and discriminator. We show that this improves the model's performance on processing long passages and results in better long-range text comprehension.