DALE: Generative Data Augmentation for Low-Resource Legal NLP
This addresses the challenge of data scarcity in legal NLP, where specialized language hinders traditional augmentation methods, though it is incremental as it builds on existing encoder-decoder models.
The paper tackles the problem of generating effective data augmentations for low-resource legal NLP by introducing DALE, a framework that uses a novel unsupervised text denoising objective to produce coherent and diverse augmentations, resulting in performance improvements of 1%-50% across 13 datasets.
We present DALE, a novel and effective generative Data Augmentation framework for low-resource LEgal NLP. DALE addresses the challenges existing frameworks pose in generating effective data augmentations of legal documents - legal language, with its specialized vocabulary and complex semantics, morphology, and syntax, does not benefit from data augmentations that merely rephrase the source sentence. To address this, DALE, built on an Encoder-Decoder Language Model, is pre-trained on a novel unsupervised text denoising objective based on selective masking - our masking strategy exploits the domain-specific language characteristics of templatized legal documents to mask collocated spans of text. Denoising these spans helps DALE acquire knowledge about legal concepts, principles, and language usage. Consequently, it develops the ability to generate coherent and diverse augmentations with novel contexts. Finally, DALE performs conditional generation to generate synthetic augmentations for low-resource Legal NLP tasks. We demonstrate the effectiveness of DALE on 13 datasets spanning 6 tasks and 4 low-resource settings. DALE outperforms all our baselines, including LLMs, qualitatively and quantitatively, with improvements of 1%-50%.