To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment
This work addresses the challenge of domain transfer in legal NLP, showing that minimal adaptation can be more robust than fine-tuning, which is incremental but impactful for practitioners with limited labeled data.
The paper tackled the problem of legal case entailment by using pretrained language models without domain adaptation, achieving the highest scores in the COLIEE 2021 task and surpassing the second-best team by over six percentage points.
There has been mounting evidence that pretrained language models fine-tuned on large and diverse supervised datasets can transfer well to a variety of out-of-domain tasks. In this work, we investigate this transfer ability to the legal domain. For that, we participated in the legal case entailment task of COLIEE 2021, in which we use such models with no adaptations to the target domain. Our submissions achieved the highest scores, surpassing the second-best team by more than six percentage points. Our experiments confirm a counter-intuitive result in the new paradigm of pretrained language models: given limited labeled data, models with little or no adaptation to the target task can be more robust to changes in the data distribution than models fine-tuned on it. Code is available at https://github.com/neuralmind-ai/coliee.