Evaluating the Role of Large Language Models in Legal Practice in India
This addresses the problem of AI integration into legal practice for professionals in India, highlighting both potential augmentations and limitations, though it is incremental as it applies existing methods to a new context.
The paper empirically evaluates how well large language models (LLMs) like GPT, Claude, and Llama perform key legal tasks in India, such as issue spotting and drafting, finding they often match or surpass junior lawyers in these areas but struggle with specialized legal research due to hallucinations and inaccuracies.
The integration of Artificial Intelligence(AI) into the legal profession raises significant questions about the capacity of Large Language Models(LLM) to perform key legal tasks. In this paper, I empirically evaluate how well LLMs, such as GPT, Claude, and Llama, perform key legal tasks in the Indian context, including issue spotting, legal drafting, advice, research, and reasoning. Through a survey experiment, I compare outputs from LLMs with those of a junior lawyer, with advanced law students rating the work on helpfulness, accuracy, and comprehensiveness. LLMs excel in drafting and issue spotting, often matching or surpassing human work. However, they struggle with specialised legal research, frequently generating hallucinations, factually incorrect or fabricated outputs. I conclude that while LLMs can augment certain legal tasks, human expertise remains essential for nuanced reasoning and the precise application of law.