Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4
This work addresses the challenge of automating legal reasoning for law students and NLP applications, but it is incremental as it applies existing prompting methods to a new domain-specific task.
The paper tackled the problem of legal argument reasoning in civil procedures by developing a prompt-based system using GPT4 with ensembling strategies, achieving a Macro F1 of 0.8095 on validation and 0.7315 (5th out of 21 teams) on the test set.
In this paper, we present our system for the SemEval Task 5, The Legal Argument Reasoning Task in Civil Procedure Challenge. Legal argument reasoning is an essential skill that all law students must master. Moreover, it is important to develop natural language processing solutions that can reason about a question given terse domain-specific contextual information. Our system explores a prompt-based solution using GPT4 to reason over legal arguments. We also evaluate an ensemble of prompting strategies, including chain-of-thought reasoning and in-context learning. Overall, our system results in a Macro F1 of .8095 on the validation dataset and .7315 (5th out of 21 teams) on the final test set. Code for this project is available at https://github.com/danschumac1/CivilPromptReasoningGPT4.