That's So FETCH: Fashioning Ensemble Techniques for LLM Classification in Civil Legal Intake and Referral
This addresses the critical need for efficient and accurate legal problem identification in civil legal aid systems to prevent misdirection and adverse outcomes for applicants, though it is incremental as it builds on existing ensemble and LLM methods.
The paper tackled the problem of accurately classifying legal issues from user queries to improve intake and referral in civil legal aid, achieving a classification accuracy (hits@2) of 97.37% using a mix of inexpensive models, which exceeds the performance of GPT-5.
Each year millions of people seek help for their legal problems by calling a legal aid program hotline, walking into a legal aid office, or using a lawyer referral service. The first step to match them to the right help is to identify the legal problem the applicant is experiencing. Misdirection has consequences. Applicants may miss a deadline, experience physical abuse, lose housing or lose custody of children while waiting to connect to the right legal help. We introduce and evaluate the FETCH classifier for legal issue classification and describe two methods for improving accuracy: a hybrid LLM/ML ensemble classification method, and the automatic generation of follow-up questions to enrich the initial problem narrative. We employ a novel data set of 419 real-world queries to a nonprofit lawyer referral service. Ultimately, we show classification accuracy (hits@2) of 97.37\% using a mix of inexpensive models, exceeding the performance of the current state-of-the-art GPT-5 model. Our approach shows promise in significantly reducing the cost of guiding users of the legal system to the right resource for their problem while achieving high accuracy.