Na Prática, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jurídica
This addresses the need for reliable AI in legal practice by demonstrating the importance of domain specialization and theoretically grounded evaluation, though it is incremental as it builds on existing AI evaluation methods.
The study tackled the problem of evaluating AI systems' understanding of law by testing four AI models on tasks simulating lawyers' daily work, with results showing that the domain-specialized model JusIA consistently outperformed general-purpose systems like ChatGPT and Gemini.
This study presents the Jusbrasil Study on the Use of General-Purpose AIs in Law, proposing an experimental evaluation protocol combining legal theory, such as material correctness, systematic coherence, and argumentative integrity, with empirical assessment by 48 legal professionals. Four systems (JusIA, ChatGPT Free, ChatGPT Pro, and Gemini) were tested in tasks simulating lawyers' daily work. JusIA, a domain-specialized model, consistently outperformed the general-purpose systems, showing that both domain specialization and a theoretically grounded evaluation are essential for reliable legal AI outputs.