Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law
For legal AI practitioners, this work highlights the vulnerability of LLMs to data contamination and demonstrates the superiority of neuro-symbolic approaches for robust legal reasoning.
The paper investigates whether LLMs' performance in tax law reasoning reflects genuine ability or data contamination, finding that contamination inflates performance. It shows that neuro-symbolic frameworks provide more reliable and robust legal reasoning with improved generalization.
Recent advances in large language models (LLMs) have significantly enhanced automated legal reasoning. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability. We show that performance can be inflated by contamination. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations. Our findings indicate that legal reasoning is inherently compositional and that neuro-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations.