Evaluating Semantic and Syntactic Understanding in Large Language Models for Payroll Systems
This work addresses the problem of deploying LLMs in high-stakes, accuracy-demanding settings like payroll systems, though it is incremental as it provides practical guidance rather than a novel breakthrough.
The researchers tackled the problem of unreliable numerical calculation and auditability in large language models by evaluating their semantic and syntactic understanding in a synthetic payroll system, finding that careful prompting is sufficient in some regimes while explicit computation is required in others.
Large language models are now used daily for writing, search, and analysis, and their natural language understanding continues to improve. However, they remain unreliable on exact numerical calculation and on producing outputs that are straightforward to audit. We study synthetic payroll system as a focused, high-stakes example and evaluate whether models can understand a payroll schema, apply rules in the right order, and deliver cent-accurate results. Our experiments span a tiered dataset from basic to complex cases, a spectrum of prompts from minimal baselines to schema-guided and reasoning variants, and multiple model families including GPT, Claude, Perplexity, Grok and Gemini. Results indicate clear regimes where careful prompting is sufficient and regimes where explicit computation is required. The work offers a compact, reproducible framework and practical guidance for deploying LLMs in settings that demand both accuracy and assurance.