STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability
This work addresses reliability issues in LLM-generated structured outputs for production applications, offering practical tools for model selection and prompt refinement, though it is incremental in building on existing evaluation methods.
The paper tackles the problem of inconsistent structured outputs from LLMs by introducing a framework combining STED, a novel similarity metric, and consistency scoring, demonstrating superior performance (0.86-0.90 similarity for semantic equivalents) and revealing significant model variations, such as Claude-3.7-Sonnet maintaining near-perfect reliability at high temperatures.
Large Language Models (LLMs) are increasingly deployed for structured data generation, yet output consistency remains critical for production applications. We introduce a comprehensive framework for evaluating and improving consistency in LLM-generated structured outputs. Our approach combines: (1) STED (Semantic Tree Edit Distance), a novel similarity metric balancing semantic flexibility with structural strictness when comparing JSON outputs, and (2) a consistency scoring framework aggregating multiple STED measurements across repeated generations to quantify reliability. Through systematic experiments on synthetic datasets with controlled schema, expression, and semantic variations, we demonstrate STED achieves superior performance ($0.86-0.90$ similarity for semantic equivalents, $0.0$ for structural breaks) compared to existing metrics including TED, BERTScore, and DeepDiff. Applying our framework to benchmark six LLMs reveals significant variations: Claude-3.7-Sonnet demonstrates exceptional consistency, maintaining near-perfect structural reliability even at high temperatures ($T=0.9$), while models like Claude-3-Haiku and Nova-Pro exhibit substantial degradation requiring careful tuning. Our framework enables practical applications including targeted model selection for structured tasks, iterative prompt refinement for reproducible results, and diagnostic analysis to identify inconsistency root causes. This work provides theoretical foundations and practical tools for ensuring reliable structured output generation in LLM-based production systems.