LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
This provides a benchmark for researchers and practitioners working on LLM-based parsing or ETL applications, though it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating Large Language Models (LLMs) for structured data extraction from text by creating LLMStructBench, a benchmark with diverse parsing scenarios and complementary metrics, showing that prompting strategy matters more than model size for structural validity but increases semantic errors.
We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.