Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets
This work addresses the impact of structural constraints in text-to-table generation for researchers and practitioners, but it is incremental as it focuses on evaluation rather than introducing new methods.
The paper tackled the problem of evaluating structured decoding for text-to-table generation with large language models, finding that it significantly improves table validity and alignment in numerical scenarios but can degrade performance in text-heavy contexts.
We present a comprehensive evaluation of structured decoding for text-to-table generation with large language models (LLMs). While previous work has primarily focused on unconstrained generation of tables, the impact of enforcing structural constraints during generation remains underexplored. We systematically compare schema-guided (structured) decoding to standard one-shot prompting across three diverse benchmarks - E2E, Rotowire, and Livesum - using open-source LLMs of up to 32B parameters, assessing the performance of table generation approaches in resource-constrained settings. Our experiments cover a wide range of evaluation metrics at cell, row, and table levels. Results demonstrate that structured decoding significantly enhances the validity and alignment of generated tables, particularly in scenarios demanding precise numerical alignment (Rotowire), but may degrade performance in contexts involving densely packed textual information (E2E) or extensive aggregation over lengthy texts (Livesum). We further analyze the suitability of different evaluation metrics and discuss the influence of model size.