OmniStruct: Universal Text-to-Structure Generation across Diverse Schemas
This addresses the need for universal structured generation in tasks like information extraction and table generation, but it is incremental as it builds on existing datasets and synthetic data methods.
The paper tackles the problem of assessing and improving Large Language Models' ability to generate structured outputs from text across diverse schemas, introducing OmniStruct as a benchmark and showing that fine-tuning smaller models on synthetic data can rival GPT-4o's performance without supervised data.
The ability of Large Language Models (LLMs) to generate structured outputs that follow arbitrary schemas is crucial to a wide range of downstream tasks that require diverse structured representations of results such as information extraction, table generation, and function calling. While modern LLMs excel in generating unstructured responses in natural language, whether this advancement translates to a strong performance on text-to-structure tasks remains unclear. To bridge this gap, we first introduce OmniStruct, a comprehensive benchmark for assessing LLMs' capabilities on diverse text-to-structure tasks such as information extraction, table generation, and function calling. We build OmniStruct by identifying existing datasets across a wide range of tasks that are suitable for a structured answer format, and adapting them under a unified text-to-structure problem setting. To facilitate the development of efficient text-to-structure models, we collect high-quality training data via synthetic task generation. Without using any supervised data for OmniStruct tasks, our experiments demonstrate the possibility of fine-tuning much smaller models on synthetic data into universal structured generation models that can rival the performance of GPT-4o.