StructuredRAG: JSON Response Formatting with Large Language Models
This work addresses the problem of unreliable structured output generation in LLMs for developers and researchers in Compound AI Systems, though it is incremental as it focuses on benchmarking and evaluation rather than novel solutions.
The authors tackled the challenge of evaluating LLMs' ability to generate structured JSON outputs by introducing StructuredRAG, a benchmark of six tasks, and found an average success rate of 82.55% across experiments with models like Gemini 1.5 Pro and Llama 3 8B-instruct.
The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON, is crucial for their use in Compound AI Systems. However, evaluating and improving this capability remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions. We evaluate two state-of-the-art LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting. Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro. We observe that task complexity significantly influences performance, with tasks involving lists or composite object outputs proving more challenging. Our findings highlight the need for further research into improving the reliability and consistency of structured output generation in LLMs. We have open-sourced our experimental code and results at github.com/weaviate/structured-rag.