A Role-Based LLM Framework for Structured Information Extraction from Healthy Food Policies
This work addresses information extraction challenges in the healthy food policy domain, offering a more reliable and transparent method, though it is incremental as it builds on existing LLM approaches with specialized roles.
The study tackled the problem of misinformation and structural inconsistency in extracting information from healthy food policies by proposing a role-based LLM framework, which outperformed baselines like zero-shot and chain-of-thought methods on 608 policies from the Healthy Food Policy Project database.
Current Large Language Model (LLM) approaches for information extraction (IE) in the healthy food policy domain are often hindered by various factors, including misinformation, specifically hallucinations, misclassifications, and omissions that result from the structural diversity and inconsistency of policy documents. To address these limitations, this study proposes a role-based LLM framework that automates the IE from unstructured policy data by assigning specialized roles: an LLM policy analyst for metadata and mechanism classification, an LLM legal strategy specialist for identifying complex legal approaches, and an LLM food system expert for categorizing food system stages. This framework mimics expert analysis workflows by incorporating structured domain knowledge, including explicit definitions of legal mechanisms and classification criteria, into role-specific prompts. We evaluate the framework using 608 healthy food policies from the Healthy Food Policy Project (HFPP) database, comparing its performance against zero-shot, few-shot, and chain-of-thought (CoT) baselines using Llama-3.3-70B. Our proposed framework demonstrates superior performance in complex reasoning tasks, offering a reliable and transparent methodology for automating IE from health policies.