Report-based Recommendations for Policy Making and Agency Operations: Dataset and LLM Evaluation
This addresses the need for automated policy-making support in organizations, but it is incremental as it builds on existing LLM capabilities for a specific application.
The paper tackles the problem of generating policy recommendations from reports by introducing a new benchmark task and evaluating state-of-the-art LLMs, showing they can highlight key issues and learning points in recommendations.
Large Language Models (LLMs) are extensively used in text generation tasks. These generative capabilities bring us to a point where LLMs could potentially provide useful insights in policy making or agency operations. In this paper, we introduce a new task consisting of generating recommendations which can be used to inform future actions and improvements of agencies work within private and public organisations. In particular, we present the first benchmark and coherent evaluation for developing recommendation systems to inform organisation policies. This task is clearly different from usual product or user recommendation systems, but rather aims at providing a basis to suggest policy improvements based on the conclusions drawn from reports. Our results demonstrate that state-of-the-art LLMs have the potential to emphasize and reflect on key issues and learning points within generated recommendations.