Structsum Generation for Faster Text Comprehension
This addresses the problem of slow text comprehension for users by providing faster alternatives through structured outputs, though it is incremental as it builds on existing LLM capabilities with new prompting methods.
The paper tackles the problem of generating structured text representations (tables and mind maps) with LLMs, showing current models struggle with this task. They present prompting strategies and critiques that improve accuracy by +37pp for mind maps and +15pp for tables, and a user study demonstrates these structured representations reduce comprehension time by 42.9% for tables and 31.9% for mind maps without accuracy loss.
We consider the task of generating structured representations of text using large language models (LLMs). We focus on tables and mind maps as representative modalities. Tables are more organized way of representing data, while mind maps provide a visually dynamic and flexible approach, particularly suitable for sparse content. Despite the effectiveness of LLMs on different tasks, we show that current models struggle with generating structured outputs. In response, we present effective prompting strategies for both of these tasks. We introduce a taxonomy of problems around factuality, global and local structure, common to both modalities and propose a set of critiques to tackle these issues resulting in an absolute improvement in accuracy of +37pp (79%) for mind maps and +15pp (78%) for tables. To evaluate semantic coverage of generated structured representations we propose Auto-QA, and we verify the adequacy of Auto-QA using SQuAD dataset. We further evaluate the usefulness of structured representations via a text comprehension user study. The results show a significant reduction in comprehension time compared to text when using table (42.9%) and mind map (31.9%), without loss in accuracy.