Exploring Iterative Controllable Summarization with Large Language Models
This addresses the need for more adaptable summarization tools for users with specific preferences, though it is incremental as it builds on existing iterative methods.
The paper tackles the problem of precisely controlling summary attributes like length or topic in large language models for summarization, finding that LLMs struggle more with numerical attributes than linguistic ones, and proposes a guide-to-explain framework that reduces iteration counts and improves alignment with desired attributes.
Large language models (LLMs) have demonstrated remarkable performance in abstractive summarization tasks. However, their ability to precisely control summary attributes (e.g., length or topic) remains underexplored, limiting their adaptability to specific user preferences. In this paper, we systematically explore the controllability of LLMs. To this end, we revisit summary attribute measurements and introduce iterative evaluation metrics, failure rate and average iteration count to precisely evaluate controllability of LLMs, rather than merely assessing errors. Our findings show that LLMs struggle more with numerical attributes than with linguistic attributes. To address this challenge, we propose a guide-to-explain framework (GTE) for controllable summarization. Our GTE framework enables the model to identify misaligned attributes in the initial draft and guides it in self-explaining errors in the previous output. By allowing the model to reflect on its misalignment, GTE generates well-adjusted summaries that satisfy the desired attributes with robust effectiveness, requiring surprisingly fewer iterations than other iterative approaches.