CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization
This addresses the need for broader dissemination of scientific knowledge by enabling fine-grained control in summarization, though it is incremental as it builds on existing single-attribute control research.
The paper tackles the problem of controlling multiple attributes simultaneously in scientific document summarization, introducing CCSBench as the first evaluation benchmark for compositional controllable summarization, and finds that large language models have significant limitations in balancing trade-offs between control attributes, especially implicit ones.
To broaden the dissemination of scientific knowledge to diverse audiences, it is desirable for scientific document summarization systems to simultaneously control multiple attributes such as length and empirical focus. However, existing research typically focuses on controlling single attributes, leaving the compositional control of multiple attributes underexplored. To address this gap, we introduce CCSBench, the first evaluation benchmark for compositional controllable summarization in the scientific domain. Our benchmark enables fine-grained control over both explicit attributes (e.g., length), which are objective and straightforward, and implicit attributes (e.g., conceptual or empirical focus), which are more subjective and abstract. We conduct extensive experiments using various large language models (LLMs) under various settings, including in-context learning, parameter-efficient fine-tuning, and two-stage modular methods for balancing control over different attributes. Our findings reveal significant limitations in LLMs capabilities in balancing trade-offs between control attributes, especially implicit ones that require deeper understanding and abstract reasoning.