CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarization
This addresses a domain-specific problem for cybersecurity professionals by providing a dataset to improve summarization of CTI reports, though it is incremental as it focuses on data creation rather than novel methods.
The authors tackled the lack of suitable datasets for Cyber Threat Intelligence (CTI) summarization by introducing CTISum, a new benchmark dataset, and found that current state-of-the-art models face significant challenges on it, indicating that automatic CTI summarization remains an open problem.
Cyber Threat Intelligence (CTI) summarization involves generating concise and accurate highlights from web intelligence data, which is critical for providing decision-makers with actionable insights to swiftly detect and respond to cyber threats in the cybersecurity domain. Despite that, the development of efficient techniques for summarizing CTI reports, comprising facts, analytical insights, attack processes, and more, has been hindered by the lack of suitable datasets. To address this gap, we introduce CTISum, a new benchmark dataset designed for the CTI summarization task. Recognizing the significance of understanding attack processes, we also propose a novel fine-grained subtask: attack process summarization, which aims to help defenders assess risks, identify security gaps, and uncover vulnerabilities. Specifically, a multi-stage annotation pipeline is designed to collect and annotate CTI data from diverse web sources, alongside a comprehensive benchmarking of CTISum using both extractive, abstractive and LLMs-based summarization methods. Experimental results reveal that current state-of-the-art models face significant challenges when applied to CTISum, highlighting that automatic summarization of CTI reports remains an open research problem. The code and example dataset can be made publicly available at https://github.com/pengwei-iie/CTISum.