POLAR: Automating Cyber Threat Prioritization through LLM-Powered Assessment
This addresses performance gaps in LLM-powered CTI systems for security analysts, though it is incremental as it builds on existing work to analyze failures rather than propose a new solution.
The paper investigated intrinsic vulnerabilities of Large Language Models (LLMs) in cyber threat intelligence (CTI), identifying spurious correlations, contradictory knowledge, and constrained generalization as key limitations through large-scale evaluations across benchmarks and real-world reports.
Large Language Models (LLMs) are intensively used to assist security analysts in counteracting the rapid exploitation of cyber threats, wherein LLMs offer cyber threat intelligence (CTI) to support vulnerability assessment and incident response. While recent work has shown that LLMs can support a wide range of CTI tasks such as threat analysis, vulnerability detection, and intrusion defense, significant performance gaps persist in practical deployments. In this paper, we investigate the intrinsic vulnerabilities of LLMs in CTI, focusing on challenges that arise from the nature of the threat landscape itself rather than the model architecture. Using large-scale evaluations across multiple CTI benchmarks and real-world threat reports, we introduce a novel categorization methodology that integrates stratification, autoregressive refinement, and human-in-the-loop supervision to reliably analyze failure instances. Through extensive experiments and human inspections, we reveal three fundamental vulnerabilities: spurious correlations, contradictory knowledge, and constrained generalization, that limit LLMs in effectively supporting CTI. Subsequently, we provide actionable insights for designing more robust LLM-powered CTI systems to facilitate future research.