On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions
This addresses the challenge of inconsistent interpretation of cyberattack descriptions by security professionals, which can impact cybersecurity operations and decisions, though it is incremental in applying existing NLP methods to this domain.
The paper tackled the problem of interpreting ambiguous cyberattack descriptions by comparing direct use of large language models (LLMs) like GPT-3.5 with supervised fine-tuning of small-scale LLMs like BERT for predicting ATT&CK tactics, finding that fine-tuned models provide clearer differentiation while direct LLMs offer broader interpretations but are limited by inherent ambiguity.
The volume, variety, and velocity of change in vulnerabilities and exploits have made incident threat analysis challenging with human expertise and experience along. Tactics, Techniques, and Procedures (TTPs) are to describe how and why attackers exploit vulnerabilities. However, a TTP description written by one security professional can be interpreted very differently by another, leading to confusion in cybersecurity operations or even business, policy, and legal decisions. Meanwhile, advancements in AI have led to the increasing use of Natural Language Processing (NLP) algorithms to assist the various tasks in cyber operations. With the rise of Large Language Models (LLMs), NLP tasks have significantly improved because of the LLM's semantic understanding and scalability. This leads us to question how well LLMs can interpret TTPs or general cyberattack descriptions to inform analysts of the intended purposes of cyberattacks. We propose to analyze and compare the direct use of LLMs (e.g., GPT-3.5) versus supervised fine-tuning (SFT) of small-scale-LLMs (e.g., BERT) to study their capabilities in predicting ATT&CK tactics. Our results reveal that the small-scale-LLMs with SFT provide a more focused and clearer differentiation between the ATT&CK tactics (if such differentiation exists). On the other hand, direct use of LLMs offer a broader interpretation of cyberattack techniques. When treating more general cases, despite the power of LLMs, inherent ambiguity exists and limits their predictive power. We then summarize the challenges and recommend research directions on LLMs to treat the inherent ambiguity of TTP descriptions used in various cyber operations.