On Automating Security Policies with Contemporary LLMs
This work addresses the need for more robust and adaptive security enforcement for organizations facing sophisticated cyber threats, but it is incremental as it builds on existing LLM and RAG techniques.
The paper tackles the problem of automating security policy enforcement in complex computing environments by presenting a framework that uses large language models (LLMs) with in-context learning and retrieval-augmented generation (RAG), resulting in significant improvements in precision, recall, and F1-score compared to a non-RAG baseline.
The complexity of modern computing environments and the growing sophistication of cyber threats necessitate a more robust, adaptive, and automated approach to security enforcement. In this paper, we present a framework leveraging large language models (LLMs) for automating attack mitigation policy compliance through an innovative combination of in-context learning and retrieval-augmented generation (RAG). We begin by describing how our system collects and manages both tool and API specifications, storing them in a vector database to enable efficient retrieval of relevant information. We then detail the architectural pipeline that first decomposes high-level mitigation policies into discrete tasks and subsequently translates each task into a set of actionable API calls. Our empirical evaluation, conducted using publicly available CTI policies in STIXv2 format and Windows API documentation, demonstrates significant improvements in precision, recall, and F1-score when employing RAG compared to a non-RAG baseline.