Toward Cybersecurity-Expert Small Language Models
This addresses the problem of deploying AI in cybersecurity for practitioners by providing efficient, expert-level models, though it is incremental as it builds on existing small language model approaches with domain-specific enhancements.
The paper tackles the lack of high-quality, domain-specific models in cybersecurity by presenting CyberPal 2.0, a family of small language models (4B-20B parameters) that outperform baselines and match or surpass various frontier models on cybersecurity benchmarks, with the 20B model ranking first on threat-investigation tasks and the 4B model ranking second.
Large language models (LLMs) are transforming everyday applications, yet deployment in cybersecurity lags due to a lack of high-quality, domain-specific models and training datasets. To address this gap, we present CyberPal 2.0, a family of cybersecurity-expert small language models (SLMs) ranging from 4B-20B parameters. To train CyberPal 2.0, we generate an enriched chain-of-thought cybersecurity instruction dataset built with our data enrichment and formatting pipeline, SecKnowledge 2.0, which integrates expert-in-the-loop steering of reasoning formats alongside LLM-driven multi-step grounding, yielding higher-fidelity, task-grounded reasoning traces for security tasks. Across diverse cybersecurity benchmarks, CyberPal 2.0 consistently outperforms its baselines and matches or surpasses various open and closed-source frontier models, while remaining a fraction of their size. On core cyber threat intelligence knowledge tasks, our models outperform almost all tested frontier models, ranking second only to Sec-Gemini v1. On core threat-investigation tasks, such as correlating vulnerabilities and bug tickets with weaknesses, our best 20B-parameter model outperforms GPT-4o, o1, o3-mini, and Sec-Gemini v1, ranking first, while our smallest 4B-parameter model ranks second.