On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTs
This work addresses incident classification for SOCs and CSIRTs, but it is incremental as it applies existing methods to a specific domain.
The study compared open-source and proprietary models for security incident classification using real data and prompt engineering, finding that proprietary models had higher accuracy but open-source models offered benefits in privacy, cost, and data sovereignty.
In this study, we evaluate open-source models for security incident classification, comparing them with proprietary models. We utilize a dataset of anonymized real incidents, categorized according to the NIST SP 800-61r3 taxonomy and processed using five prompt-engineering techniques (PHP, SHP, HTP, PRP, and ZSL). The results indicate that, although proprietary models still exhibit higher accuracy, locally deployed open-source models provide advantages in privacy, cost-effectiveness, and data sovereignty.