Intrusion Detection at Scale with the Assistance of a Command-line Language Model
This addresses the need for scalable, automated intrusion detection in enterprise security, moving beyond hand-crafted rules to reduce false negatives and improve generalization to new attacks.
The paper tackles the problem of intrusion detection at scale by introducing a system that uses large-scale pre-training on tens of millions of command lines to train a language model, achieving effectiveness verified on 30 million training and 10 million test samples.
Intrusion detection is a long standing and crucial problem in security. A system capable of detecting intrusions automatically is on great demand in enterprise security solutions. Existing solutions rely heavily on hand-crafted rules designed by security operators, which suffer from high false negative rates and poor generalization ability to new, zero-day attacks at scale. AI and machine learning offer promising solutions to address the issues, by inspecting abnormal user behaviors intelligently and automatically from data. However, existing learning-based intrusion detection systems in the literature are mostly designed for small data, and they lack the ability to leverage the power of big data in cloud environments. In this paper, we target at this problem and introduce an intrusion detection system which incorporates large-scale pre-training, so as to train a large language model based on tens of millions of command lines for AI-based intrusion detection. Experiments performed on 30 million training samples and 10 million test samples verify the effectiveness of our solution.