Cry Wolf: Toward an Experimentation Platform and Dataset for Human Factors in Cyber Security Analysis
This work addresses the problem of limited experimentation platforms and datasets for cybersecurity analysts, though it is incremental as it builds on existing tools and theories.
The paper tackles the challenge of studying human factors in cybersecurity analysis by introducing Cry Wolf, an open-source web application for user studies, and providing a dataset of 73 true and false IDS alarms. Results from an experiment using these tools show that many alerts in the dataset offer good discrimination for participants' task performance, as measured by difficulty and discrimination indices.
Computer network defense is a partnership between automated systems and human cyber security analysts. The system behaviors, for example raising a high proportion of false alarms, likely impact cyber analyst performance. Experimentation in the analyst-system domain is challenging due to lack of access to security experts, the usability of attack datasets, and the training required to use security analysis tools. This paper describes Cry Wolf, an open source web application for user studies of cyber security analysis tasks. This paper also provides an open-access dataset of 73 true and false Intrusion Detection System (IDS) alarms derived from real-world examples of "impossible travel" scenarios. Cry Wolf and the impossible travel dataset were used in an experiment on the impact of IDS false alarm rate on analysts' abilities to correctly classify IDS alerts as true or false alarms. Results from that experiment are used to evaluate the quality of the dataset using difficulty and discrimination index measures drawn from classical test theory. Many alerts in the dataset provide good discrimination for participants' overall task performance.