Forming IDEAS Interactive Data Exploration & Analysis System
This addresses the challenge for cybersecurity analysts in detecting advanced attacks, though it appears incremental as it builds on existing data science approaches.
The paper tackles the problem of insufficient analytical tools for cybersecurity analysts by developing IDEAS, an interactive data exploration and analysis system, which aims to improve work efficiency and support evidence-based decisions through easy-to-use data science tools.
Modern cyber security operations collect an enormous amount of logging and alerting data. While analysts have the ability to query and compute simple statistics and plots from their data, current analytical tools are too simple to admit deep understanding. To detect advanced and novel attacks, analysts turn to manual investigations. While commonplace, current investigations are time-consuming, intuition-based, and proving insufficient. Our hypothesis is that arming the analyst with easy-to-use data science tools will increase their work efficiency, provide them with the ability to resolve hypotheses with scientific inquiry of their data, and support their decisions with evidence over intuition. To this end, we present our work to build IDEAS (Interactive Data Exploration and Analysis System). We present three real-world use-cases that drive the system design from the algorithmic capabilities to the user interface. Finally, a modular and scalable software architecture is discussed along with plans for our pilot deployment with a security operation command.