Jared Smith

Sean Oesch, Robert Bridges, Jared Smith et al.

Gartner, a large research and advisory company, anticipates that by 2024 80% of security operation centers (SOCs) will use machine learning (ML) based solutions to enhance their operations. In light of such widespread adoption, it is vital for the research community to identify and address usability concerns. This work presents the results of the first in situ usability assessment of ML-based tools. With the support of the US Navy, we leveraged the national cyber range, a large, air-gapped cyber testbed equipped with state-of-the-art network and user emulation capabilities, to study six US Naval SOC analysts' usage of two tools. Our analysis identified several serious usability issues, including multiple violations of established usability heuristics form user interface design. We also discovered that analysts lacked a clear mental model of how these tools generate scores, resulting in mistrust and/or misuse of the tools themselves. Surprisingly, we found no correlation between analysts' level of education or years of experience and their performance with either tool, suggesting that other factors such as prior background knowledge or personality play a significant role in ML-based tool usage. Our findings demonstrate that ML-based security tool vendors must put a renewed focus on working with analysts, both experienced and inexperienced, to ensure that their systems are usable and useful in real-world security operations settings.

Jordan Holland, Jared Smith, Max Schuchard

We examine the extent of needless traffic exposure by the routing infrastructure to nations geographically irrelevant to packet transmission. We quantify what countries are geographically logical to observe on a network path traveling between two nations through the use of convex hulls circumscribing major population centers. We then compare that to the nation states observed in over 2.5 billion measured paths. We examine both the entire geographic topology of the Internet and a subset of the topology that a Tor user would typically interact with. We find that 44% of paths across the entire geographic topology of the Internet and 33% of paths in the user experience subset unnecessarily expose traffic to one or more nations. Finally, we consider the scenario where countries exercise both legal and physical control over autonomous systems, gaining access to traffic outside of their geographic borders, but carried by organizations that fall under the AS's registered country's legal jurisdiction. At least 49% of paths in both measurements expose traffic to a geographically irrelevant country when considering both the physical and legal countries that a path traverses.