More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool
This addresses practical usability issues for cybersecurity analysts, but it is incremental as it builds on existing XAI critiques without presenting new methods.
The paper tackled the challenge of integrating Explainable AI (XAI) into cybersecurity workflows, finding that state-of-the-art saliency techniques like SHAP or LIME are ineffective for non-technical users and disrupt real-time analysis, and it suggests that higher-level explanations or LLMs could help.
We share observations and challenges from an ongoing effort to implement Explainable AI (XAI) in a domain-specific workflow for cybersecurity analysts. Specifically, we briefly describe a preliminary case study on the use of XAI for source code classification, where accurate assessment and timeliness are paramount. We find that the outputs of state-of-the-art saliency explanation techniques (e.g., SHAP or LIME) are lost in translation when interpreted by people with little AI expertise, despite these techniques being marketed for non-technical users. Moreover, we find that popular XAI techniques offer fewer insights for real-time human-AI workflows when they are post hoc and too localized in their explanations. Instead, we observe that cyber analysts need higher-level, easy-to-digest explanations that can offer as little disruption as possible to their workflows. We outline unaddressed gaps in practical and effective XAI, then touch on how emerging technologies like Large Language Models (LLMs) could mitigate these existing obstacles.