Fanoos: Multi-Resolution, Multi-Strength, Interactive Explanations for Learned Systems
It addresses the need for digestible explanations for diverse stakeholders in safety-critical applications, though it appears incremental by building on existing techniques.
The paper tackles the problem of limited explainability in learned systems by introducing Fanoos, a framework that combines formal verification, heuristic search, and user interaction to generate adjustable explanations, demonstrated on a learned controller for an inverted double pendulum and a CPU usage model.
Machine learning is becoming increasingly important to control the behavior of safety and financially critical components in sophisticated environments, where the inability to understand learned components in general, and neural nets in particular, poses serious obstacles to their adoption. Explainability and interpretability methods for learned systems have gained considerable academic attention, but the focus of current approaches on only one aspect of explanation, at a fixed level of abstraction, and limited if any formal guarantees, prevents those explanations from being digestible by the relevant stakeholders (e.g., end users, certification authorities, engineers) with their diverse backgrounds and situation-specific needs. We introduce Fanoos, a framework for combining formal verification techniques, heuristic search, and user interaction to explore explanations at the desired level of granularity and fidelity. We demonstrate the ability of Fanoos to produce and adjust the abstractness of explanations in response to user requests on a learned controller for an inverted double pendulum and on a learned CPU usage model.