SMACE: A New Method for the Interpretability of Composite Decision Systems
This addresses interpretability for business processes using composite decision systems, but it is incremental as it builds on existing interpretability methods.
The paper tackles the problem of explaining decisions in composite systems that combine multiple models and decision rules, proposing SMACE to generate intuitive feature rankings for end users, and shows it outperforms existing model-agnostic methods by providing meaningful feature rankings where others fail.
Interpretability is a pressing issue for decision systems. Many post hoc methods have been proposed to explain the predictions of a single machine learning model. However, business processes and decision systems are rarely centered around a unique model. These systems combine multiple models that produce key predictions, and then apply decision rules to generate the final decision. To explain such decisions, we propose the Semi-Model-Agnostic Contextual Explainer (SMACE), a new interpretability method that combines a geometric approach for decision rules with existing interpretability methods for machine learning models to generate an intuitive feature ranking tailored to the end user. We show that established model-agnostic approaches produce poor results on tabular data in this setting, in particular giving the same importance to several features, whereas SMACE can rank them in a meaningful way.