Principal Sensitivity Analysis
This provides a method for interpreting classifier behavior, which is an incremental improvement in explainable AI for researchers and practitioners.
The paper tackles the problem of analyzing what knowledge classifiers learn by introducing Principal Sensitivity Analysis (PSA), which identifies input directions to which classifiers are most sensitive, and demonstrates its ability to decompose classifier knowledge through visualizations on artificial and real data.
We present a novel algorithm (Principal Sensitivity Analysis; PSA) to analyze the knowledge of the classifier obtained from supervised machine learning techniques. In particular, we define principal sensitivity map (PSM) as the direction on the input space to which the trained classifier is most sensitive, and use analogously defined k-th PSM to define a basis for the input space. We train neural networks with artificial data and real data, and apply the algorithm to the obtained supervised classifiers. We then visualize the PSMs to demonstrate the PSA's ability to decompose the knowledge acquired by the trained classifiers.