Understanding Text Classification Data and Models Using Aggregated Input Salience
This work addresses the need for model developers to efficiently diagnose and improve model reliability in text classification, though it is incremental in automating existing manual analysis.
The paper tackles the challenge of identifying when text classification models rely on spurious patterns by proposing a method to aggregate input salience across datasets, enabling detection of problematic data and model behaviors.
Realizing when a model is right for a wrong reason is not trivial and requires a significant effort by model developers. In some cases an input salience method, which highlights the most important parts of the input, may reveal problematic reasoning. But scrutinizing highlights over many data instances is tedious and often infeasible. Furthermore, analyzing examples in isolation does not reveal general patterns in the data or in the model's behavior. In this paper we aim to address these issues and go from understanding single examples to understanding entire datasets and models. The methodology we propose is based on aggregated salience maps, to which we apply clustering, nearest neighbor search and visualizations. Using this methodology we address multiple distinct but common model developer needs by showing how problematic data and model behavior can be identified and explained -- a necessary first step for improving the model.