Benchmarking Counterfactual Interpretability in Deep Learning Models for Time Series Classification
This work addresses the need for standardized evaluation in interpretability for time series models, though it is incremental as it focuses on benchmarking rather than introducing new methods.
The paper tackled the lack of benchmarking for counterfactual interpretability methods in deep learning for time series classification by redesigning metrics and systematically evaluating 6 methods on 30 datasets with 3 classifiers, finding that performance varies across metrics and models.
The popularity of deep learning methods in the time series domain boosts interest in interpretability studies, including counterfactual (CF) methods. CF methods identify minimal changes in instances to alter the model predictions. Despite extensive research, no existing work benchmarks CF methods in the time series domain. Additionally, the results reported in the literature are inconclusive due to the limited number of datasets and inadequate metrics. In this work, we redesign quantitative metrics to accurately capture desirable characteristics in CFs. We specifically redesign the metrics for sparsity and plausibility and introduce a new metric for consistency. Combined with validity, generation time, and proximity, we form a comprehensive metric set. We systematically benchmark 6 different CF methods on 20 univariate datasets and 10 multivariate datasets with 3 different classifiers. Results indicate that the performance of CF methods varies across metrics and among different models. Finally, we provide case studies and a guideline for practical usage.