Making deep neural networks right for the right scientific reasons by interacting with their explanations
This addresses the issue of unreliable AI models in scientific domains like plant phenotyping, though it appears incremental as it builds on existing interactive learning and explanation methods.
The paper tackles the problem of deep neural networks using confounding factors in datasets by introducing explanatory interactive learning (XIL), where scientists provide feedback on model explanations during training. Results show XIL helps avoid 'Clever Hans' behavior and appropriately influences trust in models.
Deep neural networks have shown excellent performances in many real-world applications. Unfortunately, they may show "Clever Hans"-like behavior -- making use of confounding factors within datasets -- to achieve high performance. In this work, we introduce the novel learning setting of "explanatory interactive learning" (XIL) and illustrate its benefits on a plant phenotyping research task. XIL adds the scientist into the training loop such that she interactively revises the original model via providing feedback on its explanations. Our experimental results demonstrate that XIL can help avoiding Clever Hans moments in machine learning and encourages (or discourages, if appropriate) trust into the underlying model.