Using ScrutinAI for Visual Inspection of DNN Performance in a Medical Use Case
This addresses the challenge of costly and variable expert labeling in medical AI, though it is incremental as it applies an existing tool to a specific use case.
The researchers tackled the problem of distinguishing DNN weaknesses due to label variations from true model flaws in medical imaging, using their Visual Analytics tool ScrutinAI on an intracranial hemorrhage dataset, and found it facilitated root cause analysis.
Our Visual Analytics (VA) tool ScrutinAI supports human analysts to investigate interactively model performanceand data sets. Model performance depends on labeling quality to a large extent. In particular in medical settings, generation of high quality labels requires in depth expert knowledge and is very costly. Often, data sets are labeled by collecting opinions of groups of experts. We use our VA tool to analyse the influence of label variations between different experts on the model performance. ScrutinAI facilitates to perform a root cause analysis that distinguishes weaknesses of deep neural network (DNN) models caused by varying or missing labeling quality from true weaknesses. We scrutinize the overall detection of intracranial hemorrhages and the more subtle differentiation between subtypes in a publicly available data set.