CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays
This addresses safety concerns for clinical deployment of AI models trained on non-exhaustive disease sets, though it is incremental as it systematically tests existing methods on new scenarios.
The study evaluated deep learning models for chest X-ray interpretation when diseases not present during training (unseen diseases) appear, finding that models often misclassify unseen diseases as 'no disease' but can still detect seen diseases when co-occurring with unseen ones, and that penultimate layer features aid in detecting unseen diseases with small labeled data.
We systematically evaluate the performance of deep learning models in the presence of diseases not labeled for or present during training. First, we evaluate whether deep learning models trained on a subset of diseases (seen diseases) can detect the presence of any one of a larger set of diseases. We find that models tend to falsely classify diseases outside of the subset (unseen diseases) as "no disease". Second, we evaluate whether models trained on seen diseases can detect seen diseases when co-occurring with diseases outside the subset (unseen diseases). We find that models are still able to detect seen diseases even when co-occurring with unseen diseases. Third, we evaluate whether feature representations learned by models may be used to detect the presence of unseen diseases given a small labeled set of unseen diseases. We find that the penultimate layer of the deep neural network provides useful features for unseen disease detection. Our results can inform the safe clinical deployment of deep learning models trained on a non-exhaustive set of disease classes.