Evaluating the impact of an explainable machine learning system on the interobserver agreement in chest radiograph interpretation
This addresses the problem of variability in radiologist interpretations for clinical decision-making, though it is an incremental improvement.
The study evaluated an explainable AI system (VinDr-CXR) for chest radiograph interpretation and found it improved interobserver agreement among six radiologists by 1.5% in mean Fleiss' Kappa, while agreement between radiologists and the system increased by 3.3% in mean Cohen's Kappa after consulting AI suggestions.
We conducted a prospective study to measure the clinical impact of an explainable machine learning system on interobserver agreement in chest radiograph interpretation. The AI system, which we call as it VinDr-CXR when used as a diagnosis-supporting tool, significantly improved the agreement between six radiologists with an increase of 1.5% in mean Fleiss' Kappa. In addition, we also observed that, after the radiologists consulted AI's suggestions, the agreement between each radiologist and the system was remarkably increased by 3.3% in mean Cohen's Kappa. This work has been accepted for publication in IEEE Access and this paper is our short version submitted to the Midwest Machine Learning Symposium (MMLS 2023), Chicago, IL, USA.