Uncertainty estimation for classification and risk prediction on medical tabular data
This work addresses uncertainty estimation to improve decision support tools and user trust in healthcare, where data is scarce and predictions involve rare conditions, but it is incremental as it refines existing heuristics and compares known methods.
The paper tackled the problem of uncertainty estimation for classification and risk prediction on medical tabular data by expanding heuristics for selecting techniques and comparing methods like ensembles and auto-encoders, finding that auto-encoders outperform ensembles in detecting out-of-domain examples.
In a data-scarce field such as healthcare, where models often deliver predictions on patients with rare conditions, the ability to measure the uncertainty of a model's prediction could potentially lead to improved effectiveness of decision support tools and increased user trust. This work advances the understanding of uncertainty estimation for classification and risk prediction on medical tabular data, in a two-fold way. First, we expand and refine the set of heuristics to select an uncertainty estimation technique, introducing tests for clinically-relevant scenarios such as generalization to uncommon pathologies, changes in clinical protocol and simulations of corrupted data. We furthermore differentiate these heuristics depending on the clinical use-case. Second, we observe that ensembles and related techniques perform poorly when it comes to detecting out-of-domain examples, a critical task which is carried out more successfully by auto-encoders. These remarks are enriched by considerations of the interplay of uncertainty estimation with class imbalance, post-modeling calibration and other modeling procedures. Our findings are supported by an array of experiments on toy and real-world data.