An Empirical Framework for Domain Generalization in Clinical Settings
This work addresses the challenge of degraded model performance across new hospitals or populations in clinical settings, but it is incremental as it benchmarks existing methods and provides recommendations without introducing new techniques.
The authors tackled the problem of domain generalization in clinical machine learning by benchmarking eight methods on multi-site clinical time series and medical imaging data, finding that these methods did not consistently improve out-of-distribution performance over standard training on medical imaging, but showed limited gains in some realistic induced-shift scenarios for time series data.
Clinical machine learning models experience significantly degraded performance in datasets not seen during training, e.g., new hospitals or populations. Recent developments in domain generalization offer a promising solution to this problem by creating models that learn invariances across environments. In this work, we benchmark the performance of eight domain generalization methods on multi-site clinical time series and medical imaging data. We introduce a framework to induce synthetic but realistic domain shifts and sampling bias to stress-test these methods over existing non-healthcare benchmarks. We find that current domain generalization methods do not consistently achieve significant gains in out-of-distribution performance over empirical risk minimization on real-world medical imaging data, in line with prior work on general imaging datasets. However, a subset of realistic induced-shift scenarios in clinical time series data do exhibit limited performance gains. We characterize these scenarios in detail, and recommend best practices for domain generalization in the clinical setting.