ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects
This provides a domain-specific benchmark for researchers evaluating causal inference methods in healthcare, but it is incremental as it builds on existing simulator-based approaches.
The authors tackled the lack of realistic benchmarks for causal effect estimation in healthcare by proposing a simulator of Alzheimer's disease, which models complexities like latent variables and effect heterogeneity, and they used it to compare estimators, though no concrete performance numbers were provided.
Simulators make unique benchmarks for causal effect estimation since they do not rely on unverifiable assumptions or the ability to intervene on real-world systems, but are often too simple to capture important aspects of real applications. We propose a simulator of Alzheimer's disease aimed at modeling intricacies of healthcare data while enabling benchmarking of causal effect and policy estimators. We fit the system to the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and ground hand-crafted components in results from comparative treatment trials and observational treatment patterns. The simulator includes parameters which alter the nature and difficulty of the causal inference tasks, such as latent variables, effect heterogeneity, length of observed history, behavior policy and sample size. We use the simulator to compare estimators of average and conditional treatment effects.