Batch-Adaptive Annotations for Causal Inference with Complex-Embedded Outcomes
This work addresses the problem of causal inference with complex-embedded outcomes for policymakers and decision-makers, particularly in resource-constrained settings such as healthcare or social services.
The authors tackled the problem of estimating causal effects with missing or erroneous outcome information, achieving efficient average treatment effect estimation with missing data by allocating data annotation in batches, with demonstrated versatility on simulated and real-world datasets. The authors' method combines expert labels and noisy imputed labels into a doubly robust causal estimator.
Estimating the causal effects of an intervention on outcomes is crucial to policy and decision-making. But often, information about outcomes can be missing or subject to non-standard measurement error. It may be possible to reveal ground-truth outcome information at a cost, for example via data annotation or follow-up; but budget constraints entail that only a fraction of the dataset can be labeled. In this setting, we optimize which data points should be sampled for outcome information and, therefore, efficient average treatment effect estimation with missing data. We do so by allocating data annotation in batches. We extend to settings where outcomes may be recorded in unstructured data that can be annotated at a cost, such as text or images, for example, in healthcare or social services. Our motivating application is a collaboration with a street outreach provider with millions of case notes, where it is possible to expertly label some, but not all, ground-truth outcomes. We demonstrate how expert labels and noisy imputed labels can be combined efficiently and responsibly into a doubly robust causal estimator. We run experiments on simulated data and two real-world datasets, including one on street outreach interventions in homelessness services, to show the versatility of our proposed method.