Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources
This work addresses a critical issue for policymakers and practitioners in social domains like human services and education, but it is incremental as it builds on existing targeting strategies with empirical evidence.
The paper tackles the problem of selecting individuals for limited-resource interventions by comparing risk-based targeting with treatment effect-based targeting using data from 5 real-world RCTs, finding that treatment effect targeting substantially outperforms risk-based targeting when accurate estimates are possible, even with biased estimates or normative preferences for higher-risk individuals.
Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. Policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of which individuals would benefit more from the intervention, while observational data creates a substantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed ``risk-based targeting" where the model is just used to predict each individual's status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effective machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that when treatment effects can be estimated with high accuracy (which we simulate by allowing the model to partially observe outcomes in advance), treatment effect based targeting substantially outperforms risk-based targeting, even when treatment effect estimates are biased. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. However, the features and data actually available in most RCTs we examine do not suffice for accurate estimates of heterogeneous treatment effects. Our results suggest treatment effect targeting has significant potential benefits, but realizing these benefits requires improvements to data collection and model training beyond what is currently common in practice.