Robust Sampling for Active Statistical Inference
This work addresses a critical bottleneck in AI-assisted data collection for researchers in computational social science and survey research, offering a robust solution to improve estimation accuracy.
The paper tackles the problem of noisy results in active statistical inference due to inaccurate uncertainty estimates by introducing robust sampling strategies that ensure estimators are never worse than uniform sampling and often outperform standard active inference when uncertainty estimates are reliable.
Active statistical inference is a new method for inference with AI-assisted data collection. Given a budget on the number of labeled data points that can be collected and assuming access to an AI predictive model, the basic idea is to improve estimation accuracy by prioritizing the collection of labels where the model is most uncertain. The drawback, however, is that inaccurate uncertainty estimates can make active sampling produce highly noisy results, potentially worse than those from naive uniform sampling. In this work, we present robust sampling strategies for active statistical inference. Robust sampling ensures that the resulting estimator is never worse than the estimator using uniform sampling. Furthermore, with reliable uncertainty estimates, the estimator usually outperforms standard active inference. This is achieved by optimally interpolating between uniform and active sampling, depending on the quality of the uncertainty scores, and by using ideas from robust optimization. We demonstrate the utility of the method on a series of real datasets from computational social science and survey research.