Revisiting Active Sequential Prediction-Powered Mean Estimation
For practitioners using prediction-powered estimation, this work reveals that simple constant query probabilities can outperform adaptive uncertainty-based methods, challenging prior assumptions.
This paper revisits active sequential prediction-powered mean estimation, finding that the optimal query strategy often reduces to a constant probability rather than using uncertainty-based sampling. The authors provide a non-asymptotic analysis and show that a no-regret learning approach converges to this constant probability, with simulations confirming the theoretical results.
In this work, we revisit the problem of active sequential prediction-powered mean estimation, where at each round one must decide the query probability of the ground-truth label upon observing the covariates of a sample. Furthermore, if the label is not queried, the prediction from a machine learning model is used instead. Prior work proposed an elegant scheme that determines the query probability by combining an uncertainty-based suggestion with a constant probability that encodes a soft constraint on the query probability. We explored different values of the mixing parameter and observed an intriguing empirical pattern: the smallest confidence width tends to occur when the weight on the constant probability is close to one, thereby reducing the influence of the uncertainty-based component. Motivated by this observation, we develop a non-asymptotic analysis of the estimator and establish a data-dependent bound on its confidence interval. Our analysis further suggests that when a no-regret learning approach is used to determine the query probability and control this bound, the query probability converges to the constraint of the max value of the query probability when it is chosen obliviously to the current covariates. We also conduct simulations that corroborate these theoretical findings.