No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference
For practitioners using PPI++ to combine gold-standard and noisy labels, this work clarifies when the method actually helps versus hurts, preventing over-reliance on asymptotic guarantees.
This paper provides a finite-sample analysis of Prediction-Powered Inference (PPI++), showing that contrary to asymptotic claims, PPI++ can have worse estimation error than using gold-standard labels alone when the correlation between pseudo- and gold-standard labels is below a threshold that depends on the number of labeled samples. For Gaussian data, the correlation must be at least 1/√(n-2) for improvement.
Prediction-Powered Inference (PPI) is a popular strategy for combining gold-standard and possibly noisy pseudo-labels to perform statistical estimation. Prior work has shown an asymptotic \enquote{free lunch} for PPI++, an adaptive form of PPI, showing that the \textit{asymptotic} variance of PPI++ is always less than or equal to the variance obtained from using gold-standard labels alone. Notably, this result holds \textit{regardless of the quality of the pseudo-labels}. In this work, we demystify this result by conducting an exact finite-sample analysis of the estimation error of PPI++ on the mean estimation problem. We give a \enquote{no free lunch} result, characterizing the settings (and sample sizes) where PPI++ has provably worse estimation error than using gold-standard labels alone. Specifically, PPI++ will outperform if and only if the correlation between pseudo- and gold-standard is above a certain level that depends on the number of labeled samples ($n$). In some cases our results simplify considerably: For Gaussian data, for instance, the correlation must be at least $1/\sqrt{n - 2}$ in order to see improvement. More broadly, by providing exact non-asymptotic expressions for the variance of PPI++ under sample splitting, we aim to empower practitioners to transparently reason about the benefits of PPI++ in specific applications. In experiments, we illustrate that our theoretical findings hold on real-world datasets.