Learning Populations of Parameters
This provides a foundational improvement in statistical estimation for population-level parameter analysis, applicable across domains like social sciences and biology.
The paper tackles the problem of estimating the histogram of unknown parameters from binomial observations, showing that with sufficiently many entities, error can be reduced from Θ(1/√t) to O(1/t), which is information-theoretically optimal, and demonstrates practical performance on datasets from politics, sports analytics, and gender ratios.
Consider the following estimation problem: there are $n$ entities, each with an unknown parameter $p_i \in [0,1]$, and we observe $n$ independent random variables, $X_1,\ldots,X_n$, with $X_i \sim $ Binomial$(t, p_i)$. How accurately can one recover the "histogram" (i.e. cumulative density function) of the $p_i$'s? While the empirical estimates would recover the histogram to earth mover distance $Θ(\frac{1}{\sqrt{t}})$ (equivalently, $\ell_1$ distance between the CDFs), we show that, provided $n$ is sufficiently large, we can achieve error $O(\frac{1}{t})$ which is information theoretically optimal. We also extend our results to the multi-dimensional parameter case, capturing settings where each member of the population has multiple associated parameters. Beyond the theoretical results, we demonstrate that the recovery algorithm performs well in practice on a variety of datasets, providing illuminating insights into several domains, including politics, sports analytics, and variation in the gender ratio of offspring.