Finite-Sample Maximum Likelihood Estimation of Location
This addresses a theoretical gap in statistical estimation for finite samples, which is incremental but relevant for statisticians and machine learning practitioners dealing with non-asymptotic scenarios.
The paper tackles the problem of finite-sample maximum likelihood estimation for location parameters, showing that a theory similar to asymptotic optimality can be recovered by using the Fisher information of a smoothed version of the distribution, with smoothing radius decaying with sample size.
We consider 1-dimensional location estimation, where we estimate a parameter $λ$ from $n$ samples $λ+ η_i$, with each $η_i$ drawn i.i.d. from a known distribution $f$. For fixed $f$ the maximum-likelihood estimate (MLE) is well-known to be optimal in the limit as $n \to \infty$: it is asymptotically normal with variance matching the Cramér-Rao lower bound of $\frac{1}{n\mathcal{I}}$, where $\mathcal{I}$ is the Fisher information of $f$. However, this bound does not hold for finite $n$, or when $f$ varies with $n$. We show for arbitrary $f$ and $n$ that one can recover a similar theory based on the Fisher information of a smoothed version of $f$, where the smoothing radius decays with $n$.