Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Models
This work provides refined theoretical insights for statisticians and machine learning researchers working on mixture models, though it is incremental as it builds on existing Wasserstein distance methods.
The paper tackles the problem of deriving convergence rates for maximum likelihood estimation in finite mixture models, showing that penalizing the log-likelihood to discourage vanishing weights leads to new loss functions that capture heterogeneous convergence rates, with a subset of components converging significantly faster than previously known.
We revisit the classical problem of deriving convergence rates for the maximum likelihood estimator (MLE) in finite mixture models. The Wasserstein distance has become a standard loss function for the analysis of parameter estimation in these models, due in part to its ability to circumvent label switching and to accurately characterize the behaviour of fitted mixture components with vanishing weights. However, the Wasserstein distance is only able to capture the worst-case convergence rate among the remaining fitted mixture components. We demonstrate that when the log-likelihood function is penalized to discourage vanishing mixing weights, stronger loss functions can be derived to resolve this shortcoming of the Wasserstein distance. These new loss functions accurately capture the heterogeneity in convergence rates of fitted mixture components, and we use them to sharpen existing pointwise and uniform convergence rates in various classes of mixture models. In particular, these results imply that a subset of the components of the penalized MLE typically converge significantly faster than could have been anticipated from past work. We further show that some of these conclusions extend to the traditional MLE. Our theoretical findings are supported by a simulation study to illustrate these improved convergence rates.