Spectral M-estimation with Applications to Hidden Markov Models
This work addresses sample efficiency and robustness issues in nonconvex estimation for practitioners in statistics and machine learning, though it appears incremental as it builds on existing spectral and M-estimation frameworks.
The authors tackled the problem of method of moment estimators being sample-inefficient and sensitive to misspecification by developing an M-estimator that achieves optimal sample efficiency rates and maintains prediction accuracy comparable to other spectral estimators, with empirical gains demonstrated on hidden Markov models.
Method of moment estimators exhibit appealing statistical properties, such as asymptotic unbiasedness, for nonconvex problems. However, they typically require a large number of samples and are extremely sensitive to model misspecification. In this paper, we apply the framework of M-estimation to develop both a generalized method of moments procedure and a principled method for regularization. Our proposed M-estimator obtains optimal sample efficiency rates (in the class of moment-based estimators) and the same well-known rates on prediction accuracy as other spectral estimators. It also makes it straightforward to incorporate regularization into the sample moment conditions. We demonstrate empirically the gains in sample efficiency from our approach on hidden Markov models.