A Sober Look at Spectral Learning
This work questions the practical effectiveness of spectral learning for machine learning practitioners, highlighting a gap between theory and practice in estimation methods.
The paper empirically compares spectral learning (SL) and expectation maximization (EM) for latent variable models, finding that SL often produces negative probabilities while EM yields better estimates without getting stuck in local optima, challenging theoretical assumptions.
Spectral learning recently generated lots of excitement in machine learning, largely because it is the first known method to produce consistent estimates (under suitable conditions) for several latent variable models. In contrast, maximum likelihood estimates may get trapped in local optima due to the non-convex nature of the likelihood function of latent variable models. In this paper, we do an empirical evaluation of spectral learning (SL) and expectation maximization (EM), which reveals an important gap between the theory and the practice. First, SL often leads to negative probabilities. Second, EM often yields better estimates than spectral learning and it does not seem to get stuck in local optima. We discuss how the rank of the model parameters and the amount of training data can yield negative probabilities. We also question the common belief that maximum likelihood estimators are necessarily inconsistent.