Improving the Performance of the LSTM and HMM Model via Hybridization
This is an incremental analysis for researchers in language modeling, focusing on model comparison rather than new applications.
The paper tackled the problem of comparing language models by analyzing the hidden state structures of HMM and LSTM models, finding that the simpler HMM can approximate the LSTM effectively.
Language models based on deep neural networks and traditional stochastic modelling have become both highly functional and effective in recent times. In this work, a general survey into the two types of language modelling is conducted. We investigate the effectiveness of the Hidden Markov Model (HMM), and the Long Short-Term Memory Model (LSTM). We analyze the hidden state structures common to both models, and present an analysis on structural similarity of the hidden states, common to both HMM's and LSTM's. We compare the LSTM's predictive accuracy and hidden state output with respect to the HMM for a varying number of hidden states. In this work, we justify that the less complex HMM can serve as an appropriate approximation of the LSTM model.