Bivariate Beta-LSTM
This work addresses a specific bottleneck in LSTM design for researchers and practitioners in sequence modeling, offering a novel method to improve gate flexibility and correlation modeling, though it is incremental as it builds on existing LSTM frameworks.
The paper tackled the inflexibility of sigmoid gates in LSTMs by proposing a bivariate Beta distribution gate structure, which allows probabilistic modeling and customization of cell state flow, and demonstrated effectiveness across tasks like sentence classification and image caption generation with observed higher gradient values in training.
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing multi-modality or skewness. Besides, the previous models lack modeling on the correlation between the gates, which would be a new method to adopt inductive bias for a relationship between previous and current input. This paper proposes a new gate structure with the bivariate Beta distribution. The proposed gate structure enables probabilistic modeling on the gates within the LSTM cell so that the modelers can customize the cell state flow with priors and distributions. Moreover, we theoretically show the higher upper bound of the gradient compared to the sigmoid function, and we empirically observed that the bivariate Beta distribution gate structure provides higher gradient values in training. We demonstrate the effectiveness of bivariate Beta gate structure on the sentence classification, image classification, polyphonic music modeling, and image caption generation.