A study of latent monotonic attention variants
This addresses convergence and efficiency problems in speech recognition for researchers and practitioners, but it is incremental as it builds on existing attention methods.
The paper tackled the problem of non-monotonic attention in end-to-end speech recognition models, which causes convergence, stability, generalization, and efficiency issues, by introducing a mathematically clean solution using a latent variable for monotonicity. The result showed that their monotonic models performed as well as the global soft attention baseline on the Switchboard 300h dataset.
End-to-end models reach state-of-the-art performance for speech recognition, but global soft attention is not monotonic, which might lead to convergence problems, to instability, to bad generalisation, cannot be used for online streaming, and is also inefficient in calculation. Monotonicity can potentially fix all of this. There are several ad-hoc solutions or heuristics to introduce monotonicity, but a principled introduction is rarely found in literature so far. In this paper, we present a mathematically clean solution to introduce monotonicity, by introducing a new latent variable which represents the audio position or segment boundaries. We compare several monotonic latent models to our global soft attention baseline such as a hard attention model, a local windowed soft attention model, and a segmental soft attention model. We can show that our monotonic models perform as good as the global soft attention model. We perform our experiments on Switchboard 300h. We carefully outline the details of our training and release our code and configs.