Orthogonality Constrained Multi-Head Attention For Keyword Spotting
This work addresses a specific bottleneck in keyword spotting systems by enhancing multi-head attention to reduce redundancy, which is incremental but targeted at improving efficiency in speech recognition applications.
The paper tackles the problem of redundancy in multi-head attention mechanisms for keyword spotting by proposing a regularization technique that penalizes positional and contextual non-orthogonality between attention heads, resulting in a significant performance improvement for the keyword 'Hey Snapdragon'.
Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences, it retrieves richer information than a single-head attention which only summarizes the whole sequence into one context vector. However, a naive use of the multi-head attention does not guarantee such richness as the attention heads may have positional and representational redundancy. In this paper, we propose a regularization technique for multi-head attention mechanism in an end-to-end neural keyword spotting system. Augmenting regularization terms which penalize positional and contextual non-orthogonality between the attention heads encourages to output different representations from separate subsequences, which in turn enables leveraging structured information without explicit sequence models such as hidden Markov models. In addition, intra-head contextual non-orthogonality regularization encourages each attention head to have similar representations across keyword examples, which helps classification by reducing feature variability. The experimental results demonstrate that the proposed regularization technique significantly improves the keyword spotting performance for the keyword "Hey Snapdragon".