Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognition
This addresses the issue of attention head redundancy in speech recognition systems, which is an incremental improvement for enhancing model efficiency and performance.
The paper tackled the problem of attention heads becoming highly correlated during training in Conformer-based automatic speech recognition, and found that introducing diversity-promoting auxiliary loss functions improved word error rates by up to 6% relative on the Librispeech corpus.
Attention layers are an integral part of modern end-to-end automatic speech recognition systems, for instance as part of the Transformer or Conformer architecture. Attention is typically multi-headed, where each head has an independent set of learned parameters and operates on the same input feature sequence. The output of multi-headed attention is a fusion of the outputs from the individual heads. We empirically analyze the diversity between representations produced by the different attention heads and demonstrate that the heads become highly correlated during the course of training. We investigate a few approaches to increasing attention head diversity, including using different attention mechanisms for each head and auxiliary training loss functions to promote head diversity. We show that introducing diversity-promoting auxiliary loss functions during training is a more effective approach, and obtain WER improvements of up to 6% relative on the Librispeech corpus. Finally, we draw a connection between the diversity of attention heads and the similarity of the gradients of head parameters.