Segmental Attention Decoding With Long Form Acoustic Encodings
This addresses a fundamental problem in speech processing for models handling long audio inputs, though it appears incremental as it builds on existing attention-based methods.
The paper tackled the incompatibility of attention-based encoder-decoder models with long-form acoustic encodings by proposing modifications like injecting positional encodings and long-form training, which closed the accuracy gap between continuous and segmented encodings.
We address the fundamental incompatibility of attention-based encoder-decoder (AED) models with long-form acoustic encodings. AED models trained on segmented utterances learn to encode absolute frame positions by exploiting limited acoustic context beyond segment boundaries, but fail to generalize when decoding long-form segments where these cues vanish. The model loses ability to order acoustic encodings due to permutation invariance of keys and values in cross-attention. We propose four modifications: (1) injecting explicit absolute positional encodings into cross-attention for each decoded segment, (2) long-form training with extended acoustic context to eliminate implicit absolute position encoding, (3) segment concatenation to cover diverse segmentations needed during training, and (4) semantic segmentation to align AED-decoded segments with training segments. We show these modifications close the accuracy gap between continuous and segmented acoustic encodings, enabling auto-regressive use of the attention decoder.