Progressive Alignment Objectives for Aligner-Encoder based ASR
Improves training stability and accuracy of Aligner-Encoder ASR models for speech recognition.
Aligner-Encoder ASR models suffer from brittle alignment in upper layers. The proposed InterAligner with InterCTC reduces WER on LibriSpeech from 5.0/7.8 to 3.1/5.6, with largest gains on long utterances.
Aligner-Encoders are recently proposed seq2seq end-to-end ASR models that replace decoder attention by predicting the uth token directly from the u-th encoder position, so the encoder must learn the alignment internally without cross-attention or a transducer lattice. In practice, this alignment often forms abruptly in the upper layers, making training sensitive and brittle on long utterances. We propose InterAligner, which adds an intermediate Aligner objective so alignment can form progressively across depth, together with an intermediate CTC loss (InterCTC) to stabilize optimization. On LibriSpeech with a 17-layer Conformer, a final-only Aligner reaches 5.0/7.8 WER (test-clean/other). InterCTC improves to 3.4/6.0, and InterAligner further reduces WER to 3.1/5.6 with the largest gains on long utterances.