Lego-Features: Exporting modular encoder features for streaming and deliberation ASR
This work addresses modularity and efficiency in speech recognition systems, particularly for streaming and deliberation tasks, though it is incremental as it builds on existing modular representation research.
The paper tackles the problem of tight coupling between encoder and decoder in end-to-end speech recognition by introducing Lego-Features, modular encoder features that enable zero-shot stitching of encoders and decoders without fine-tuning, and shows they maintain high-quality performance in streaming and deliberation settings, outperforming N-best hypotheses without needing acoustic supplements.
In end-to-end (E2E) speech recognition models, a representational tight-coupling inevitably emerges between the encoder and the decoder. We build upon recent work that has begun to explore building encoders with modular encoded representations, such that encoders and decoders from different models can be stitched together in a zero-shot manner without further fine-tuning. While previous research only addresses full-context speech models, we explore the problem in a streaming setting as well. Our framework builds on top of existing encoded representations, converting them to modular features, dubbed as Lego-Features, without modifying the pre-trained model. The features remain interchangeable when the model is retrained with distinct initializations. Though sparse, we show that the Lego-Features are powerful when tested with RNN-T or LAS decoders, maintaining high-quality downstream performance. They are also rich enough to represent the first-pass prediction during two-pass deliberation. In this scenario, they outperform the N-best hypotheses, since they do not need to be supplemented with acoustic features to deliver the best results. Moreover, generating the Lego-Features does not require beam search or auto-regressive computation. Overall, they present a modular, powerful and cheap alternative to the standard encoder output, as well as the N-best hypotheses.