On the N-gram Approximation of Pre-trained Language Models
This work addresses the under-explored potential of PLMs in ASR, offering incremental improvements for domain-specific language modeling tasks.
This study tackled the problem of using pre-trained language models (PLMs) for language modeling in Automatic Speech Recognition (ASR) by approximating GPT-2 into an n-gram model, resulting in a 15% improvement in test perplexity over a baseline trigram and an additional 5% gain with a vocabulary-restricted decoding method.
Large pre-trained language models (PLMs) have shown remarkable performance across various natural language understanding (NLU) tasks, particularly in low-resource settings. Nevertheless, their potential in Automatic Speech Recognition (ASR) remains largely unexplored. This study investigates the potential usage of PLMs for language modelling in ASR. We compare the application of large-scale text sampling and probability conversion for approximating GPT-2 into an n-gram model. Furthermore, we introduce a vocabulary-restricted decoding method for random sampling, and evaluate the effects of domain difficulty and data size on the usability of generated text. Our findings across eight domain-specific corpora support the use of sampling-based approximation and show that interpolating with a large sampled corpus improves test perplexity over a baseline trigram by 15%. Our vocabulary-restricted decoding method pushes this improvement further by 5% in domain-specific settings.