Smooth Operators: LLMs Translating Imperfect Hints into Disfluency-Rich Transcripts
This work addresses the need for accurate disfluency detection to improve automatic speech and language processing systems, making it incremental by applying LLMs to a known bottleneck in a novel way.
The paper tackles the problem of detecting disfluencies in spoken language by proposing a method that uses large language models (LLMs) to generate fully annotated disfluency-rich transcripts from imperfect textual inputs with timestamps, demonstrating that LLMs can effectively handle these imperfections.
Accurate detection of disfluencies in spoken language is crucial for enhancing the performance of automatic speech and language processing systems, as well as fostering the development of more inclusive speech and language technologies. Leveraging the growing trend of large language models (LLMs) as versatile learners capable of processing both lexical and non-lexical inputs (e.g., audio and video), we propose a novel approach to transcribing disfluencies as explicit tokens with timestamps, enabling the generation of fully annotated disfluency-rich transcripts. Our method integrates acoustic representations extracted from an audio encoder with textual inputs of varying quality: clean transcriptions without disfluencies, time-aligned transcriptions from aligners, or outputs from phoneme-based ASR models -- all of which may contain imperfections. Importantly, our experiments demonstrate that textual inputs do not need to be flawless. As long as they include timestamp-related cues, LLMs can effectively smooth the input and produce fully disfluency-annotated transcripts, underscoring their robustness in handling imperfect hints.