Named Entity Detection and Injection for Direct Speech Translation
This addresses a specific bottleneck in speech translation for applications requiring accurate entity handling, but it is incremental as it builds on existing methods with dictionary injection.
The paper tackled the problem of named entities being poorly handled in speech-to-text translation, particularly for locations and person names, by leveraging dictionaries of likely entities to improve model outputs. The result was a 31% reduction in person name errors.
In a sentence, certain words are critical for its semantic. Among them, named entities (NEs) are notoriously challenging for neural models. Despite their importance, their accurate handling has been neglected in speech-to-text (S2T) translation research, and recent work has shown that S2T models perform poorly for locations and notably person names, whose spelling is challenging unless known in advance. In this work, we explore how to leverage dictionaries of NEs known to likely appear in a given context to improve S2T model outputs. Our experiments show that we can reliably detect NEs likely present in an utterance starting from S2T encoder outputs. Indeed, we demonstrate that the current detection quality is sufficient to improve NE accuracy in the translation with a 31% reduction in person name errors.