Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling
This system aims to improve video accessibility by automatically generating subtitles for streaming services, addressing a domain-specific problem.
The paper tackles automatic character-aware subtitle generation by proposing an audio-visual method that produces a full transcript with precise speech timestamps and speaker identification, evaluated on TV sitcoms like Seinfeld, Fraiser, and Scrubs.
The goal of this paper is automatic character-aware subtitle generation. Given a video and a minimal amount of metadata, we propose an audio-visual method that generates a full transcript of the dialogue, with precise speech timestamps, and the character speaking identified. The key idea is to first use audio-visual cues to select a set of high-precision audio exemplars for each character, and then use these exemplars to classify all speech segments by speaker identity. Notably, the method does not require face detection or tracking. We evaluate the method over a variety of TV sitcoms, including Seinfeld, Fraiser and Scrubs. We envision this system being useful for the automatic generation of subtitles to improve the accessibility of the vast amount of videos available on modern streaming services. Project page : \url{https://www.robots.ox.ac.uk/~vgg/research/look-listen-recognise/}