Text-Free Image-to-Speech Synthesis Using Learned Segmental Units
This work addresses the problem of generating spoken image descriptions for users who may benefit from audio output, such as visually impaired individuals, by eliminating the need for text-based intermediate steps.
This paper introduces the first model for directly synthesizing spoken audio captions for images without requiring natural language text as an intermediate representation. It achieves this by connecting image captioning and speech synthesis modules with self-supervised, visually grounded discrete sub-word speech units. The model demonstrates fluent, natural-sounding audio captions on Flickr8k and a new MSCOCO spoken caption dataset, capturing diverse visual semantics.
In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. Instead, we connect the image captioning module and the speech synthesis module with a set of discrete, sub-word speech units that are discovered with a self-supervised visual grounding task. We conduct experiments on the Flickr8k spoken caption dataset in addition to a novel corpus of spoken audio captions collected for the popular MSCOCO dataset, demonstrating that our generated captions also capture diverse visual semantics of the images they describe. We investigate several different intermediate speech representations, and empirically find that the representation must satisfy several important properties to serve as drop-in replacements for text.