Variable-rate discrete representation learning
This work addresses the challenge of efficiently representing variable-rate signals like speech for AI applications, offering a novel approach to event-based modeling.
The authors tackled the problem of uneven information distribution in perceptual signals like speech by proposing slow autoencoders (SlowAEs) for unsupervised learning of variable-rate discrete representations, resulting in event-based representations that adapt to information density while enabling faithful reconstruction and grammatical speech generation.
Semantically meaningful information content in perceptual signals is usually unevenly distributed. In speech signals for example, there are often many silences, and the speed of pronunciation can vary considerably. In this work, we propose slow autoencoders (SlowAEs) for unsupervised learning of high-level variable-rate discrete representations of sequences, and apply them to speech. We show that the resulting event-based representations automatically grow or shrink depending on the density of salient information in the input signals, while still allowing for faithful signal reconstruction. We develop run-length Transformers (RLTs) for event-based representation modelling and use them to construct language models in the speech domain, which are able to generate grammatical and semantically coherent utterances and continuations.