Parallelizing Autoregressive Generation with Variational State Space Models
This addresses a bottleneck in sequence modeling for tasks like language generation, offering speed improvements, though it is incremental as it builds on existing state space models and variational autoencoders.
The paper tackles the problem of enabling parallel generation in autoregressive sequence models, which traditionally require sequential processing, by proposing a variational state space model (VSSM) that allows both training and generation to be conducted in parallel, achieving competitive generation quality with models like Transformers and Mamba SSM on datasets such as MNIST and CIFAR.
Attention-based models such as Transformers and recurrent models like state space models (SSMs) have emerged as successful methods for autoregressive sequence modeling. Although both enable parallel training, none enable parallel generation due to their autoregressiveness. We propose the variational SSM (VSSM), a variational autoencoder (VAE) where both the encoder and decoder are SSMs. Since sampling the latent variables and decoding them with the SSM can be parallelized, both training and generation can be conducted in parallel. Moreover, the decoder recurrence allows generation to be resumed without reprocessing the whole sequence. Finally, we propose the autoregressive VSSM that can be conditioned on a partial realization of the sequence, as is common in language generation tasks. Interestingly, the autoregressive VSSM still enables parallel generation. We highlight on toy problems (MNIST, CIFAR) the empirical gains in speed-up and show that it competes with traditional models in terms of generation quality (Transformer, Mamba SSM).