Latent-Autoregressive GP-VAE Language Model
This is an incremental proof-of-concept for language modeling that suggests latent probabilistic geometry can support temporal structure.
The authors tackled the problem of modeling sequential dynamics in language by transferring them to a continuous latent space using a Gaussian Process integrated into a Variational Autoencoder, with results showing stable training and consistent behavior in sequential and parallel sampling variants.
We investigate a fully Latent AutoRegressive scheme based on a Gaussian Process (GP) integrated into a Variational Autoencoder (VAE). In this setting, sequential dynamics are transferred from the observation space to a continuous latent space, while linguistic generation remains parallel through a non-autoregressive decoder. We present a complete methodological formulation, including a causal GP prior, a structured amortized posterior, and a training protocol based on a regularized ELBO. Empirical evaluation, conducted within a deliberately constrained proof-of-concept (POC) framework, shows that the model can be trained stably and that the sequential and parallel sampling variants exhibit consistent behavior. Overall, the results suggest that part of the temporal structure in a language model can be supported by the probabilistic geometry of the latent space rather than by explicit neural operations.