World Machine: Towards Generative World Modeling for Time-Series
This work addresses the computational inefficiency of transformers for time-series modeling, offering a more scalable approach for generative world modeling.
World Machine introduces a transformer-based generative world model for time series that uses latent states to adapt to varying observed data and contexts, achieving linear scaling in computational and memory costs compared to the quadratic scaling of traditional transformers. Experiments on the Toy1D synthetic dataset validate its feasibility and demonstrate capabilities beyond conventional transformers.
World models represent a paradigm shift in generative AI, pursuing predictive understanding and controllable simulation of environments in a structured and generalizable way. We present World Machine, a generative world-modeling architecture for time series. It is a transformer-based architecture with latent states that enables adaptation to different amounts of observed data and contexts. This shows an improvement over traditional transformers, which have a computational and memory cost that scales quadratically with the context. Experiments on a proposed synthetic dataset, Toy1D, validate the approach's feasibility, demonstrate capabilities not found in conventional transformers, and highlight the contributions of each component of the training protocol.