Temporal Difference Variational Auto-Encoder
This addresses the need for more efficient and abstract world models in AI planning, though it appears incremental as it builds on existing generative and reinforcement learning methods.
The paper tackles the problem of building a mental simulator for agents that can represent abstract states, handle uncertainty, and perform temporal abstraction, proposing TD-VAE which learns representations for future states and can be rolled out without single-step transitions.
To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.