XLVIN: eXecuted Latent Value Iteration Nets
This addresses the problem of enabling effective planning in deep reinforcement learning for tasks requiring long-range reasoning, though it appears incremental by building on VINs.
The paper tackled the limitations of Value Iteration Networks (VINs) by proposing eXecuted Latent Value Iteration Networks (XLVINs), which successfully deploy VIN-style models on generic environments, matching VIN performance in discrete, fixed MDPs and providing significant improvements to model-free baselines across three general MDP setups.
Value Iteration Networks (VINs) have emerged as a popular method to incorporate planning algorithms within deep reinforcement learning, enabling performance improvements on tasks requiring long-range reasoning and understanding of environment dynamics. This came with several limitations, however: the model is not incentivised in any way to perform meaningful planning computations, the underlying state space is assumed to be discrete, and the Markov decision process (MDP) is assumed fixed and known. We propose eXecuted Latent Value Iteration Networks (XLVINs), which combine recent developments across contrastive self-supervised learning, graph representation learning and neural algorithmic reasoning to alleviate all of the above limitations, successfully deploying VIN-style models on generic environments. XLVINs match the performance of VIN-like models when the underlying MDP is discrete, fixed and known, and provides significant improvements to model-free baselines across three general MDP setups.