Sparse Imagination for Efficient Visual World Model Planning
This work addresses a bottleneck in robotics and other resource-constrained domains by enabling more efficient real-time decision-making, though it is incremental as it builds on existing transformer-based world models.
The paper tackles the computational inefficiency of world model planning in real-time applications by proposing a sparse imagination method that reduces token processing, achieving significant inference speed improvements while maintaining task performance.
World model based planning has significantly improved decision-making in complex environments by enabling agents to simulate future states and make informed choices. However, ensuring the prediction accuracy of world models often demands substantial computational resources, posing a major challenge for real-time applications. This computational burden is particularly restrictive in robotics, where resources are severely constrained. To address this limitation, we propose a Sparse Imagination for Efficient Visual World Model Planning, which enhances computational efficiency by reducing the number of tokens processed during forward prediction. Our method leverages a sparsely trained vision-based world model based on transformers with randomized grouped attention strategy, allowing the model to adaptively adjust the number of tokens processed based on the computational resource. By enabling sparse imagination (rollout), our approach significantly accelerates planning while maintaining high control fidelity. Experimental results demonstrate that sparse imagination preserves task performance while dramatically improving inference efficiency, paving the way for the deployment of world models in real-time decision-making scenarios.