Laplacian Representations for Decision-Time Planning
This addresses the problem of decision-time planning in model-based RL for researchers and practitioners, offering a novel method to improve performance in offline goal-conditioned tasks, though it appears incremental as it builds on existing representation ideas.
The paper tackled the challenge of planning with learned models in reinforcement learning by proposing Laplacian representations to capture state-space distances at multiple time scales, resulting in the ALPS algorithm outperforming baselines on offline goal-conditioned RL tasks from OGBench.
Planning with a learned model remains a key challenge in model-based reinforcement learning (RL). In decision-time planning, state representations are critical as they must support local cost computation while preserving long-horizon structure. In this paper, we show that the Laplacian representation provides an effective latent space for planning by capturing state-space distances at multiple time scales. This representation preserves meaningful distances and naturally decomposes long-horizon problems into subgoals, also mitigating the compounding errors that arise over long prediction horizons. Building on these properties, we introduce ALPS, a hierarchical planning algorithm, and demonstrate that it outperforms commonly used baselines on a selection of offline goal-conditioned RL tasks from OGBench, a benchmark previously dominated by model-free methods.