Finding the Time to Think: Learning Planning Budgets in Real-Time RL
For real-time RL applications where deliberation time is costly, this work provides a practical method to learn planning budgets, improving performance over fixed delays.
This paper introduces variable-delay real-time RL, where agents choose how long to deliberate while the environment progresses. A lightweight gating policy trained on top of a planner selects state-dependent planning budgets, outperforming fixed-budget and heuristic baselines in Pac-Man, Tetris, Snake, Speed Hex, and Speed Go.
Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs.