Uncertainty-Aware Decision Transformer for Stochastic Driving Environments
This work addresses the challenge of safe and effective planning for self-driving cars in unpredictable environments, representing an incremental improvement over existing decision transformer methods.
The paper tackles the problem of offline reinforcement learning failing in stochastic driving environments due to incorrect assumptions about consistent action outcomes, and introduces UNREST, which estimates uncertainties and uses truncated returns to achieve superior performance in various driving scenarios.
Offline Reinforcement Learning (RL) enables policy learning without active interactions, making it especially appealing for self-driving tasks. Recent successes of Transformers inspire casting offline RL as sequence modeling, which, however, fails in stochastic environments with incorrect assumptions that identical actions can consistently achieve the same goal. In this paper, we introduce an UNcertainty-awaRE deciSion Transformer (UNREST) for planning in stochastic driving environments without introducing additional transition or complex generative models. Specifically, UNREST estimates uncertainties by conditional mutual information between transitions and returns. Discovering 'uncertainty accumulation' and 'temporal locality' properties of driving environments, we replace the global returns in decision transformers with truncated returns less affected by environments to learn from actual outcomes of actions rather than environment transitions. We also dynamically evaluate uncertainty at inference for cautious planning. Extensive experiments demonstrate UNREST's superior performance in various driving scenarios and the power of our uncertainty estimation strategy.