Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
This work addresses offline GCRL for domains like autonomous navigation and locomotion, where data collection is costly and unsafe, offering an incremental improvement over existing methods.
The paper tackled the challenge of offline goal-conditioned reinforcement learning (GCRL) by proposing a physics-informed regularizer derived from the Eikonal PDE to improve value learning, resulting in significant performance and generalization gains, especially in stitching regimes and large-scale navigation tasks.
Offline Goal-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting interactive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state-action space and to generalize across long-horizon tasks. To improve on these challenges, we propose a \emph{Physics-informed (Pi)} regularized loss for value learning, derived from the Eikonal Partial Differential Equation (PDE) and which induces a geometric inductive bias in the learned value function. Unlike generic gradient penalties that are primarily used to stabilize training, our formulation is grounded in continuous-time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Eikonal-regularized HIQL (Eik-HIQL), yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks.