SVL: Goal-Conditioned Reinforcement Learning as Survival Learning
For researchers in goal-conditioned RL, SVL offers a more stable and sample-efficient alternative to TD methods, particularly for long-horizon tasks.
SVL reframes goal-conditioned RL as survival learning, modeling time-to-goal as a probability distribution to avoid bootstrapping instability. It matches or surpasses hierarchical TD and Monte Carlo baselines on offline GCRL benchmarks, excelling on long-horizon tasks.
Standard approaches to goal-conditioned reinforcement learning (GCRL) that rely on temporal-difference learning can be unstable and sample-inefficient due to bootstrapping. While recent work has explored contrastive and supervised formulations to improve stability, we present a probabilistic alternative, called survival value learning (SVL), that reframes GCRL as a survival learning problem by modeling the time-to-goal from each state as a probability distribution. This structured distributional Monte Carlo perspective yields a closed-form identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities, enabling value estimation via a hazard model trained via maximum likelihood on both event and right-censored trajectories. We introduce three practical value estimators, including finite-horizon truncation and two binned infinite-horizon approximations to capture long-horizon objectives. Experiments on offline GCRL benchmarks show that SVL combined with hierarchical actors matches or surpasses strong hierarchical TD and Monte Carlo baselines, excelling on complex, long-horizon tasks.