Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets
This work addresses a critical bottleneck in training goal-conditioned GFlowNets for high-dimensional problems, offering an incremental improvement over prior methods.
The paper tackles the challenge of training goal-conditioned GFlowNets, which suffer from sparse rewards and limited trajectory coverage, by proposing Retrospective Backward Synthesis (RBS) to enrich training trajectories. The method improves sample efficiency significantly and outperforms baselines on standard benchmarks.
Generative Flow Networks (GFlowNets), a new family of probabilistic samplers, have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates, in contrast to standard return maximization approaches (e.g., reinforcement learning) which often converge to a single optimal solution. Recent works have focused on developing goal-conditioned GFlowNets, which aim to train a single GFlowNet capable of achieving different outcomes as the task specifies. However, training such models is challenging due to extremely sparse rewards, particularly in high-dimensional problems. Moreover, previous methods suffer from the limited coverage of explored trajectories during training, which presents more pronounced challenges when only offline data is available. In this work, we propose a novel method called \textbf{R}etrospective \textbf{B}ackward \textbf{S}ynthesis (\textbf{RBS}) to address these critical problems. Specifically, RBS synthesizes new backward trajectories in goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby introducing copious learnable signals for effectively tackling the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by a large margin and outperforms strong baselines on various standard evaluation benchmarks.