Rooted Absorbed Prefix Trajectory Balance with Submodular Replay for GFlowNet Training
This addresses mode collapse issues in GFlowNets for fine-tuning LLMs, representing an incremental improvement with domain-specific applications.
The paper tackled mode collapse in Generative Flow Networks (GFlowNets) for fine-tuning large language models, proposing RapTB and SubM methods that improved optimization performance and molecular diversity while preserving high validity in tasks like molecule generation.
Generative Flow Networks (GFlowNets) enable fine-tuning large language models to approximate reward-proportional posteriors, but they remain prone to mode collapse, manifesting as prefix collapse and length bias. We attribute this to two factors: (i) weak credit assignment to early prefixes, and (ii) biased replay that induces a shifted, non-representative training flow distribution. We propose Rooted absorbed prefix Trajectory Balance RapTB, an objective that anchors subtrajectory supervision at the root and propagates terminal rewards to intermediate prefixes via absorbed suffix-based backups, providing dense prefix-level learning signals. To mitigate replay-induced distribution shift, we further introduce SubM, a submodular replay refresh strategy that promotes both high reward and diversity. Empirically, on tasks such as molecule generation with LLM using SMILES strings, RapTB combined with SubM consistently improves optimization performance and molecular diversity while preserving high validity.