Generative Flow Networks as Entropy-Regularized RL
This work provides an incremental connection between RL and GFlowNets, potentially benefiting researchers in generative modeling and reinforcement learning by enabling integration of RL principles.
The paper tackles the problem of training generative flow networks (GFlowNets) by redefining it as an entropy-regularized reinforcement learning (RL) problem, showing that standard soft RL algorithms can be competitive with established GFlowNet methods in probabilistic modeling tasks.
The recently proposed generative flow networks (GFlowNets) are a method of training a policy to sample compositional discrete objects with probabilities proportional to a given reward via a sequence of actions. GFlowNets exploit the sequential nature of the problem, drawing parallels with reinforcement learning (RL). Our work extends the connection between RL and GFlowNets to a general case. We demonstrate how the task of learning a generative flow network can be efficiently redefined as an entropy-regularized RL problem with a specific reward and regularizer structure. Furthermore, we illustrate the practical efficiency of this reformulation by applying standard soft RL algorithms to GFlowNet training across several probabilistic modeling tasks. Contrary to previously reported results, we show that entropic RL approaches can be competitive against established GFlowNet training methods. This perspective opens a direct path for integrating RL principles into the realm of generative flow networks.