Maximum entropy GFlowNets with soft Q-learning
Sobhan Mohammadpour Emmanuel Bengio Emma Frejinger Pierre-Luc Bacon
Generative Flow Networks (GFNs) have emerged as a powerful tool for sampling
discrete objects from unnormalized distributions, offering a scalable
alternative to Markov Chain Monte Carlo (MCMC) methods. While GFNs draw
inspiration from maximum entropy reinforcement learning (RL), the connection
between the two has largely been unclear and seemingly applicable only in
specific cases. This paper addresses the connection by constructing an
appropriate reward function, thereby establishing an exact relationship between
GFNs and maximum entropy RL. This construction allows us to introduce maximum
entropy GFNs, which, in contrast to GFNs with uniform backward policy, achieve
the maximum entropy attainable by GFNs without constraints on the state space.