Towards Understanding and Improving GFlowNet Training
Max W. Shen Emmanuel Bengio Ehsan Hajiramezanali Andreas Loukas Kyunghyun Cho Tommaso Biancalani
Generative flow networks (GFlowNets) are a family of algorithms that learn a
generative policy to sample discrete objects $x$ with non-negative reward
$R(x)$. Learning objectives guarantee the GFlowNet samples $x$ from the target
distribution $p^*(x) \propto R(x)$ when loss is globally minimized over all
states or trajectories, but it is unclear how well they perform with practical
limits on training resources. We introduce an efficient evaluation strategy to
compare the learned sampling distribution to the target reward distribution. As
flows can be underdetermined given training data, we clarify the importance of
learned flows to generalization and matching $p^*(x)$ in practice. We
investigate how to learn better flows, and propose (i) prioritized replay
training of high-reward $x$, (ii) relative edge flow policy parametrization,
and (iii) a novel guided trajectory balance objective, and show how it can
solve a substructure credit assignment problem. We substantially improve sample
efficiency on biochemical design tasks.