Thompson sampling for improved exploration in GFlowNets
Jarrid Rector-Brooks Kanika Madan Moksh Jain Maksym Korablyov Cheng-Hao Liu Sarath Chandar Nikolay Malkin Yoshua Bengio
Generative flow networks (GFlowNets) are amortized variational inference
algorithms that treat sampling from a distribution over compositional objects
as a sequential decision-making problem with a learnable action policy. Unlike
other algorithms for hierarchical sampling that optimize a variational bound,
GFlowNet algorithms can stably run off-policy, which can be advantageous for
discovering modes of the target distribution. Despite this flexibility in the
choice of behaviour policy, the optimal way of efficiently selecting
trajectories for training has not yet been systematically explored. In this
paper, we view the choice of trajectories for training as an active learning
problem and approach it using Bayesian techniques inspired by methods for
multi-armed bandits. The proposed algorithm, Thompson sampling GFlowNets
(TS-GFN), maintains an approximate posterior distribution over policies and
samples trajectories from this posterior for training. We show in two domains
that TS-GFN yields improved exploration and thus faster convergence to the
target distribution than the off-policy exploration strategies used in past
work.