Estimating optimal PAC-Bayes bounds with Hamiltonian Monte Carlo
Szilvia Ujváry Gergely Flamich Vincent Fortuin José Miguel Hernández Lobato
An important yet underexplored question in the PAC-Bayes literature is how
much tightness we lose by restricting the posterior family to factorized
Gaussian distributions when optimizing a PAC-Bayes bound. We investigate this
issue by estimating data-independent PAC-Bayes bounds using the optimal
posteriors, comparing them to bounds obtained using MFVI. Concretely, we (1)
sample from the optimal Gibbs posterior using Hamiltonian Monte Carlo, (2)
estimate its KL divergence from the prior with thermodynamic integration, and
(3) propose three methods to obtain high-probability bounds under different
assumptions. Our experiments on the MNIST dataset reveal significant tightness
gaps, as much as 5-6\% in some cases.