Connecting the Dots: Is Mode-Connectedness the Key to Feasible
  Sample-Based Inference in Bayesian Neural Networks?
Emanuel Sommer Lisa Wimmer Theodore Papamarkou Ludwig Bothmann Bernd Bischl David Rügamer
A major challenge in sample-based inference (SBI) for Bayesian neural
networks is the size and structure of the networks' parameter space. Our work
shows that successful SBI is possible by embracing the characteristic
relationship between weight and function space, uncovering a systematic link
between overparameterization and the difficulty of the sampling problem.
Through extensive experiments, we establish practical guidelines for sampling
and convergence diagnosis. As a result, we present a Bayesian deep ensemble
approach as an effective solution with competitive performance and uncertainty
quantification.