Can a Confident Prior Replace a Cold Posterior?
Martin Marek Brooks Paige Pavel Izmailov
Benchmark datasets used for image classification tend to have very low levels
of label noise. When Bayesian neural networks are trained on these datasets,
they often underfit, misrepresenting the aleatoric uncertainty of the data. A
common solution is to cool the posterior, which improves fit to the training
data but is challenging to interpret from a Bayesian perspective. We explore
whether posterior tempering can be replaced by a confidence-inducing prior
distribution. First, we introduce a "DirClip" prior that is practical to sample
and nearly matches the performance of a cold posterior. Second, we introduce a
"confidence prior" that directly approximates a cold likelihood in the limit of
decreasing temperature but cannot be easily sampled. Lastly, we provide several
general insights into confidence-inducing priors, such as when they might
diverge and how fine-tuning can mitigate numerical instability.