Partially Stochastic Infinitely Deep Bayesian Neural Networks
Sergio Calvo-Ordonez Matthieu Meunier Francesco Piatti Yuantao Shi
In this paper, we present Partially Stochastic Infinitely Deep Bayesian
Neural Networks, a novel family of architectures that integrates partial
stochasticity into the framework of infinitely deep neural networks. Our new
class of architectures is designed to improve the limitations of existing
architectures around computational efficiency at training and inference time.
To do this, we leverage the advantages of partial stochasticity in the
infinite-depth limit which include the benefits of full stochasticity e.g.
robustness, uncertainty quantification, and memory efficiency, whilst improving
their limitations around computational efficiency at training and inference
time. We present a variety of architectural configurations, offering
flexibility in network design including different methods for weight partition.
We also provide mathematical guarantees on the expressivity of our models by
establishing that our network family qualifies as Universal Conditional
Distribution Approximators. Lastly, empirical evaluations across multiple tasks
show that our proposed architectures achieve better downstream task performance
and uncertainty quantification than their counterparts while being
significantly more efficient.