Bayesian Semi-structured Subspace Inference
Daniel Dold David Rügamer Beate Sick Oliver Dürr
Semi-structured regression models enable the joint modeling of interpretable
structured and complex unstructured feature effects. The structured model part
is inspired by statistical models and can be used to infer the input-output
relationship for features of particular importance. The complex unstructured
part defines an arbitrary deep neural network and thereby provides enough
flexibility to achieve competitive prediction performance. While these models
can also account for aleatoric uncertainty, there is still a lack of work on
accounting for epistemic uncertainty. In this paper, we address this problem by
presenting a Bayesian approximation for semi-structured regression models using
subspace inference. To this end, we extend subspace inference for joint
posterior sampling from a full parameter space for structured effects and a
subspace for unstructured effects. Apart from this hybrid sampling scheme, our
method allows for tunable complexity of the subspace and can capture multiple
minima in the loss landscape. Numerical experiments validate our approach's
efficacy in recovering structured effect parameter posteriors in
semi-structured models and approaching the full-space posterior distribution of
MCMC for increasing subspace dimension. Further, our approach exhibits
competitive predictive performance across simulated and real-world datasets.