Efficient Incremental Belief Updates Using Weighted Virtual Observations
David Tolpin
We present an algorithmic solution to the problem of incremental belief
updating in the context of Monte Carlo inference in Bayesian statistical models
represented by probabilistic programs. Given a model and a sample-approximated
posterior, our solution constructs a set of weighted observations to condition
the model such that inference would result in the same posterior. This problem
arises e.g. in multi-level modelling, incremental inference, inference in
presence of privacy constraints. First, a set of virtual observations is
selected, then, observation weights are found through a computationally
efficient optimization procedure such that the reconstructed posterior
coincides with or closely approximates the original posterior. We implement and
apply the solution to a number of didactic examples and case studies, showing
efficiency and robustness of our approach. The provided reference
implementation is agnostic to the probabilistic programming language or the
inference algorithm, and can be applied to most mainstream probabilistic
programming environments.