A Bias-Variance Decomposition for Ensembles over Multiple Synthetic
  Datasets
Ossi Räisä Antti Honkela
Recent studies have highlighted the benefits of generating multiple synthetic
datasets for supervised learning, from increased accuracy to more effective
model selection and uncertainty estimation. These benefits have clear empirical
support, but the theoretical understanding of them is currently very light. We
seek to increase the theoretical understanding by deriving bias-variance
decompositions for several settings of using multiple synthetic datasets. Our
theory predicts multiple synthetic datasets to be especially beneficial for
high-variance downstream predictors, and yields a simple rule of thumb to
select the appropriate number of synthetic datasets in the case of mean-squared
error and Brier score. We investigate how our theory works in practice by
evaluating the performance of an ensemble over many synthetic datasets for
several real datasets and downstream predictors. The results follow our theory,
showing that our insights are also practically relevant.