Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active
  Learning
Wei Tan Lan Du Wray Buntine
The effectiveness of active learning largely depends on the sampling
efficiency of the acquisition function. Expected Loss Reduction (ELR) focuses
on a Bayesian estimate of the reduction in classification error, and more
general costs fit in the same framework. We propose Bayesian Estimate of Mean
Proper Scores (BEMPS) to estimate the increase in strictly proper scores such
as log probability or negative mean square error within this framework. We also
prove convergence results for this general class of costs. To facilitate better
experimentation with the new acquisition functions, we develop a complementary
batch AL algorithm that encourages diversity in the vector of expected changes
in scores for unlabeled data. To allow high-performance classifiers, we combine
deep ensembles, and dynamic validation set construction on pretrained models,
and further speed up the ensemble process with the idea of Monte Carlo Dropout.
Extensive experiments on both texts and images show that the use of mean square
error and log probability with BEMPS yields robust acquisition functions and
well-calibrated classifiers, and consistently outperforms the others tested.
The advantages of BEMPS over the others are further supported by a set of
qualitative analyses, where we visualise their sampling behaviour using data
maps and t-SNE plots.