A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets
  given Small Pilot Data
Ethan Harvey Wansu Chen David M. Kent Michael C. Hughes
Practitioners building classifiers often start with a smaller pilot dataset
and plan to grow to larger data in the near future. Such projects need a
toolkit for extrapolating how much classifier accuracy may improve from a 2x,
10x, or 50x increase in data size. While existing work has focused on finding a
single "best-fit" curve using various functional forms like power laws, we
argue that modeling and assessing the uncertainty of predictions is critical
yet has seen less attention. In this paper, we propose a Gaussian process model
to obtain probabilistic extrapolations of accuracy or similar performance
metrics as dataset size increases. We evaluate our approach in terms of error,
likelihood, and coverage across six datasets. Though we focus on medical tasks
and image modalities, our open source approach generalizes to any kind of
classifier.