Unlocking Unlabeled Data: Ensemble Learning with the Hui- Walter
  Paradigm for Performance Estimation in Online and Static Settings
Kevin Slote Elaine Lee
In the realm of machine learning and statistical modeling, practitioners
often work under the assumption of accessible, static, labeled data for
evaluation and training. However, this assumption often deviates from reality
where data may be private, encrypted, difficult- to-measure, or unlabeled. In
this paper, we bridge this gap by adapting the Hui-Walter paradigm, a method
traditionally applied in epidemiology and medicine, to the field of machine
learning. This approach enables us to estimate key performance metrics such as
false positive rate, false negative rate, and priors in scenarios where no
ground truth is available. We further extend this paradigm for handling online
data, opening up new possibilities for dynamic data environments. Our
methodology involves partitioning data into latent classes to simulate multiple
data populations (if natural populations are unavailable) and independently
training models to replicate multiple tests. By cross-tabulating binary
outcomes across ensemble categorizers and multiple populations, we are able to
estimate unknown parameters through Gibbs sampling, eliminating the need for
ground-truth or labeled data. This paper showcases the potential of our
methodology to transform machine learning practices by allowing for accurate
model assessment under dynamic and uncertain data conditions.