Beyond Active Learning: Leveraging the Full Potential of Human
  Interaction via Auto-Labeling, Human Correction, and Human Verification
Nathan Beck Krishnateja Killamsetty Suraj Kothawade Rishabh Iyer
Active Learning (AL) is a human-in-the-loop framework to interactively and
adaptively label data instances, thereby enabling significant gains in model
performance compared to random sampling. AL approaches function by selecting
the hardest instances to label, often relying on notions of diversity and
uncertainty. However, we believe that these current paradigms of AL do not
leverage the full potential of human interaction granted by automated label
suggestions. Indeed, we show that for many classification tasks and datasets,
most people verifying if an automatically suggested label is correct take
$3\times$ to $4\times$ less time than they do changing an incorrect suggestion
to the correct label (or labeling from scratch without any suggestion).
Utilizing this result, we propose CLARIFIER (aCtive LeARnIng From tIEred
haRdness), an Interactive Learning framework that admits more effective use of
human interaction by leveraging the reduced cost of verification. By targeting
the hard (uncertain) instances with existing AL methods, the intermediate
instances with a novel label suggestion scheme using submodular mutual
information functions on a per-class basis, and the easy (confident) instances
with highest-confidence auto-labeling, CLARIFIER can improve over the
performance of existing AL approaches on multiple datasets -- particularly on
those that have a large number of classes -- by almost 1.5$\times$ to 2$\times$
in terms of relative labeling cost.