An Interactive Human-Machine Learning Interface for Collecting and
  Learning from Complex Annotations
Jonathan Erskine Matt Clifford Alexander Hepburn Raúl Santos-Rodríguez
Human-Computer Interaction has been shown to lead to improvements in machine
learning systems by boosting model performance, accelerating learning and
building user confidence. In this work, we aim to alleviate the expectation
that human annotators adapt to the constraints imposed by traditional labels by
allowing for extra flexibility in the form that supervision information is
collected. For this, we propose a human-machine learning interface for binary
classification tasks which enables human annotators to utilise counterfactual
examples to complement standard binary labels as annotations for a dataset.
Finally we discuss the challenges in future extensions of this work.