ALE: A Simulation-Based Active Learning Evaluation Framework for the
  Parameter-Driven Comparison of Query Strategies for NLP
Philipp Kohl Nils Freyer Yoka Krämer Henri Werth Steffen Wolf Bodo Kraft Matthias Meinecke Albert Zündorf
Supervised machine learning and deep learning require a large amount of
labeled data, which data scientists obtain in a manual, and time-consuming
annotation process. To mitigate this challenge, Active Learning (AL) proposes
promising data points to annotators they annotate next instead of a subsequent
or random sample. This method is supposed to save annotation effort while
maintaining model performance. However, practitioners face many AL strategies
for different tasks and need an empirical basis to choose between them. Surveys
categorize AL strategies into taxonomies without performance indications.
Presentations of novel AL strategies compare the performance to a small subset
of strategies. Our contribution addresses the empirical basis by introducing a
reproducible active learning evaluation (ALE) framework for the comparative
evaluation of AL strategies in NLP. The framework allows the implementation of
AL strategies with low effort and a fair data-driven comparison through
defining and tracking experiment parameters (e.g., initial dataset size, number
of data points per query step, and the budget). ALE helps practitioners to make
more informed decisions, and researchers can focus on developing new, effective
AL strategies and deriving best practices for specific use cases. With best
practices, practitioners can lower their annotation costs. We present a case
study to illustrate how to use the framework.