ActiveGLAE: A Benchmark for Deep Active Learning with Transformers
Lukas Rauch Matthias Aßenmacher Denis Huseljic Moritz Wirth Bernd Bischl Bernhard Sick
Deep active learning (DAL) seeks to reduce annotation costs by enabling the
model to actively query instance annotations from which it expects to learn the
most. Despite extensive research, there is currently no standardized evaluation
protocol for transformer-based language models in the field of DAL. Diverse
experimental settings lead to difficulties in comparing research and deriving
recommendations for practitioners. To tackle this challenge, we propose the
ActiveGLAE benchmark, a comprehensive collection of data sets and evaluation
guidelines for assessing DAL. Our benchmark aims to facilitate and streamline
the evaluation process of novel DAL strategies. Additionally, we provide an
extensive overview of current practice in DAL with transformer-based language
models. We identify three key challenges - data set selection, model training,
and DAL settings - that pose difficulties in comparing query strategies. We
establish baseline results through an extensive set of experiments as a
reference point for evaluating future work. Based on our findings, we provide
guidelines for researchers and practitioners.