This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language
  Models
Seraphina Goldfarb-Tarrant Eddie Ungless Esma Balkir Su Lin Blodgett
Bias research in NLP seeks to analyse models for social biases, thus helping
NLP practitioners uncover, measure, and mitigate social harms. We analyse the
body of work that uses prompts and templates to assess bias in language models.
We draw on a measurement modelling framework to create a taxonomy of attributes
that capture what a bias test aims to measure and how that measurement is
carried out. By applying this taxonomy to 90 bias tests, we illustrate
qualitatively and quantitatively that core aspects of bias test
conceptualisations and operationalisations are frequently unstated or
ambiguous, carry implicit assumptions, or be mismatched. Our analysis
illuminates the scope of possible bias types the field is able to measure, and
reveals types that are as yet under-researched. We offer guidance to enable the
community to explore a wider section of the possible bias space, and to better
close the gap between desired outcomes and experimental design, both for bias
and for evaluating language models more broadly.