Debiasing should be Good and Bad: Measuring the Consistency of Debiasing
  Techniques in Language Models
Robert Morabito Jad Kabbara Ali Emami
Debiasing methods that seek to mitigate the tendency of Language Models (LMs)
to occasionally output toxic or inappropriate text have recently gained
traction. In this paper, we propose a standardized protocol which distinguishes
methods that yield not only desirable results, but are also consistent with
their mechanisms and specifications. For example, we ask, given a debiasing
method that is developed to reduce toxicity in LMs, if the definition of
toxicity used by the debiasing method is reversed, would the debiasing results
also be reversed? We used such considerations to devise three criteria for our
new protocol: Specification Polarity, Specification Importance, and Domain
Transferability. As a case study, we apply our protocol to a popular debiasing
method, Self-Debiasing, and compare it to one we propose, called Instructive
Debiasing, and demonstrate that consistency is as important an aspect to
debiasing viability as is simply a desirable result. We show that our protocol
provides essential insights into the generalizability and interpretability of
debiasing methods that may otherwise go overlooked.