Whose wife is it anyway? Assessing bias against same-gender
  relationships in machine translation
Ian Stewart Rada Mihalcea
Machine translation often suffers from biased data and algorithms that can
lead to unacceptable errors in system output. While bias in gender norms has
been investigated, less is known about whether MT systems encode bias about
social relationships, e.g. sentences such as "the lawyer kissed her wife." We
investigate the degree of bias against same-gender relationships in MT systems,
using generated template sentences drawn from several noun-gender languages
(e.g. Spanish). We find that three popular MT services consistently fail to
accurately translate sentences concerning relationships between nouns of the
same gender. The error rate varies considerably based on the context, e.g.
same-gender sentences referencing high female-representation occupations are
translated with lower accuracy. We provide this work as a case study in the
evaluation of intrinsic bias in NLP systems, with respect to social
relationships.