Emerging Challenges in Personalized Medicine: Assessing Demographic
  Effects on Biomedical Question Answering Systems
Sagi Shaier Kevin Bennett Lawrence Hunter Katharina von der Wense
State-of-the-art question answering (QA) models exhibit a variety of social
biases (e.g., with respect to sex or race), generally explained by similar
issues in their training data. However, what has been overlooked so far is that
in the critical domain of biomedicine, any unjustified change in model output
due to patient demographics is problematic: it results in the unfair treatment
of patients. Selecting only questions on biomedical topics whose answers do not
depend on ethnicity, sex, or sexual orientation, we ask the following research
questions: (RQ1) Do the answers of QA models change when being provided with
irrelevant demographic information? (RQ2) Does the answer of RQ1 differ between
knowledge graph (KG)-grounded and text-based QA systems? We find that
irrelevant demographic information change up to 15% of the answers of a
KG-grounded system and up to 23% of the answers of a text-based system,
including changes that affect accuracy. We conclude that unjustified answer
changes caused by patient demographics are a frequent phenomenon, which raises
fairness concerns and should be paid more attention to.