Zero-shot sampling of adversarial entities in biomedical question
  answering
R. Patrick Xian Alex J. Lee Vincent Wang Qiming Cui Russell Ro Reza Abbasi-Asl
The increasing depth of parametric domain knowledge in large language models
(LLMs) is fueling their rapid deployment in real-world applications. In
high-stakes and knowledge-intensive tasks, understanding model vulnerabilities
is essential for quantifying the trustworthiness of model predictions and
regulating their use. The recent discovery of named entities as adversarial
examples in natural language processing tasks raises questions about their
potential guises in other settings. Here, we propose a powerscaled
distance-weighted sampling scheme in embedding space to discover diverse
adversarial entities as distractors. We demonstrate its advantage over random
sampling in adversarial question answering on biomedical topics. Our approach
enables the exploration of different regions on the attack surface, which
reveals two regimes of adversarial entities that markedly differ in their
characteristics. Moreover, we show that the attacks successfully manipulate
token-wise Shapley value explanations, which become deceptive in the
adversarial setting. Our investigations illustrate the brittleness of domain
knowledge in LLMs and reveal a shortcoming of standard evaluations for
high-capacity models.