Why do universal adversarial attacks work on large language models?:
  Geometry might be the answer
Varshini Subhash Anna Bialas Weiwei Pan Finale Doshi-Velez
Transformer based large language models with emergent capabilities are
becoming increasingly ubiquitous in society. However, the task of understanding
and interpreting their internal workings, in the context of adversarial
attacks, remains largely unsolved. Gradient-based universal adversarial attacks
have been shown to be highly effective on large language models and potentially
dangerous due to their input-agnostic nature. This work presents a novel
geometric perspective explaining universal adversarial attacks on large
language models. By attacking the 117M parameter GPT-2 model, we find evidence
indicating that universal adversarial triggers could be embedding vectors which
merely approximate the semantic information in their adversarial training
region. This hypothesis is supported by white-box model analysis comprising
dimensionality reduction and similarity measurement of hidden representations.
We believe this new geometric perspective on the underlying mechanism driving
universal attacks could help us gain deeper insight into the internal workings
and failure modes of LLMs, thus enabling their mitigation.