Unnatural language processing: How do language models handle
  machine-generated prompts?
Corentin Kervadec Francesca Franzon Marco Baroni
Language model prompt optimization research has shown that semantically and
grammatically well-formed manually crafted prompts are routinely outperformed
by automatically generated token sequences with no apparent meaning or
syntactic structure, including sequences of vectors from a model's embedding
space. We use machine-generated prompts to probe how models respond to input
that is not composed of natural language expressions. We study the behavior of
models of different sizes in multiple semantic tasks in response to both
continuous and discrete machine-generated prompts, and compare it to the
behavior in response to human-generated natural-language prompts. Even when
producing a similar output, machine-generated and human prompts trigger
different response patterns through the network processing pathways, including
different perplexities, different attention and output entropy distributions,
and different unit activation profiles. We provide preliminary insight into the
nature of the units activated by different prompt types, suggesting that only
natural language prompts recruit a genuinely linguistic circuit.