Addressing Order Sensitivity of In-Context Demonstration Examples in
  Causal Language Models
Yanzheng Xiang Hanqi Yan Lin Gui Yulan He
In-context learning has become a popular paradigm in natural language
processing. However, its performance can be significantly influenced by the
order of in-context demonstration examples. In this paper, we found that causal
language models (CausalLMs) are more sensitive to this order compared to prefix
language models (PrefixLMs). We attribute this phenomenon to the
auto-regressive attention masks within CausalLMs, which restrict each token
from accessing information from subsequent tokens. This results in different
receptive fields for samples at different positions, thereby leading to
representation disparities across positions. To tackle this challenge, we
introduce an unsupervised fine-tuning method, termed the Information-Augmented
and Consistency-Enhanced approach. This approach utilizes contrastive learning
to align representations of in-context examples across different positions and
introduces a consistency loss to ensure similar representations for inputs with
different permutations. This enhances the model's predictive consistency across
permutations. Experimental results on four benchmarks suggest that our proposed
method can reduce the sensitivity to the order of in-context examples and
exhibit robust generalizability, particularly when demonstrations are sourced
from a pool different from that used in the training phase, or when the number
of in-context examples differs from what is used during training.