ICDPO: Effectively Borrowing Alignment Capability of Others via
  In-context Direct Preference Optimization
Feifan Song Yuxuan Fan Xin Zhang Peiyi Wang Houfeng Wang
Large Language Models (LLMs) rely on Human Preference Alignment (HPA) to
ensure the generation of safe content. Due to the heavy cost associated with
fine-tuning, fine-tuning-free methods have emerged, typically modifying LLM
decoding with external auxiliary methods. However, these methods do not
essentially enhance the LLM itself. In this paper, we rethink the derivation
procedures of DPO, based on which we conversely build an instant scorer using
the states of the LLM before and after In-context Learning (ICL). Accordingly,
we propose a novel approach called In-Context Direct Preference Optimization
(ICDPO). It enables LLMs to borrow the HPA capabilities from superior LLMs with
ICL, generating well-aligned responses as estimated by the aforementioned
instant scorer, thereby enhancing the final performance. ICDPO can be further
enhanced with a two-stage retriever and an upgraded scorer, both offering
benefits. Extensive experiments show its effectiveness, particularly in
outperforming two fine-tuning-free baselines, and it exhibits competitiveness
with SFT + LoRA. We also conduct detailed analyses to offer comprehensive
insights into ICDPO.