MetaAligner: Conditional Weak-to-Strong Correction for Generalizable
  Multi-Objective Alignment of Language Models
Kailai Yang Zhiwei Liu Qianqian Xie Tianlin Zhang Nirui Song Jimin Huang Ziyan Kuang Sophia Ananiadou
Recent advancements in large language models (LLMs) aim to tackle
heterogeneous human expectations and values via multi-objective preference
alignment. However, existing methods are parameter-adherent to the policy
model, leading to two key limitations: (1) the high-cost repetition of their
alignment algorithms for each new target model; (2) they cannot expand to
unseen objectives due to their static alignment objectives. In this work, we
propose Meta-Objective Aligner (MetaAligner), a model that performs conditional
weak-to-strong correction for weak responses to approach strong responses.
MetaAligner is the first policy-agnostic and generalizable method for
multi-objective preference alignment, which enables plug-and-play alignment by
decoupling parameter updates from the policy models and facilitates zero-shot
preference alignment for unseen objectives via in-context learning.
Experimental results show that MetaAligner achieves significant and balanced
improvements in multi-objective alignments on 11 policy models with up to 63x
more parameters, and outperforms previous alignment methods with down to 22.27x
less computational resources. The model also accurately aligns with unseen
objectives, marking the first step towards generalizable multi-objective
preference alignment.