RealBehavior: A Framework for Faithfully Characterizing Foundation
  Models' Human-like Behavior Mechanisms
Enyu Zhou Rui Zheng Zhiheng Xi Songyang Gao Xiaoran Fan Zichu Fei Jingting Ye Tao Gui Qi Zhang Xuanjing Huang
Reports of human-like behaviors in foundation models are growing, with
psychological theories providing enduring tools to investigate these behaviors.
However, current research tends to directly apply these human-oriented tools
without verifying the faithfulness of their outcomes. In this paper, we
introduce a framework, RealBehavior, which is designed to characterize the
humanoid behaviors of models faithfully. Beyond simply measuring behaviors, our
framework assesses the faithfulness of results based on reproducibility,
internal and external consistency, and generalizability. Our findings suggest
that a simple application of psychological tools cannot faithfully characterize
all human-like behaviors. Moreover, we discuss the impacts of aligning models
with human and social values, arguing for the necessity of diversifying
alignment objectives to prevent the creation of models with restricted
characteristics.