RealBehavior: A Framework for Faithfully Characterizing Foundation Models' Human-like Behavior Mechanisms
This addresses the issue of ensuring accurate behavioral analysis in AI models for researchers and developers, though it is incremental in refining existing evaluation methods.
The paper tackles the problem of unfaithful characterization of human-like behaviors in foundation models by introducing the RealBehavior framework, which assesses faithfulness through reproducibility, consistency, and generalizability, finding that direct application of psychological tools is insufficient.
Reports of human-like behaviors in foundation models are growing, with psychological theories providing enduring tools to investigate these behaviors. However, current research tends to directly apply these human-oriented tools without verifying the faithfulness of their outcomes. In this paper, we introduce a framework, RealBehavior, which is designed to characterize the humanoid behaviors of models faithfully. Beyond simply measuring behaviors, our framework assesses the faithfulness of results based on reproducibility, internal and external consistency, and generalizability. Our findings suggest that a simple application of psychological tools cannot faithfully characterize all human-like behaviors. Moreover, we discuss the impacts of aligning models with human and social values, arguing for the necessity of diversifying alignment objectives to prevent the creation of models with restricted characteristics.