Step-Level Preference Learning for Generative Agents in Social Simulations
This work addresses the lack of fine-grained human supervision for intermediate decision steps in generative agents, improving their long-horizon behavior in social simulations.
The authors introduce a method to collect step-level human preference annotations for LLM-based generative agents in social simulations, resulting in a dataset of 57K annotations. Step-level preference learning improves simulation fidelity, coordination, and interaction quality, leading to more socially effective agent behavior.
Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.