Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.
Blogger's Review: The proposed step-level preference learning method offers a new perspective for training generative agents, particularly in scenarios lacking fine-grained human annotations. Leveraging human preferences for supervision significantly enhances simulation outcomes. This approach has the potential to advance the field of social simulations, especially in more complex human behavior interaction contexts. Future research could explore how to implement this method in broader applications.