Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?
This addresses the trustworthiness of LLMs in computational social science and role-playing applications, but it is incremental as it builds on existing simulation methods.
The paper tackled the problem of reliability in LLM-based social simulations by introducing TrustSim, an evaluation dataset covering 10 topics, and found inconsistencies in simulated roles across 14 LLMs, with no strong correlation to general performance. It proposed AdaORPO, a reinforcement learning-based algorithm, to improve reliability across 7 LLMs.
Large Language Models (LLMs) are increasingly employed for simulations, enabling applications in role-playing agents and Computational Social Science (CSS). However, the reliability of these simulations is under-explored, which raises concerns about the trustworthiness of LLMs in these applications. In this paper, we aim to answer ``How reliable is LLM-based simulation?'' To address this, we introduce TrustSim, an evaluation dataset covering 10 CSS-related topics, to systematically investigate the reliability of the LLM simulation. We conducted experiments on 14 LLMs and found that inconsistencies persist in the LLM-based simulated roles. In addition, the consistency level of LLMs does not strongly correlate with their general performance. To enhance the reliability of LLMs in simulation, we proposed Adaptive Learning Rate Based ORPO (AdaORPO), a reinforcement learning-based algorithm to improve the reliability in simulation across 7 LLMs. Our research provides a foundation for future studies to explore more robust and trustworthy LLM-based simulations.