RADIUS: Ranking, Distribution, and Significance - A Comprehensive Alignment Suite for Survey Simulation
This work addresses the need for standardized evaluation in survey simulation, which is critical for decision-making applications, though it is incremental as it builds on existing metrics by adding ranking alignment.
The authors tackled the problem of evaluating survey simulations with LLMs, where existing metrics are fragmented and overlook ranking alignment, by introducing RADIUS, a comprehensive alignment suite that includes ranking and distribution measures with significance testing, resulting in an open-source tool for more meaningful and reproducible assessment.
Simulation of surveys using LLMs is emerging as a powerful application for generating human-like responses at scale. Prior work evaluates survey simulation using metrics borrowed from other domains, which are often ad hoc, fragmented, and non-standardized, leading to results that are difficult to compare. Moreover, existing metrics focus mainly on accuracy or distributional measures, overlooking the critical dimension of ranking alignment. In practice, a simulation can achieve high accuracy while still failing to capture the option most preferred by humans - a distinction that is critical in decision-making applications. We introduce RADIUS, a comprehensive two-dimensional alignment suite for survey simulation that captures: 1) RAnking alignment and 2) DIstribUtion alignment, each complemented by statistical Significance testing. RADIUS highlights the limitations of existing metrics, enables more meaningful evaluation of survey simulation, and provides an open-source implementation for reproducible and comparable assessment.