CrowdVLA: Embodied Vision-Language-Action Agents for Context-Aware Crowd Simulation
This addresses the limitation of existing crowd simulation methods that reduce navigation to geometry and collision avoidance, producing incremental improvements for applications in robotics, urban planning, and virtual environments.
The paper tackles the problem of creating more intentional and context-aware crowd simulations by introducing CrowdVLA, which models pedestrians as Vision-Language-Action agents that interpret scene semantics and social norms. The result is a shift from motion-centric synthesis to perception-driven decision making, enabling crowds that move meaningfully rather than just realistically.
Crowds do not merely move; they decide. Human navigation is inherently contextual: people interpret the meaning of space, social norms, and potential consequences before acting. Sidewalks invite walking, crosswalks invite crossing, and deviations are weighed against urgency and safety. Yet most crowd simulation methods reduce navigation to geometry and collision avoidance, producing motion that is plausible but rarely intentional. We introduce CrowdVLA, a new formulation of crowd simulation that models each pedestrian as a Vision-Language-Action (VLA) agent. Instead of replaying recorded trajectories, CrowdVLA enables agents to interpret scene semantics and social norms from visual observations and language instructions, and to select actions through consequence-aware reasoning. CrowdVLA addresses three key challenges-limited agent-centric supervision in crowd datasets, unstable per-frame control, and success-biased datasets-through: (i) agent-centric visual supervision via semantically reconstructed environments and Low-Rank Adaptation (LoRA) fine-tuning of a pretrained vision-language model, (ii) a motion skill action space that bridges symbolic decision making and continuous locomotion, and (iii) exploration-based question answering that exposes agents to counterfactual actions and their outcomes through simulation rollouts. Our results shift crowd simulation from motion-centric synthesis toward perception-driven, consequence-aware decision making, enabling crowds that move not just realistically, but meaningfully.