ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation
This work addresses the need for controllable and realistic driving scenario generation for autonomous vehicle testing, offering a novel approach that integrates multimodal control with vectorized latent spaces.
ScenarioControl introduces the first vision-language control mechanism for learned driving scenario generation, enabling diverse, realistic 3D scenario rollouts from text or image inputs. The method outperforms all tested baselines in control adherence and fidelity.
We introduce ScenarioControl, the first vision-language control mechanism for learned driving scenario generation. Given a text prompt or an input image, Scenario-Control synthesizes diverse, realistic 3D scenario rollouts - including map, 3D boxes of reactive actors over time, pedestrians, driving infrastructure, and ego camera observations. The method generates scenes in a vectorized latent space that represents road structure and dynamic agents jointly. To connect multimodal control with sparse vectorized scene elements, we propose a cross-global control mechanism that integrates crossattention with a lightweight global-context branch, enabling fine-grained control over road layout and traffic conditions while preserving realism. The method produces temporally consistent scenario rollouts from the perspectives different actors in the scene, supporting long-horizon continuation of driving scenarios. To facilitate training and evaluation, we release a dataset with text annotations aligned to vectorized map structures. Extensive experiments validate that the control adherence and fidelity of ScenarioControl compare favorable to all tested methods across all experiments. Project webpage: https://light.princeton.edu/ScenarioControl