A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving
For autonomous driving researchers, K-Risk addresses the lack of annotated high-risk scenarios by providing a standardized, multi-dimensional dataset that bridges trajectory data with interpretable language supervision.
The paper introduces K-Risk, a dataset of 31,398 high-risk driving events (including 1,036 near-collisions) with LLM-generated semantic annotations, combining 20 trajectory datasets from multiple countries. It provides structured descriptions, risk analyses, and action recommendations validated in a closed-loop simulator.
Safe autonomous driving requires both rapid responses to common high-risk events and deeper reasoning over rare, extreme long-tail scenarios in traffic safety. These scenarios are severely under-represented in naturalistic driving data, and existing trajectory and language-augmented datasets seldom provide high-risk event labels, semantic annotations, and verifiable safety signals. Here we present K-Risk, a knowledge-augmented dataset that combines structured driving trajectories with large language model generated semantic annotations for safety-critical driving scenarios. K-Risk integrates 20 human-driven and autonomous-vehicle trajectory datasets from Europe, China, and the United States, covering highways, urban freeways, intersections, and roundabouts. Using a unified risk-centric extraction pipeline, K-Risk curates 31,398 high-risk events, together with a 1,036-event extreme subset of near-collision cases. Each event is released as a synchronized trajectory, metadata, and language triplet containing structured scenario descriptions, abnormal-behavior notifications, and, for a representative subset, causal risk analyses and action recommendations validated through a closed-loop simulator with iterative reflection. By combining multi-dimensional risk annotations, interpretable language supervision, and verifiable decisions, K-Risk bridges structured traffic trajectories, semantic reasoning, and decision supervision, providing a standardized foundation for developing and evaluating next-generation risk-aware autonomous driving agents.