Weather-Robust Scene Semantics with Vision-Aligned 4D Radar
For autonomous driving and outdoor robotics, this work enables reliable semantic understanding in weather conditions where cameras and LiDAR fail.
The paper presents a method for weather-robust scene understanding by aligning 4D radar data with frozen vision-language models, achieving under 90% hallucination in adverse weather where cameras fail.
Cameras and LiDAR degrade in rain, fog, and snow, while millimeter-wave radar remains largely unaffected. We align a radar encoder to frozen SigLIP vision embeddings and decode structured scene captions through a frozen vision-language model (VLM) with approximately 7M trainable parameters. On K-RADAR with held-out fog, light snow, and heavy snow sequences, all radar configurations outperform a camera baseline that collapses to over 90% hallucination. We identify a token-norm mismatch as the dominant failure mode when bridging radar to a frozen VLM and show that projector-output LayerNorm resolves it. Analysis of encoder complexity, caption format, and pooling strategy reveals tradeoffs that inform future radar-VLM pipeline design.