Multimodal Large Language Model Framework for Safe and Interpretable Grid-Integrated EVs
This work addresses safety and interpretability issues for drivers and grid operators in e-mobility, though it is incremental as it applies existing methods like YOLOv8 and LLMs to a new domain.
The paper tackles the challenge of ensuring safe and interpretable interactions for electric vehicles (EVs) integrated into smart grids by developing a multimodal large language model (LLM) framework that processes sensor data to generate natural-language alerts for drivers, validated with real-world data to demonstrate effectiveness in critical situations like proximity to pedestrians and other vehicles.
The integration of electric vehicles (EVs) into smart grids presents unique opportunities to enhance both transportation systems and energy networks. However, ensuring safe and interpretable interactions between drivers, vehicles, and the surrounding environment remains a critical challenge. This paper presents a multi-modal large language model (LLM)-based framework to process multimodal sensor data - such as object detection, semantic segmentation, and vehicular telemetry - and generate natural-language alerts for drivers. The framework is validated using real-world data collected from instrumented vehicles driving on urban roads, ensuring its applicability to real-world scenarios. By combining visual perception (YOLOv8), geocoded positioning, and CAN bus telemetry, the framework bridges raw sensor data and driver comprehension, enabling safer and more informed decision-making in urban driving scenarios. Case studies using real data demonstrate the framework's effectiveness in generating context-aware alerts for critical situations, such as proximity to pedestrians, cyclists, and other vehicles. This paper highlights the potential of LLMs as assistive tools in e-mobility, benefiting both transportation systems and electric networks by enabling scalable fleet coordination, EV load forecasting, and traffic-aware energy planning. Index Terms - Electric vehicles, visual perception, large language models, YOLOv8, semantic segmentation, CAN bus, prompt engineering, smart grid.