The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applications
This work addresses safety improvements in autonomous driving and Intelligent Transportation Systems, but it is incremental as it applies existing MLLMs to a new data type (thermal images).
This study tackled the problem of object detection in thermal images for transportation applications by using Multimodal Large Language Models (MLLMs) like GPT-4 and Gemini, achieving Mean Absolute Percentage Error (MAPE) rates such as 70.39% for pedestrian classification and varying results for other objects.
The integration of thermal imaging data with Multimodal Large Language Models (MLLMs) constitutes an exciting opportunity for improving the safety and functionality of autonomous driving systems and many Intelligent Transportation Systems (ITS) applications. This study investigates whether MLLMs can understand complex images from RGB and thermal cameras and detect objects directly. Our goals were to 1) assess the ability of the MLLM to learn from information from various sets, 2) detect objects and identify elements in thermal cameras, 3) determine whether two independent modality images show the same scene, and 4) learn all objects using different modalities. The findings showed that both GPT-4 and Gemini were effective in detecting and classifying objects in thermal images. Similarly, the Mean Absolute Percentage Error (MAPE) for pedestrian classification was 70.39% and 81.48%, respectively. Moreover, the MAPE for bike, car, and motorcycle detection were 78.4%, 55.81%, and 96.15%, respectively. Gemini produced MAPE of 66.53%, 59.35% and 78.18% respectively. This finding further demonstrates that MLLM can identify thermal images and can be employed in advanced imaging automation technologies for ITS applications.