ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery
This addresses the need for dedicated evaluation benchmarks in thermal vision-language understanding, which is critical for applications like surveillance and autonomous driving, but the work is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the problem that vision-language models (VLMs) fail to generalize to thermal imagery, which encodes temperature rather than color, by introducing ThermEval-B, a benchmark of 55,000 thermal visual question answering pairs. The results show that 25 evaluated VLMs consistently fail at temperature-grounded reasoning and degrade under colormap transformations, with only marginal gains from interventions like fine-tuning.
Vision language models (VLMs) achieve strong performance on RGB imagery, but they do not generalize to thermal images. Thermal sensing plays a critical role in settings where visible light fails, including nighttime surveillance, search and rescue, autonomous driving, and medical screening. Unlike RGB imagery, thermal images encode physical temperature rather than color or texture, requiring perceptual and reasoning capabilities that existing RGB-centric benchmarks do not evaluate. We introduce ThermEval-B, a structured benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal vision language understanding. ThermEval-B integrates public datasets with our newly collected ThermEval-D, the first dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments. Evaluating 25 open-source and closed-source VLMs, we find that models consistently fail at temperature-grounded reasoning, degrade under colormap transformations, and default to language priors or fixed responses, with only marginal gains from prompting or supervised fine-tuning. These results demonstrate that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions, positioning ThermEval as a benchmark to drive progress in thermal vision language modeling.