PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models
This work enables physics-aware semantic understanding in VLMs for the first time, addressing a critical bottleneck in handling optically ambiguous scenes.
PolarVLM integrates polarimetric physical parameters into vision-language models to resolve optical ambiguities like reflections and transparent objects, achieving a 25.4% overall improvement over RGB baselines across five tasks, with gains of 26.6% in reflection recognition and 34.0% in glass counting.
Mainstream vision-language models (VLMs) fundamentally struggle with severe optical ambiguities, such as reflections and transparent objects, due to the inherent limitations of standard RGB inputs. While polarization imaging captures polarimetric physical parameters that resolve these ambiguities, existing methods are constrained by fixed-format outputs and remain isolated from open-ended reasoning. To bridge this semantic-physical gap, we introduce PolarVLM, the first multimodal framework integrating polarimetric physical parameters into VLMs. By employing a dual-stream architecture and a progressive two-stage training strategy, PolarVLM effectively prevents physical misinterpretations while preserving general visual abilities. Complementing our architecture, we construct PolarVQA, the first benchmark for polarization-aware VQA, featuring 75K physics-grounded instruction-tuning pairs targeting reflective and transparent scenes. Experiments show that PolarVLM surpasses the RGB baseline by 25.4% overall across five evaluation tasks, with remarkable gains of 26.6% in reflection recognition and 34.0% in glass counting, successfully unlocking physics-aware semantic understanding.