Thinking in 360°: Humanoid Visual Search in the Wild
This work addresses the challenge of creating more human-like AI agents for visual search in complex real-world environments, representing a novel benchmark and method for an important domain-specific problem.
The paper tackles the problem of developing embodied visual search agents that can efficiently search 360° panoramic images like humans, by proposing a humanoid visual search approach and introducing the H* Bench benchmark with challenging real-world scenes. The results show that even top models initially achieve only ~30% success, but post-training techniques improve Qwen2.5-VL's success rate over threefold to 47.38% for object search and 24.94% for path search.
Humans rely on the synergistic control of head (cephalomotor) and eye (oculomotor) to efficiently search for visual information in 360°. However, prior approaches to visual search are limited to a static image, neglecting the physical embodiment and its interaction with the 3D world. How can we develop embodied visual search agents as efficient as humans while bypassing the constraints imposed by real-world hardware? To this end, we propose humanoid visual search where a humanoid agent actively rotates its head to search for objects or paths in an immersive world represented by a 360° panoramic image. To study visual search in visually-crowded real-world scenarios, we build H* Bench, a new benchmark that moves beyond household scenes to challenging in-the-wild scenes that necessitate advanced visual-spatial reasoning capabilities, such as transportation hubs, large-scale retail spaces, urban streets, and public institutions. Our experiments first reveal that even top-tier proprietary models falter, achieving only ~30% success in object and path search. We then use post-training techniques to enhance the open-source Qwen2.5-VL, increasing its success rate by over threefold for both object search (14.83% to 47.38%) and path search (6.44% to 24.94%). Notably, the lower ceiling of path search reveals its inherent difficulty, which we attribute to the demand for sophisticated spatial commonsense. Our results not only show a promising path forward but also quantify the immense challenge that remains in building MLLM agents that can be seamlessly integrated into everyday human life.