Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction
For researchers working on visual planning with VLMs, this work provides a training-free strategy that improves efficiency by reusing learned patterns, though the gains are incremental over existing thinking-with-images approaches.
The paper addresses the perceptual bottleneck in Vision-Language Models (VLMs) for visual planning tasks. By introducing Pattern Induction, an online inductive learning strategy that discovers reusable visual patterns, the method achieves a desirable balance between accuracy and efficiency, solving tasks beyond initial VLM capabilities while reducing computational overhead.
Planning from raw visual input remains a significant challenge for current Vision-Language Models (VLMs), when the complexity of input is beyond their one-step perception capability. Motivated by recent advances in Thinking with Images (TWI), a reasonable solution is to decompose the perception process into simpler steps by iteratively acquiring and incorporating local visual evidence. However, even though current VLMs are well-trained in general TWI ability, their perceptual bottleneck in the planning domain remains. To tackle this challenge, we formulate TWI as a tool to gradually build and reflect an accurate internal world model. We find that the resulting training-free planning strategy enables VLMs to solve tasks that are far beyond their initial capabilities, at the cost that too many TWI operations would significantly increase the computational overhead. To further improve efficiency, we propose Pattern Inference, a novel TWI strategy enabling VLMs to actively recognize known visual patterns in the new tasks and directly infer local world model structures. To obtain these patterns, we propose Pattern Induction, an online inductive learning strategy treating visual patterns as composite and reusable experts, which are autonomously discovered and optimized from experience. Experimental evaluations in FrozenLake, Crafter and CubeBench domains show that our approaches achieve a desirable balance between accuracy and efficiency.