Multimodal Latent Reasoning via Predictive Embeddings
This addresses the challenge of efficient and reliable multimodal reasoning for visual language models, offering a more practical alternative to existing methods, though it is incremental in improving latent space learning.
The paper tackles the problem of high inference overhead and error-prone tool calls in tool-augmented multimodal reasoning for visual language models by proposing Pearl, a framework that learns predictive embeddings from expert tool-use trajectories in latent space, eliminating explicit tool invocation at inference time. Experiments show that Pearl matches or outperforms standard supervised fine-tuning and reconstruction-based latent reasoning approaches on multiple perception benchmarks.
Tool-augmented multimodal reasoning enables visual language models (VLMs) to improve perception by interacting with external tools (e.g., cropping, depth estimation). However, such approaches incur substantial inference overhead, require specialized supervision, and are prone to erroneous tool calls. We propose Pearl (Predictive Embedding Alignment for Reasoning in Latent space), a JEPA-inspired framework that learns from expert tool-use trajectories entirely in the latent space, eliminating the need for explicit tool invocation at inference time. Unlike reconstruction-based latent reasoning methods, which autoregressively generate latent tokens and suffer from training-inference mismatch and limited support for multi-step tool use, Pearl directly learns predictive embeddings from multimodal trajectories while preserving the standard vision-language generation pipeline: it is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls. Experiments across multiple perception benchmarks show that Pearl matches or outperforms standard supervised fine-tuning and reconstruction-based latent reasoning approaches. Furthermore, we provide empirical evidence that reconstruction-based methods primarily learn embeddings rather than image edits in latent space, motivating predictive embedding learning as a more principled alternative.