DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models
Enables VLMs to handle precise continuous outputs, addressing a key limitation for perception and planning tasks.
DRIFT adapts pretrained vision-language models to continuous output tasks (e.g., temporal localization, robotic control) by combining a coarse base predictor with a flow-matching refinement module, consistently outperforming regression and generative baselines across multiple architectures.
Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.