Transferability Between Understanding and Generation in Unified Multimodal Models
This work provides practical insights for training unified multimodal models, addressing the problem of improving generative capabilities without degrading visual quality.
The paper investigates how understanding and generation tasks transfer in unified multimodal models, finding that fully shared architectures enable cross-task transfer. By training understanding tasks instead of generation, they improve generative capabilities (e.g., counting, spatial relations) while minimizing distribution shift.
Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.