Visual Self-Refinement for Autoregressive Models
This work addresses a domain-specific issue in vision-language modeling for researchers and practitioners, offering an incremental improvement.
The paper tackles the problem of suboptimal visual sequence generation in autoregressive models due to spatial-visual and sequential dependencies conflicts, proposing a plug-and-play refinement module that improves generation quality and semantic consistency.
Autoregressive models excel in sequential modeling and have proven to be effective for vision-language data. However, the spatial nature of visual signals conflicts with the sequential dependencies of next-token prediction, leading to suboptimal results. This work proposes a plug-and-play refinement module to enhance the complex spatial correspondence modeling within the generated visual sequence. This module operates as a post-pretraining step to jointly refine all generated tokens of autoregressive model, enhancing vision-language modeling under a shared sequential prediction framework. By leveraging global context and relationship across the tokens, our method mitigates the error accumulation issue within the sequential generation. Experiments demonstrate that the proposed method improves the generation quality, enhancing the model's ability to produce semantically consistent results.