Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding
This addresses a specific bottleneck in VLMs for improving multimodal AI performance, though it appears incremental as it builds on existing encoding methods.
The paper tackles the problem of irrational visual position encoding in Vision-language Models (VLMs), which inhibits comprehensive perception across granularities, and proposes Pyramid-descent Visual Position Encoding (PyPE) to enhance this, resulting in consistent improvements in general capabilities across various model sizes.
Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models' comprehensive perception performance across different levels of granularity. In this work, we propose Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. By assigning visual position indexes from the periphery to the center and expanding the central receptive field incrementally, PyPE addresses the limitations of traditional raster-scan methods and mitigates the long-term decay effects induced by Rotary Position Embedding (RoPE). Our method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements and countering the over-reliance on anchor tokens. Extensive experimental evaluations demonstrate that PyPE consistently improves the general capabilities of VLMs across various sizes. Code is available at https://github.com/SakuraTroyChen/PyPE.