TULIP: Token-length Upgraded CLIP
This addresses a bottleneck for researchers and practitioners using CLIP-like models in applications requiring longer descriptions, though it is incremental as it builds on existing architectures.
The paper tackles the problem of representing long captions in vision-language models like CLIP, which are limited to 77 tokens due to fixed positional encodings, and proposes TULIP to upgrade token length with relative position encodings, resulting in improved performance on cross-modal tasks such as retrieval and text-to-image generation.
We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip.