Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment
For researchers using Vision Transformers, this work clarifies the limited generalizability of the register solution for attention artifacts, which is an incremental contribution.
This paper reproduces and evaluates the claim that adding registers eliminates attention map artifacts in Vision Transformers, finding that while the claim holds for some models (DINO, DINOv2, OpenCLIP, DeiT3), it does not generalize universally, and the impact varies with model size.
Training Vision Transformers (ViTs) presents significant challenges, one of which is the emergence of artifacts in attention maps, hindering their interpretability. Darcet et al. (2024) investigated this phenomenon and attributed it to the need of ViTs to store global information beyond the [CLS] token. They proposed a novel solution involving the addition of empty input tokens, named registers, which successfully eliminate artifacts and improve the clarity of attention maps. In this work, we reproduce the findings of Darcet et al. (2024) and evaluate the generalizability of their claims across multiple models, including DINO, DINOv2, OpenCLIP, and DeiT3. While we confirm the validity of several of their key claims, our results reveal that some claims do not extend universally to other models. Additionally, we explore the impact of model size, extending their findings to smaller models. Finally, we untie terminology inconsistencies found in the original paper and explain their impact when generalizing to a wider range of models.