Deeper Inside Deep ViT
This work addresses training challenges for large vision models, offering incremental improvements for researchers in computer vision.
The authors tackled the practical utility and training instability of large-scale vision models like ViT-22B by examining its behavior in local environments and making modifications to stabilize training, resulting in ViT-22B outperforming ViT under the same parameter size and exploring its suitability for image generation.
There have been attempts to create large-scale structures in vision models similar to LLM, such as ViT-22B. While this research has provided numerous analyses and insights, our understanding of its practical utility remains incomplete. Therefore, we examine how this model structure reacts and train in a local environment. We also highlight the instability in training and make some model modifications to stabilize it. The ViT-22B model, trained from scratch, overall outperformed ViT in terms of performance under the same parameter size. Additionally, we venture into the task of image generation, which has not been attempted in ViT-22B. We propose an image generation architecture using ViT and investigate which between ViT and ViT-22B is a more suitable structure for image generation.