Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT
This work addresses enhancing visual classification models for researchers and practitioners, but it appears incremental as it applies known prompt techniques to a new modality.
The paper tackled adapting prompt design from instruction tuning to visual transformers for image classification, introducing Instruction-ViT with multi-modal prompts, and reported improved performance and domain adaptability in experiments on image captioning tasks.
Prompts have been proven to play a crucial role in large language models, and in recent years, vision models have also been using prompts to improve scalability for multiple downstream tasks. In this paper, we focus on adapting prompt design based on instruction tuning into a visual transformer model for image classification which we called Instruction-ViT. The key idea is to implement multi-modal prompts (text or image prompt) related to category information to guide the fine-tuning of the model. Based on the experiments of several image captionining tasks, the performance and domain adaptability were improved. Our work provided an innovative strategy to fuse multi-modal prompts with better performance and faster adaptability for visual classification models.