Improving Fine-grained Visual Understanding in VLMs through Text-Only Training
This offers a more efficient pathway for advancing VLM capabilities, particularly in resource-constrained environments, though it appears incremental as it builds on existing ideas about language understanding in VLMs.
The paper tackles the problem of high resource requirements in training Visual-Language Models (VLMs) by investigating text-only training to enhance fine-grained visual understanding, finding it comparable to conventional image-text training while reducing computational costs.
Visual-Language Models (VLMs) have become a powerful tool for bridging the gap between visual and linguistic understanding. However, the conventional learning approaches for VLMs often suffer from limitations, such as the high resource requirements of collecting and training image-text paired data. Recent research has suggested that language understanding plays a crucial role in the performance of VLMs, potentially indicating that text-only training could be a viable approach. In this work, we investigate the feasibility of enhancing fine-grained visual understanding in VLMs through text-only training. Inspired by how humans develop visual concept understanding, where rich textual descriptions can guide visual recognition, we hypothesize that VLMs can also benefit from leveraging text-based representations to improve their visual recognition abilities. We conduct comprehensive experiments on two distinct domains: fine-grained species classification and cultural visual understanding tasks. Our findings demonstrate that text-only training can be comparable to conventional image-text training while significantly reducing computational costs. This suggests a more efficient and cost-effective pathway for advancing VLM capabilities, particularly valuable in resource-constrained environments.