VIVECaption: A Split Approach to Caption Quality Improvement
This work addresses the growing need for high-quality training data for enterprise AI development, specifically for teams seeking to improve caption-image alignment without relying on potentially copyright-protected web-scraped content. It is an incremental improvement in caption quality.
This paper tackles the problem of poor caption quality, which is a bottleneck in training high-quality text-to-image and text-to-video generative models. The authors introduce VIVECaption, a two-sided approach involving a gold-standard dataset creation methodology and a model alignment strategy. They demonstrate that using a finetuned character detection model in an image captioning pipeline significantly improves holistic image-caption alignment quality.
Caption quality has emerged as a critical bottleneck in training high-quality text-to-image (T2I) and text-to-video (T2V) generative models. While visual language models (VLMs) are commonly deployed to generate captions from visual data, they suffer from hallucinations, poor compositional reasoning, and limited fine-grained understanding, resulting in misaligned image-caption pairs that degrade downstream model performance. This technical report introduces VIVECaption, a systematic two-sided approach to caption quality improvement. We first establish a comprehensive taxonomy of caption evaluation metrics, distinguishing between "universal" and "instance-grounded" metrics, with the ultimate goal of showcasing the use-cases and tradeoffs between different caption quality metrics. We then use this language to describe our two-sided approach to caption quality improvement: (1) a gold-standard dataset creation methodology using stratified sampling and (2) a model alignment strategy encompassing context alignment and parameter-level finetuning using SFT. We demonstrate our methodology on open-source models, focusing on structured caption formats that enable better parsing and downstream utilization. We ultimately show that using a finetuned character detection model in an image captioning pipeline significantly improves holistic image-caption alignment quality. Our work addresses the growing need for high-quality "vegan" training data in enterprise AI development, providing practical solutions for teams seeking to improve caption-image alignment without relying on potentially copyright-protected web-scraped content.