Asymmetric Idiosyncrasies in Multimodal Models
This work highlights a limitation in prompt-following ability for text-to-image models, specifically their failure to preserve stylistic nuances from input captions, which is important for researchers and developers aiming for more faithful image generation.
This paper investigates stylistic differences in captions generated by various captioning models and their preservation in images produced by text-to-image models. They found that text classifiers can identify the originating caption model with 99.70% accuracy, but image classifiers achieve at most 50% accuracy, indicating that text-to-image models largely erase these stylistic signatures.
In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption or the corresponding image, we train neural networks to predict the originating caption model. Our results show that text classification yields very high accuracy (99.70\%), indicating that captioning models embed distinctive stylistic signatures. In contrast, these signatures largely disappear in the generated images, with classification accuracy dropping to at most 50\% even for the state-of-the-art Flux model. To better understand this cross-modal discrepancy, we further analyze the data and find that the generated images fail to preserve key variations present in captions, such as differences in the level of detail, emphasis on color and texture, and the distribution of objects within a scene. Overall, our classification-based framework provides a novel methodology for quantifying both the stylistic idiosyncrasies of caption models and the prompt-following ability of text-to-image systems.