Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
This is an incremental survey paper that synthesizes existing research on alignment challenges in LVLMs for researchers in multimodal AI.
This survey examines the problem of alignment and misalignment in Large Vision-Language Models (LVLMs) through explainability, analyzing misalignment across object, attribute, and relational levels and reviewing mitigation strategies.
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in processing both visual and textual information. However, the critical challenge of alignment between visual and textual representations is not fully understood. This survey presents a comprehensive examination of alignment and misalignment in LVLMs through an explainability lens. We first examine the fundamentals of alignment, exploring its representational and behavioral aspects, training methodologies, and theoretical foundations. We then analyze misalignment phenomena across three semantic levels: object, attribute, and relational misalignment. Our investigation reveals that misalignment emerges from challenges at multiple levels: the data level, the model level, and the inference level. We provide a comprehensive review of existing mitigation strategies, categorizing them into parameter-frozen and parameter-tuning approaches. Finally, we outline promising future research directions, emphasizing the need for standardized evaluation protocols and in-depth explainability studies.