The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges
This is an incremental survey that synthesizes existing research on knowledge integration in visiolinguistic learning for researchers in AI and computer vision.
The survey examines how external knowledge sources like knowledge graphs and large language models address generalization gaps in visiolinguistic learning by filling missing knowledge in datasets, leading to hybrid architectures.
Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.