Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems
This work addresses the robustness of AI systems for vision-language tasks, highlighting complementary strengths and the need for diverse testing, but it is incremental as it compares existing paradigms without introducing new methods.
The study investigated how end-to-end and neuro-symbolic vision-language reasoning systems generalize across out-of-distribution settings, finding that end-to-end systems show significant performance drops in all tests, while neuro-symbolic methods perform better on most generalization tests but struggle with cross-benchmark transfer.
For vision-and-language reasoning tasks, both fully connectionist, end-to-end methods and hybrid, neuro-symbolic methods have achieved high in-distribution performance. In which out-of-distribution settings does each paradigm excel? We investigate this question on both single-image and multi-image visual question-answering through four types of generalization tests: a novel segment-combine test for multi-image queries, contrast set, compositional generalization, and cross-benchmark transfer. Vision-and-language end-to-end trained systems exhibit sizeable performance drops across all these tests. Neuro-symbolic methods suffer even more on cross-benchmark transfer from GQA to VQA, but they show smaller accuracy drops on the other generalization tests and their performance quickly improves by few-shot training. Overall, our results demonstrate the complementary benefits of these two paradigms, and emphasize the importance of using a diverse suite of generalization tests to fully characterize model robustness to distribution shift.