The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning
This work addresses the problem of safe and interpretable AI deployment for researchers and practitioners, though it is incremental as it builds on existing neuro-symbolic methods with foundation models.
The paper tackles the challenge of integrating neural and symbolic computation in Neuro-Symbolic AI by leveraging foundation models to reduce data labeling and manual engineering, resulting in NeSyGPT achieving superior accuracy and scalability in complex tasks.
Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI systems, as interpretable symbolic techniques provide formal behaviour guarantees. The challenge is how to effectively integrate neural and symbolic computation, to enable learning and reasoning from raw data. Existing pipelines that train the neural and symbolic components sequentially require extensive labelling, whereas end-to-end approaches are limited in terms of scalability, due to the combinatorial explosion in the symbol grounding problem. In this paper, we leverage the implicit knowledge within foundation models to enhance the performance in NeSy tasks, whilst reducing the amount of data labelling and manual engineering. We introduce a new architecture, called NeSyGPT, which fine-tunes a vision-language foundation model to extract symbolic features from raw data, before learning a highly expressive answer set program to solve a downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has superior accuracy over various baselines, and can scale to complex NeSy tasks. Finally, we highlight the effective use of a large language model to generate the programmatic interface between the neural and symbolic components, significantly reducing the amount of manual engineering required.