Simple and Effective Transfer Learning for Neuro-Symbolic Integration
This work addresses generalization and reasoning challenges in deep learning for researchers and practitioners in AI, though it is incremental as it builds on existing NeSy methods.
The paper tackles the issues of slow convergence, learning difficulties, and local minima in Neuro-Symbolic Integration (NeSy) methods by proposing a transfer learning approach that pretrains a neural network on the downstream task and injects its weights into the perceptual part of the NeSy model, resulting in consistent improvements across various SOTA methods and datasets.
Deep Learning (DL) techniques have achieved remarkable successes in recent years. However, their ability to generalize and execute reasoning tasks remains a challenge. A potential solution to this issue is Neuro-Symbolic Integration (NeSy), where neural approaches are combined with symbolic reasoning. Most of these methods exploit a neural network to map perceptions to symbols and a logical reasoner to predict the output of the downstream task. These methods exhibit superior generalization capacity compared to fully neural architectures. However, they suffer from several issues, including slow convergence, learning difficulties with complex perception tasks, and convergence to local minima. This paper proposes a simple yet effective method to ameliorate these problems. The key idea involves pretraining a neural model on the downstream task. Then, a NeSy model is trained on the same task via transfer learning, where the weights of the perceptual part are injected from the pretrained network. The key observation of our work is that the neural network fails to generalize only at the level of the symbolic part while being perfectly capable of learning the mapping from perceptions to symbols. We have tested our training strategy on various SOTA NeSy methods and datasets, demonstrating consistent improvements in the aforementioned problems.