Neural Theorem Provers Do Not Learn Rules Without Exploration
This addresses a critical issue for researchers in neural symbolic processing by revealing algorithmic limitations in learning true relationships, though it is incremental as it builds on existing methods.
The paper tackled the problem of neural theorem provers failing to learn underlying logical rules from data, showing that the Neural Theorem Proving model struggles to recover relationships in synthetic datasets except in simple cases. By modifying the algorithm to increase exploration, performance improved sharply, with concrete gains demonstrated in diagnostic experiments.
Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks. The Neural Theorem Proving (NTP) model by Rocktaschel et al (2017) learns embeddings for concepts and performs logical unification. While NTP is promising and effective in predicting facts accurately, we have little knowledge how well it can extract true relationship among data. To this end, we create synthetic logical datasets with injected relationships, which can be generated on-the-fly, to test neural-based relation learning algorithms including NTP. We show that it has difficulty recovering relationships in all but the simplest settings. Critical analysis and diagnostic experiments suggest that the optimization algorithm suffers from poor local minima due to its greedy winner-takes-all strategy in identifying the most informative structure (proof path) to pursue. We alter the NTP algorithm to increase exploration, which sharply improves performance. We argue and demonstate that it is insightful to benchmark with synthetic data with ground-truth relationships, for both evaluating models and revealing algorithmic issues.