Injecting Logical Constraints into Neural Networks via Straight-Through Estimators
This work addresses the problem of integrating symbolic reasoning into neural networks for neuro-symbolic AI, offering a scalable solution that is incremental over prior methods.
The paper tackles the challenge of incorporating discrete logical constraints into neural network training by using a straight-through estimator to represent constraints as a loss function, enabling gradient-based optimization. Experimental results show this method scales better than existing neuro-symbolic approaches by leveraging GPUs and batch training, and it applies to various network types like MLP, CNN, and GNN, reducing the need for labeled data.
Injecting discrete logical constraints into neural network learning is one of the main challenges in neuro-symbolic AI. We find that a straight-through-estimator, a method introduced to train binary neural networks, could effectively be applied to incorporate logical constraints into neural network learning. More specifically, we design a systematic way to represent discrete logical constraints as a loss function; minimizing this loss using gradient descent via a straight-through-estimator updates the neural network's weights in the direction that the binarized outputs satisfy the logical constraints. The experimental results show that by leveraging GPUs and batch training, this method scales significantly better than existing neuro-symbolic methods that require heavy symbolic computation for computing gradients. Also, we demonstrate that our method applies to different types of neural networks, such as MLP, CNN, and GNN, making them learn with no or fewer labeled data by learning directly from known constraints.