Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer
This work provides a new perspective on transformer mechanisms for logical reasoning in natural language processing, though it is incremental as it builds on existing attention models.
The paper tackles the challenge of developing neural architectures for logical reasoning by proposing a symbolic reasoning architecture that chains join operators to model logical expressions, and demonstrates that multi-head self-attention in transformers can be interpreted as a neural operator implementing the union bound of the join operator in probabilistic predicate space.
Developing neural architectures that are capable of logical reasoning has become increasingly important for a wide range of applications (e.g., natural language processing). Towards this grand objective, we propose a symbolic reasoning architecture that chains many join operators together to model output logical expressions. In particular, we demonstrate that such an ensemble of join-chains can express a broad subset of ''tree-structured'' first-order logical expressions, named FOET, which is particularly useful for modeling natural languages. To endow it with differentiable learning capability, we closely examine various neural operators for approximating the symbolic join-chains. Interestingly, we find that the widely used multi-head self-attention module in transformer can be understood as a special neural operator that implements the union bound of the join operator in probabilistic predicate space. Our analysis not only provides a new perspective on the mechanism of the pretrained models such as BERT for natural language understanding but also suggests several important future improvement directions.