Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoning
This work addresses the challenge of enhancing complex reasoning in language models for AI applications, representing an incremental advancement by integrating structure reasoning into existing pre-training methods.
The paper tackles the problem of endowing pre-trained language models with structure reasoning skills by proposing a unified learning framework that combines explicit structure reasoning with language pre-training, achieving significant improvements on four datasets for complex reasoning tasks.
Recent pre-trained language models (PLMs) equipped with foundation reasoning skills have shown remarkable performance on downstream complex tasks. However, the significant structure reasoning skill has been rarely studied, which involves modeling implicit structure information within the text and performing explicit logical reasoning over them to deduce the conclusion. This paper proposes a unified learning framework that combines explicit structure reasoning and language pre-training to endow PLMs with the structure reasoning skill. It first identifies several elementary structures within contexts to construct structured queries and performs step-by-step reasoning along the queries to identify the answer entity. The fusion of textual semantics and structure reasoning is achieved by using contextual representations learned by PLMs to initialize the representation space of structures, and performing stepwise reasoning on this semantic representation space. Experimental results on four datasets demonstrate that the proposed model achieves significant improvements in complex reasoning tasks involving diverse structures, and shows transferability to downstream tasks with limited training data and effectiveness for complex reasoning of KGs modality.