Injecting Relational Structural Representation in Neural Networks for Question Similarity
This work addresses the challenge of leveraging syntactic structures in neural networks for question similarity, which is important for natural language processing applications, but it is incremental as it builds on existing methods like tree kernels and pre-training.
The paper tackles the problem of effectively using syntactic parsing information in neural networks for relational tasks like question similarity by injecting structural representations through a method involving SVM with tree kernels on limited gold data, predicting labels on a large corpus, and pre-training neural networks. The results show that neural networks trained with this approach achieve more accurate models, with improvements after fine-tuning on gold standard data, as demonstrated on Quora and SemEval datasets.
Effectively using full syntactic parsing information in Neural Networks (NNs) to solve relational tasks, e.g., question similarity, is still an open problem. In this paper, we propose to inject structural representations in NNs by (i) learning an SVM model using Tree Kernels (TKs) on relatively few pairs of questions (few thousands) as gold standard (GS) training data is typically scarce, (ii) predicting labels on a very large corpus of question pairs, and (iii) pre-training NNs on such large corpus. The results on Quora and SemEval question similarity datasets show that NNs trained with our approach can learn more accurate models, especially after fine tuning on GS.