Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
This addresses the challenge of leveraging multiple syntactic representations in NLP, though it appears incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of integrating heterogeneous syntactic tree structures for NLP tasks by using knowledge distillation to combine them into a unified LSTM encoder, resulting in improved performance over tree encoders and ensemble methods on four syntax-dependent tasks.
Syntax has been shown useful for various NLP tasks, while existing work mostly encodes singleton syntactic tree using one hierarchical neural network. In this paper, we investigate a simple and effective method, Knowledge Distillation, to integrate heterogeneous structure knowledge into a unified sequential LSTM encoder. Experimental results on four typical syntax-dependent tasks show that our method outperforms tree encoders by effectively integrating rich heterogeneous structure syntax, meanwhile reducing error propagation, and also outperforms ensemble methods, in terms of both the efficiency and accuracy.