Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser
This work addresses the need for more accurate and efficient dependency parsers in natural language processing, with incremental improvements over existing methods.
The paper tackled the problem of improving dependency parsing accuracy by introducing a consensus parser from an ensemble of greedy parsers and a distilled single model, achieving state-of-the-art results on English, Chinese, and German datasets.
We introduce two first-order graph-based dependency parsers achieving a new state of the art. The first is a consensus parser built from an ensemble of independently trained greedy LSTM transition-based parsers with different random initializations. We cast this approach as minimum Bayes risk decoding (under the Hamming cost) and argue that weaker consensus within the ensemble is a useful signal of difficulty or ambiguity. The second parser is a "distillation" of the ensemble into a single model. We train the distillation parser using a structured hinge loss objective with a novel cost that incorporates ensemble uncertainty estimates for each possible attachment, thereby avoiding the intractable cross-entropy computations required by applying standard distillation objectives to problems with structured outputs. The first-order distillation parser matches or surpasses the state of the art on English, Chinese, and German.