Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing
This addresses evaluation challenges for speech and text constituency parsing, particularly in handling token mismatches, but is incremental as it builds on existing alignment and IOU concepts.
The authors tackled the problem of evaluating constituency parsing, especially for speech where word boundaries may mismatch, by introducing STRUCT-IOU, a metric that aligns trees and calculates average intersection-over-union, showing higher tolerance to plausible parses than PARSEVAL.
We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).