Structured Tree Alignment for Evaluation of (Speech) Constituency
  Parsing
Freda Shi Kevin Gimpel Karen Livescu
We present the structured average intersection-over-union ratio (STRUCT-IOU),
a similarity metric between constituency parse trees motivated by the problem
of evaluating speech parsers. STRUCT-IOU enables comparison between a
constituency parse tree (over automatically recognized spoken word boundaries)
with the ground-truth parse (over written words). To compute the metric, we
project the ground-truth parse tree to the speech domain by forced alignment,
align the projected ground-truth constituents with the predicted ones under
certain structured constraints, and calculate the average IOU score across all
aligned constituent pairs. STRUCT-IOU takes word boundaries into account and
overcomes the challenge that the predicted words and ground truth may not have
perfect one-to-one correspondence. Extending to the evaluation of text
constituency parsing, we demonstrate that STRUCT-IOU shows higher tolerance to
syntactically plausible parses than PARSEVAL (Black et al., 1991).