SMATCH++: Standardized and Extended Evaluation of Semantic Graphs
Juri Opitz
The Smatch metric is a popular method for evaluating graph distances, as is
necessary, for instance, to assess the performance of semantic graph parsing
systems. However, we observe some issues in the metric that jeopardize
meaningful evaluation. E.g., opaque pre-processing choices can affect results,
and current graph-alignment solvers do not provide us with upper-bounds.
Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore,
adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity)
are spread out, and lack a unifying framework.
  For better inspection, we divide the metric into three modules:
pre-processing, alignment, and scoring. Examining each module, we specify its
goals and diagnose potential issues, for which we discuss and test mitigation
strategies. For pre-processing, we show how to fully conform to annotation
guidelines that allow structurally deviating but valid graphs. For safer and
enhanced alignment, we show the feasibility of optimal alignment in a standard
evaluation setup, and develop a lossless graph compression method that shrinks
the search space and significantly increases efficiency. For improved scoring,
we propose standardized and extended metric calculation of fine-grained
sub-graph meaning aspects. Our code is available at
https://github.com/flipz357/smatchpp