AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing
  Evaluation Suite
Jonas Groschwitz Shay B. Cohen Lucia Donatelli Meaghan Fowlie
We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge
set for Abstract Meaning Representation (AMR) parsing with accompanying
evaluation metrics. AMR parsers now obtain high scores on the standard AMR
evaluation metric Smatch, close to or even above reported inter-annotator
agreement. But that does not mean that AMR parsing is solved; in fact, human
evaluation in previous work indicates that current parsers still quite
frequently make errors on node labels or graph structure that substantially
distort sentence meaning. Here, we provide an evaluation suite that tests AMR
parsers on a range of phenomena of practical, technical, and linguistic
interest. Our 36 categories range from seen and unseen labels, to structural
generalization, to coreference. GrAPES reveals in depth the abilities and
shortcomings of current AMR parsers.