Pento-DIARef: A Diagnostic Dataset for Learning the Incremental
  Algorithm for Referring Expression Generation from Examples
Philipp Sadler David Schlangen
NLP tasks are typically defined extensionally through datasets containing
example instantiations (e.g., pairs of image i and text t), but motivated
intensionally through capabilities invoked in verbal descriptions of the task
(e.g., "t is a description of i, for which the content of i needs to be
recognised and understood"). We present Pento-DIARef, a diagnostic dataset in a
visual domain of puzzle pieces where referring expressions are generated by a
well-known symbolic algorithm (the "Incremental Algorithm"), which itself is
motivated by appeal to a hypothesised capability (eliminating distractors
through application of Gricean maxims). Our question then is whether the
extensional description (the dataset) is sufficient for a neural model to pick
up the underlying regularity and exhibit this capability given the simple task
definition of producing expressions from visual inputs. We find that a model
supported by a vision detection step and a targeted data generation scheme
achieves an almost perfect BLEU@1 score and sentence accuracy, whereas simpler
baselines do not.