The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony
  and Sarcasm Generation
Tyler Loakman Aaron Maladry Chenghua Lin
Human evaluation is often considered to be the gold standard method of
evaluating a Natural Language Generation system. However, whilst its importance
is accepted by the community at large, the quality of its execution is often
brought into question. In this position paper, we argue that the generation of
more esoteric forms of language - humour, irony and sarcasm - constitutes a
subdomain where the characteristics of selected evaluator panels are of utmost
importance, and every effort should be made to report demographic
characteristics wherever possible, in the interest of transparency and
replicability. We support these claims with an overview of each language form
and an analysis of examples in terms of how their interpretation is affected by
different participant variables. We additionally perform a critical survey of
recent works in NLG to assess how well evaluation procedures are reported in
this subdomain, and note a severe lack of open reporting of evaluator
demographic information, and a significant reliance on crowdsourcing platforms
for recruitment.