---
abstract: 'Question Generation (QG) is fundamentally a simple syntactic transformation; however, many aspects of semantics influence what questions are good to form. We implement this observation by developing Syn-QG, a set of transparent syntactic rules leveraging universal dependencies, shallow semantic parsing, lexical resources, and custom rules which transform declarative sentences into question-answer pairs. We utilize PropBank argument descriptions and VerbNet state predicates to incorporate shallow semantic content, which helps generate questions of a descriptive nature and produce inferential and semantically richer questions than existing systems. In order to improve syntactic fluency and eliminate grammatically incorrect questions, we employ back-translation over the output of these syntactic rules. A set of crowd-sourced evaluations shows that our system can generate a larger number of highly grammatical and relevant questions than previous QG systems and that back-translation drastically improves grammaticality at a slight cost of generating irrelevant questions.'
author:
- |
    Kaustubh D. Dhole\
    Amelia Science\
    RnD, IPsoft\
    New York, NY 10004\
    `kdhole@ipsoft.com`\
    Christopher D. Manning\
    Department of Computer Science\
    Stanford University\
    Stanford, CA 94305\
    `manning@stanford.edu`\
bibliography:
- 'anthology.bib'
- 'acl2020.bib'
title: 'Syn-QG: Syntactic and Shallow Semantic Rules for Question Generation'
---

Introduction
============

Automatic Question Generation (QG) is the task of generating question-answer pairs from a declarative sentence. It has direct use in education and generating engagement, where a system automatically generates questions about passages that someone has read. A more recent secondary use is for automatic generation of questions as a data augmentation approach for training Question Answering (QA) systems. QG was initially approached by syntactic rules for question-generation, followed by some form of statistical ranking of goodness, e.g.,  [@heilmanquestion; @heilman2010good]. In recent years, as in most areas of NLP, the dominant approach has been neural network generation [@du2017learning], in particular using a sequence-to-sequence architecture, which exploits the data in the rapidly growing number of large QA data sets.

![ The SRL structure is leveraged to invoke a template, and a simple rearrangement of the modifying arguments is performed.[]{data-label="fig:srl_qgen"}](QG_asT3.png){width="\linewidth"}

Previous rule-based approaches suffer from a significant lack of variety in the questions they generate, sticking to a few simple and reliable syntactic transformation patterns. Neural architectures provide a pathway to solving this limitation since they can exploit QA datasets to learn the broad array of human question types, providing the usual neural network advantages of a data-exploiting, end-to-end trainable architecture. Nevertheless, we observe that the quality of current neural QG systems is still lacking: The generated questions lack syntactic fluency, and the models lack transparency and an easy way to improve them.

We argue that in essence QG can be governed by simple syntactic “question transformations” – while the implementation details vary, this is in accord with all major linguistic viewpoints, such as Construction Grammar and Chomskyan Generative Grammar, which emphasize grammatical rules and the existence of finite ways to create novel utterances. However, successful, fluent question generation requires more than just understanding syntactic question transformations, since felicitous questions must also observe various semantic and pragmatic constraints. We approach these by making use of semantic role labelers (SRL), previously unexploited linguistic semantic resources like VerbNet’s predicates (Figure \[fig:vn4\]) and PropBank’s rolesets and custom rules like implications, allowing us to generate a broader range of questions of a descriptive and inferential nature. A simple transformation commonly used in rule-based QG is also displayed in Figure \[fig:srl\_qgen\].

We evaluate our QG framework, Syn-QG against three QG systems on a mixture of Wikipedia and commercial text sentences outperforming existing approaches in grammaticality and relevance in a crowd-sourced human evaluation while simultaneously generating more types of questions. We also notice that back-translated questions are grammatically superior but are sometimes slightly irrelevant as compared to their original counterparts.

Related Work
============

With the advent of large-scale QA datasets [@rajpurkar2016squad; @nguyen2016ms], recent work in QG [@du2017learning; @zhou2017neural] has primarily focused on training sequence-to-sequence and attention-based architectures. @dong2019unified fine-tuned the question generation task by taking advantage of a large pre-trained language model. Success in reinforcement learning has inspired teacher-student frameworks [@wang2017joint; @tang2017question] treating QA and QG as complementary tasks and performing joint training by using results from QA as rewards for the QG task. @yuan-etal-2017-machine [@Hosking_2019; @zhang2019addressing] used evaluation metrics like BLEU, sentence perplexity, and QA probability as rewards for dealing with exposure bias.

 @chen2019natural trained a reinforcement learning based graph-to-sequence architecture by embedding the passage via a novel gated bi-directional graph neural network and generating the question via a recurrent neural network. To estimate the positions of copied words, @liu2019learning used a graph convolution network and convolved over the nodes of the dependency parse of the passage. @li2019improving jointly modeled OpenIE relations along with the passage using a gated-attention mechanism and a dual copy mechanism.

Traditionally, question generation has been tackled by numerous rule-based approaches [@heilmanquestion; @mostow2009generating; @yao2010question; @lindberg2013generating; @labutov2015deep]. @heilmanquestion [@heilman2010good] introduced an overgenerate-and-rank approach that generated multiple questions via rule-based tree transformations of the constituency parse of a declarative sentence and then ranked them using a logistic-regression ranker with manually designed features. @yao2010question described transformations of Minimal Recursion Semantics representations guaranteeing grammaticality. Other transformations have been in the past defined in terms of templates [@mazidi2014linguistic; @mazidi2015leveraging; @mazidi2016infusing; @flor2018semantic], or explicitly performed [@heilmanquestion] by searching tree patterns via Tregex, followed by their manipulation using Tsurgeon [@levy2006tregex]. @kurdi2020systematic provide a comprehensive summary of QG, analysing and comparing approaches before and after 2014.

Vis-à-vis current neural question generators, rule-based architectures are highly transparent, easily extensible, and generate well-formed questions since they perform clearly defined syntactic transformations like subject-auxiliary inversion and *WH-movement* over parse structures whilst leveraging fundamental NLP annotations like named entities, co-reference, temporal entities, etc.

However, most of the existing rule-based systems have lacked diversity, being mostly focused on generating *What*-type and boolean questions and have mainly exploited parse structures which are not semantically informed. @mazidi2016infusing [@flor2018semantic] use Dependency, SRL, and NER templates but do not handle modalities and negation in a robust manner. Moreover, there is plenty of availability of core linguistic resources like VerbNet and PropBank, which provide further unique ways to look at sentences and ask questions differently besides the generally well-established dependency and SRL parses.

Syn-QG
======

Syn-QG is a rule-based framework which generates questions by identifying potential short answers in 1) the nodes of crucial dependency relations 2) the modifying arguments of each predicate in the form of semantic roles 3) named entities and other generic entities 4) the states of VerbNet’s thematic roles in the form of semantic predicates and 5) PropBank roleset specific natural language descriptions. Each of the five heuristics works independently, generating a combined set of question-answer pairs, which are eventually back-translated. We describe each of these five sources.

Dependency Heuristics
---------------------

Dependency trees are syntactic tree structures, wherein syntactic units in the form of words are connected via directed links. The finite verb is considered as the structural root of the tree, and all other syntactic units are either directly (*nsubj, dobj,*, etc.) or indirectly (*xcomp, iobj,* etc.) dependent on this finite verb.

We present rules over such dependency trees annotated according to the Universal Dependencies (UD) format [@de-marneffe-etal-2014-universal]. To extract dependency structures, we use the parser of @gardner2018allennlp.

We make use of PropBanks predicate-argument structure (SRL) for clausal extraction of the verb headed by a select few dependency nodes which can serve as answers. These rules treat the clause as a combination of a subject, an object, the head verb and other non-core arguments. The clause is further refined with modals, auxiliaries and negations if found around the verb. Finally, we make use of a set of predefined handwritten templates, a few of which are described in Table \[dep-table\].

[| M[2cm]{} | L[3.1cm]{} | C[5cm]{} | R[4.2cm]{} |]{} **Potential Short Answer** **(Dependencies)** & **Question Template** & **Sample Fact** & **Generated Question**\
& & &\
subject ([[ ]{}]{}) & **Wh** mainAux otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}? & [[ ]{}]{} accepted captaincy during Australia’s golden era. & **Who** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}?\
& & &\
direct object([[ ]{}]{}) & **Wh** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{}? & In monsoon, India receives [[ ]{}]{}. & **What** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}?\
& & &\
open clausal complement ([[ ]{}]{}) & **Wh** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}? & The Sheriff did not try [[ ]{}]{} while the outlaws were fasting. & **What** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}?\
& & &\
copula ([[ ]{}]{}) & **How would you describe** [[ ]{}]{}? & Comets are [[ ]{}]{}. & **How would you describe** [[ ]{}]{} ?\

In each of the templates, we convert *What* to *Who/Whom*, *When* or *Where* depending on the named entity of the potential answer and *do* to *does* or *did* according to the tense and number of the subject to ensure subject-verb agreement. The pseudo code is described in Algorithm \[alg:the\_alg\] of the Appendix.

SRL Heuristics
--------------

While dependency representations are perhaps the most popular syntactic method for automatically extracting relationships between words, they lack sufficient semantic detail. Being able to answer *“Who* did *what* to *whom* and *how, why, when* and *where”* has been a central focus in understanding language. In recent decades, shallow semantic parsing has been a prominent choice in understanding these relationships and has been extensively used in question generation [@mazidi2016infusing; @flor2018semantic].

PropBank-style frames provide semantically motivated roles that arguments around a verb play. Moreover, highly accurate semantic role labeling models are being developed owing to corpora like PropBank and FrameNet. We take advantage of the SRL model of @gardner2018allennlp for extracting the roles of each verb in the sentence.

$\{SRL_1 \ldots SRL_s \} \gets SRL(w_0 \ldots w_n)$ *loop j = 0, until j = s*: $\{A_0 \ldots A_{CAU}, A_{TMP}\} \gets SRL_j$ *$loop \text{ } A_x \in SRL_j \; if \; A_x = modifier$*: $subj \gets A_0$ $A_{x}^{-} \gets \sum (A_3,A_4,... A_{TMP} - A_x)$ $verb \gets \{A_v, modals, negation\}$ $template \gets modifier_{type} \gets A_x$ $QA \gets template(subj, A_x, verb, A_{x}^{-}) $ **close**;

We succinctly describe the steps taken in Algorithm \[srl-algo\]. We first filter out all the predicates which have an *Agent* or a *Patient* and at least one other modifier like *Extent, Manner, Direction,* etc. These modifiers would serve as our short answers. We make use of a set of predefined handwritten templates described in Table \[srl-table\], which rearrange the arguments within the fact to convert it into an interrogative statement depending on the modifier.

In Figure \[fig:srl\_qgen\], the predicate “won” is modified by a *Patient* “New Mexico”, an *Agent* “Obama”, an *Extent* modifier “by a margin of 5%” and a *Temporal* modifier “in 2008”. For *Extent* as a short answer, we fill a pre-defined template “By how much mainAux nsubj otherAux verb obj modifiers ?” to get the above question-answer pair. We keep the order of arguments as they appear in the original sentence. The templates are described in Table \[srl-table\].

Named Entities, Custom Entities, and Hypernyms
----------------------------------------------

We create separate templates when any numbered SRL argument contains common named entities like *Person*, *Location*, *Organization* etc. Like @flor2018semantic, we add specific rules in the form of regexes to address special cases to differentiate between phrases like ***For how long*** and ***Till when*** instead of a generic ***When*** question type. Some of the templates are described in Table \[ner-table\] in the Appendix. The approach is described in Algorithm \[ner-algo\] in the Appendix.

[| M[2.3cm]{} | L[3cm]{} | C[5cm]{} | R[4cm]{} |]{} **Potential Short Answer** **(Verb Arguments)** & **Question Template** & **Sample Fact** & **Generated Question**\
& & &\
Locative ([[ ]{}]{}) & **Where** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? & Americans eat about 100 acres of pizza each day, with about 3 billion pizzas sold annually [[ ]{}]{} & **Where** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ?\
& & &\
Manner ([[ ]{}]{}) & **How** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? & Young Sheldon was caught unaware as the liquid was oozing out of the chamber [[ ]{}]{} & **How** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}?\
Purpose ([[ ]{}]{} and [[ ]{}]{}) & **For what purpose** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? & Collectively, South African women and children walk a daily distance equivalent to 16 trips to the moon and back [[ ]{}]{} &

**For what purpose** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ?\
Cause ([[ ]{}]{}) & **Why** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? & [[ ]{}]{} you can save up to four gallons of water every morning by turning off the tap while you brush your teeth. & **Why** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ?\
& & &\
Temporal ([[ ]{}]{}) & **When** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? **Till when** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}? & Stephen Hawking [[ ]{}]{} threw a party for time-travelers but he announced the party [[ ]{}]{} Princess Sita travelled the whole town [[ ]{}]{} & **When** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? **When** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? **Till when** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} [[ ]{}]{}?\
& & &\
Extent ([[ ]{}]{}) & **By how much** [[ ]{}]{} [[ ]{}]{} otherAux [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} ? & New Mexico was won by Obama [[ ]{}by a margin of 5%]{} in 2008. & **By how much** [[ ]{}]{} [[ ]{}]{} [[ ]{}]{} by [[ ]{}]{} [[ ]{}]{}?\

We also use WordNet [@miller1998wordnet] hypernyms of all potential short answers and replace ***What*** with the bigram ***Which hypernym***. So, for a sentence like “Hermione plays badminton at the venue”, we generate a question “***Which sport*** does Hermione play at the venue?”. For computing the hypernym, we use the sense disambiguation implementation of @pywsd14. While supersenses do display a richer lexical variety, sense definitions don’t always fit well.

Handling modals and auxilliaries
--------------------------------

During explicit inversion of the verb and arguments around it via our templates, we tried to ensure that the positions of auxiliaries are set, and negations are correctly treated. We define a few simple rules to ensure that.

-   When there are multiple auxiliaries, we only invert the first auxiliary while the second and further auxiliaries remain as they are just before the main verb.

-   We make the question auxiliary finite and agree with the subject.

-   We ensure that the object is kept immediately after the verb.

-   For passive cases, *subj-verb-obj* is changed to *obj-verb-by-subj*.

Handling Factualness via Implicature
------------------------------------

Previous rule-based approaches  [@mazidi2016infusing; @flor2018semantic] have used the NEG dependency label to identify polarity. But such an approach would suffer whenever polarities would be hierarchically entailed from their parent clauses in cases like “Picard did not fail to X” where the entailed polarity of “X” is, in fact, positive. Moreover, in one-way implications like “Bojack hesitated to X”, it would be best not to generate a question for unsure cases since it is open-ended if Bojack did or did not X. A similar example is displayed in Figure \[fig:PB3\]. For each verb representing a subordinate clause, we compute its entailed truth or falsity from its parent clause using the set of one-way and two-way implicative verbs, and verb-noun collocations provided by @karttunen2012simple. For example, the two-way implicative construction “forget to X” entails that “X” did not happen, so it would be wrong to ask questions about “X”. @karttunen2012simple provides simple implications in the form of 92 verbs and phrasal implications in the form of 9 sets of verbs and 8 sets of nouns making 1002 verb-noun collocations. The entailed polarity of a clause can be either TRUE, FALSE, or UNSURE[^1]. For FALSE clauses, we only generate a boolean question with a NO answer. For UNSURE clauses, we do not generate any question. For TRUE clauses and verbs and collocations not present in the above set, we rely on the NEG label.

VerbNet Predicate Templates
---------------------------

While SRL’s event-based representations have permitted us to generate questions that talk about the roles participants of an event play, we exploit VerbNet’s sub-event representation to ask questions on how participants’ states change across the time frame of the event. In Figure \[fig:vn4\], the event murder (VerbNet class [*murder-42.1*](https://uvi.colorado.edu/verbnet/murder-42.1)) results in a final state in which the participant *Julius Caesar* is in a *not-alive* state.

Each class in VerbNet [@schuler2005verbnet; @brown2019verbnet] includes a set of member verbs, the thematic roles used in the predicate-argument structure, accompanied with flat syntactic patterns and their corresponding semantic predicates represented in neo-Davidsonian first-order-logic formulation. These semantic predicates bring forth a temporal sequencing of sub-events tracking how participants’ states change over the course of the event. The advantage is to be able to ask questions bearing a surface form different from the source sentence but which are driven by reasoning rather than just being paraphrastic. For example, in the sentence, “Brutus murdered Julius Caesar”, the event [*murder-42.1*](https://uvi.colorado.edu/verbnet/murder-42.1) entails a final state of “death” or the *Patient* participant not being alive at the end of the event. So, we construct a template “mainAux the Patient otherAux not alive?”. Similarly, the event [*pay-68-1*](https://uvi.colorado.edu/verbnet/pay-68-1) results in a final state in which the *Recipient* “Perry” has possession of “\$100” and the *Agent* “John” has possession of “the car”, against which we define the templates as shown in Figure \[fig:vn1\].

We formulate two sets of questions: boolean type and which-type questions asking specifically about these states. We create templates for VerbNet’s stateful predicates like **has\_location, has\_possession, has\_information, seem, has\_state, cost, desire, harmed, has\_organization\_role, together, social\_interaction, authority\_relationship,** etc. which are present in 64.4% of the member verbs in VerbNet[^2]. We outline a few of the templates in Table \[vn-table\].

During inference time, we first compute the VerbNet sense, the associated thematic role mapping, and syntactic frame (along with the predicates) with the help of @brown2019verbnet’s parser. VerbNet’s predicates are governed by the sub-events in which they occur. Although VerbNet’s representation lays out a sequence of sub-events, no sub-event is explicitly mentioned as the final one[^3]. We choose all the predicates of those sub-events which are preceded by other sub-events which possess at least one process-oriented predicate.[^4]

[| M[2.8cm]{} | L[3cm]{} | C[5.5cm]{} | R[3cm]{} |]{} **Triggering Predicate and Thematic Arguments** & **Question Template** & **Sample Fact & VerbNet Predicate** & **Generated Question**\
& & &\
**HAS\_POSSESSION** (Asset,Recipient) & **Who** has Asset ? Recipient & Robert paid [[ ]{}]{} to [[ ]{}]{} for the cycle. **HAS\_POSSESSION**([[ ]{}]{},[[ ]{}]{}) & **Who** has [[ ]{}]{} ? [[ ]{}]{}\
& & &\
**HARMED** (Patient) & **What** is harmed ? Patient & The terrorists bombed [[ ]{}]{}. **HARMED**([[ ]{}]{}) & **What** is harmed ? [[ ]{}]{}\
& & &\
**NOT\_ALIVE** (Patient) & Is Patient alive ? No. & According to epics, Vishnu killed [[ ]{}]{}. **NOT\_ALIVE** ([[ ]{}]{}) & Is [[ ]{}]{} alive ? No.\

PropBank Argument Descriptions
------------------------------

PropBank rolesets’ course-grained annotation of verb-specific argument definitions (“killer”, “payer”, etc.) to represent semantic roles offers robustly specific natural language descriptions to ask questions about the exact roles participants play. Nonetheless, not all descriptions are suitable to be utilized directly in rigid templates. So, we incorporate back-translation to 1) get rid of grammatical errors propagated from incorrect parsing and template restrictions, and 2) eliminate rarely used Prop-Bank descriptions and generate highly probable questions.

While previous work in rule-based QG has used SRL templates and WordNet senses to describe the roles arguments around a verb play, previous SRL templates have always been verb-agnostic, and we believe there is a great deal of potential in PropBank descriptions. Moreover, WordNet supersenses do not always give rise to acceptable questions. On manual evaluation, question relevance decreased after incorporating templates with WordNet supersenses. Instead, we make use of PropBank’s verb-specific natural language argument descriptions to create an additional set of templates. VerbNet senses have a one-to-one mapping with PropBank rolesets via the SemLink project [@palmer2009semlink]. We hence make use of  @brown2019verbnet’s parser to find the appropriate PropBank roleset for a sentence.

However, we observed that a lot of PropBank descriptions were noisy and made use of phrases which would be unarguably rare in ordinary parlance like “breather” or “truster”. To eliminate such descriptions, we computed the mean Google N-gram probabilities [@lin2012syntactic] of all the PropBank phrases in the timespan of the last 100 years and kept only those phrases which ranked in the top 50%.

Back-Translation
----------------

Back-translation has been used quite often in grammatical error correction [@xie2018noising] and is well known to translate noisy and ungrammatical sentences to their cleaner high probability counterparts. We exploit this observation to clean questions with noisy and inconsistent PropBank descriptions like “wanter” (Figure \[fig:PB3\]). We use two state-of-the-art (SOTA) pre-trained transformer models `transformer.wmt19.en-de` and `transformer.wmt19.de-en` from @ott2019fairseq trained on the English-German and German-English translation tasks of WMT 2019.

Figure \[fig:synqg\_all\] in the Appendix shows the output of all the five sets of templates applied together over one sentence (along-with implicature).

Evaluation and Results
======================

[| M[8.7cm]{} | L[1.4cm]{} | C[1.4cm]{} | R[1.4cm]{} | R[1.4cm]{} |]{} **Architecture** & BLEU-1 & BLEU-2 & BLEU-3 & BLEU-4\
PCFG-Trans [@heilman2010good] & 28.77 & 17.81 & 12.64 & 9.47\
SeqCopyNet [@zhou2018sequential] & & & & 13.02\
NQG++ [@zhou2017neural] & 42.36 & 26.33 & 18.46 & 13.51\
MPQG [@song2017unified] & & & & 13.91\
Answer-focused Position-aware model [@sun2018answer] & 43.02 & 28.14 & 20.51 & 15.64\
To the Point Context [@li2019improving] & 44.40 & 29.48 & 21.54 & 16.37\
s2sa-at-mp-gsa [@zhao2018paragraph] & 44.51 & 29.07 & 21.06 & 15.82\
ASs2s [@kim2019improving] & & & & 16.17\
CGC-QG [@liu2019learning] & 46.58 & 30.9 & 22.82 & 17.55\
Capturing Greater Context [@tuan2019capturing] & 46.60 & **31.94** & 23.44 & 17.76\
Natural QG with RL based Graph-to-Sequence [@chen2019natural] & - & - & - & 17.94\
RefineNet [@nema2019let] & **47.27** & 31.88 & 23.65 & 18.16\
QPP&QAP [@zhang2019addressing] & - & - & - & 18.37\
ACS-QG$^*$ [@liu2020asking] & **52.30**$^*$ & **36.70**$^*$ & **28.00**$^*$ & 22.05\
UNILM$^*$ [@wang2020minilm] & - & - & - & 24.32\
ERNIE-GEN$^*$ [@xiao2020ernie] & - & - & - & 25.57\
UNILMv2$^*$ [@bao2020unilmv2] & - & - & - & 26.30\
ProphetNet$^*$ [@yan2020prophetnet] & - & - & - & **26.72**$^*$\
Syn-QG & 45.55 & 30.24 & **23.84** & **18.72**\

[| M[2.6cm]{} | L[3cm]{} | C[4cm]{} | R[2.7cm]{} | P[2cm]{} |]{} System & \#Questions Generated & Avg. \#Questions Per Sentence & Grammaticality & Relevance\
H&S & 381 & 3.81 & 3.49 & 4.23\
NQG & 100 & 1 & 3.48 & 3.28\
QPP&QAP & — & — & 3.9 & 4.03\
Syn-QG & 654 & **6.54** & **3.93** & **4.34**\

[| M[4.1cm]{} | L[3.4cm]{} | C[3.4cm]{} | R[3.4cm]{} |]{} System & Avg. novel unigrams & Avg. novel bigrams & Avg. novel trigrams\
H&S & 23.6 & 40.64 & 52.22\
Syn-QG (w/o BT) & 26.8 & 43.93 & 53.4\
Syn-QG & 39.34 & 64.08 & 76.24\
SQUAD & 42.86 & 74.2 & 86.35\
Syn-QG (BT vs w/o-BT) & 28.78 & 55.18 & 67.81\

Most of the prior QG studies have evaluated the performance of the generated questions using automatic evaluation metrics used in the machine translation literature. We use the traditional BLEU scores [@papineni2002bleu] and compare the performance of Syn-QG on the SQuAD [@rajpurkar2016squad] test split created by @zhou2017neural. BLEU measures the average n-gram precision on a set of reference sentences. A question lexically and syntactically similar to a human question would score high on such n-gram metrics. Despite not utilizing any training data, Syn-QG performs better than the previous SOTA on two evaluation metrics BLEU-3 and BLEU-4 and close to SOTA on BLEU-1 and BLEU-2 (Table \[results-table\]) at the time of submission. The high scores obtained without conducting any training arguably shed a little light on the predictable nature of the SQuAD dataset too.

Besides SRL, Dependency, and NER templates, Syn-QG’s questions also arise from VerbNet’s predicates and PropBank’s descriptions, which indeed by nature describe events not mentioned explicitly within the fact. Like in Figure \[fig:vn1\], the sentence with the event “paid” results in a question with a stateful event of “cost”. Deducible questions like these have a good chance of having a distribution of n-grams quite different from the source sentences, possibly exposing the weakness of traditional n-gram metrics and rendering them less useful for a task like QG.

In order to have a complete and more reliable evaluation to gauge the system, we also carry out a human evaluation using two of the metrics used in QG-STEC Task B [@rus2012detailed], namely grammaticality, and relevance which we define below. We compared the questions generated from our system against the constituency-based H&S [@heilmanquestion], a neural system NQG [@du2017learning] which does not depend on a separate answer extractor and QPP&QAP[^5] [@zhang2019addressing] which has outperformed existing methods. We fed a total of 100 facts randomly picked from Wikipedia and 5 commercial domains (IT, Healthcare, Sports, Banking and Politics) combined, to each of the four systems. We then conducted a crowd-sourced evaluation over Amazon Mechanical Turk for the generated questions.

-   **Grammatical Correctness**: Raters had to rate a question on how grammatically correct it is or how syntactically fluent it is, disregarding its underlying meaning.

-   **Relevance Score**: Raters had to give a score on how relevant the generated question is to the given fact. The relevance score helps us gauge whether the question should have been generated or not irrespective of its grammaticality.[^6]

Each question was evaluated by three people scoring grammaticality and relevance on a 5 point Likert scale. The inter-rater agreement (Krippendorff’s co-efficient) among human evaluations was 0.72. The instructions given to the Mturk raters are provided in the Appendix Figure \[fig:mturk\]. The results of the evaluation are shown in Table \[res-table\]. Syn-QG generates a larger number of questions than H&S and performs strongly on grammaticality ratings. Syn-QG is also able to generate highly relevant questions without the use of a ranker. Also, rule-based approaches seem to be much better at generating relevant questions than neural ones.

QG-STEC also used variety and question types as their evaluation criteria and rewarded systems to generate questions meeting a range of specific question types. Syn-QG’s questions cover each of those question types.

Since many times, despite the ability to paraphrase (Table \[vocab-dist-table\]), back-translated outputs tend to change the meaning of the original sentence, we also measured back-translation’s impact on the above QG metrics. We considered questions generated from 50 facts of Wikipedia measuring the grammaticality and relevance before and after back-translation. While grammaticality increased from 3.54 to 4.11, question relevance fell a bit from 3.96 to 3.88. This observation, along with the performance of QPP&QAP shown in Table \[results-table\], accentuates that while neural models are learning syntactic structures well, there is still some progress to be made to generate relevant questions.

Discussion
==========

We introduced Syn-QG, a set of broad coverage rules leveraging event-based and sub-event based sentence views along with verb-specific argument descriptions. Automatic and manual evaluations show that Syn-QG is able to generate a large number of diverse and highly relevant questions with better fluency. Verb-focused rules help approach long-distance dependencies and reduce the need for explicit sentence simplification by breaking down a sentence into clauses while custom rules like implications serve a purpose similar to a re-ranker to discard irrelevant questions but with increased determinism. While our work focuses on sentence-level QG, it would be interesting to see how questions generated from VerbNet predicates would have an impact on multi-sentence or passage level QG, where the verb-agnostic states of the participants would change as a function of multiple verbs. The larger goal of QG is currently far from being solved. Understanding abstract representations, leveraging world knowledge, and reasoning about them is crucial. However, we believe that with an extensible and transparent architecture, it is very much possible to keep improving the system continuously in order to achieve this larger goal.

Acknowledgments {#acknowledgments .unnumbered}
===============

We thank the three anonymous reviewers for their helpful comments and invaluable suggestions. We also thank the members of Amelia Science, RnD IPsoft, India - Manjunath Hegde, Anant Khandelwal, Ashish Shrivastava for their work in QG and especially Viswa Teja Ravi, for helping in replicating  @mazidi2016infusing’s work. We also thank Uday Chinta and IPsoft, India, for supporting and providing access to Amazon Mechanical Turk.

Appendices {#sec:appendix}
==========

$\{d_0 \ldots d_n \} \gets dependency(w_0 \ldots w_n)$ *loop i = 0, until i = n*: $d_v \gets parent(d_i) $ $\{A_0 \ldots A_{CAU}\} \gets SRL(d_v)$ $subj \gets A_0$ $obj \gets A_1$ $obj \gets A_2$ $A_{x} \gets \sum (A_3,A_4,... A_{TMP})$ $verb \gets \{d_v, modals, negation\}$ $template \gets dep_{type} \gets d_i$ $QA \gets template(subj, obj, verb, A_{x}) $ **close**;

$\{SRL_1 \ldots SRL_s \} \gets SRL(w_0 \ldots w_n)$ *loop j = 0, until j = s*: $\{A_0 \ldots A_{CAU}, A_{TMP}\} \gets SRL_j$ *$loop \text{ } A_x \in SRL_j \; if \; A_x\;contains\;a\;NE$*: $subj \gets A_0$ $A_{x}^{-} \gets \sum (A_3,A_4,... A_{TMP} - A_x)$ $verb \gets \{A_v, modals, negation\}$ $template \gets NE_{type} \gets A_x$ $QA \gets template(subj, A_x, verb, A_{x}^{-}) $ **close**;

[| M[2cm]{} | L[3cm]{} | C[5.7cm]{} | R[3.6cm]{} |]{} **Potential Short Answer** **(Named Entities)** & **Question Template** & **Sample Fact** & **Generated Question**\
Location & **Where** mainAux subj otherAux verb obj modifiers ? & [[ ]{}]{} [[ ]{}]{} & **Where** was [[ ]{}]{}?\
Person & **Who** mainAux subj otherAux verb obj modifiers ? **Whom** mainAux obj otherAux verb modifiers & [[ ]{}]{}[[ ]{}]{} & **Whom** did [[ ]{}]{}?\
Date & **When** mainAux subj otherAux verb obj modifiers ? & [[ ]{}]{} [[ ]{}]{} & **When** did [[ ]{}]{}?\
Number & **How many** mainAux subj otherAux verb obj modifiers? & [[ ]{}]{} [[ ]{}]{} & **How many** [[ ]{}]{}?\
Phone Number & **At what number** mainAux subj otherAux verb obj modifiers ? & [[ ]{}]{} [[ ]{}]{} & **At what phone number** [[ ]{}]{}?\
Duration & **For how long** mainAux subj otherAux verb obj modifiers? & [[ ]{}]{} [[ ]{}]{} & **For how long** [[ ]{}]{}?\
Organization & **Which organization** mainAux subj otherAux verb obj modifiers? & [[ ]{}]{} [[ ]{}]{} & **Which organization** [[ ]{}]{}?\

![image](SynQG_All2.png){width="\textwidth" height="\textheight"}

![image](vn_ui_img.PNG){width="\textwidth" height="\textheight"}

[^1]: Unsure clauses appear in one-way implicatives when it’s unclear if the clause is true or false under either an affirmative or a negative parent clause.

[^2]: Out of 4854 member verbs, there are 3128 members whose syntactic frame contains at least one of these predicates.

[^3]: or a sub-event, which is an outcome of a process

[^4]: Out of 174 VerbNet predicates, we manually categorize 84 predicates like HAS\_LOCATION, HAS\_POSSESSION as stateful predicates and the remaining ones like DESCRIBE, TRANSFER, etc. as process-oriented predicates.

[^5]: Since the QPP&QAP model does not have a separate answer extractor, we use the answer spans computed from Syn-QG (412 in total after discarding overlaps).

[^6]: In cases when the grammaticality is extremely low like 1 or 2, the relevance score will also tend to be low. Otherwise, we assume that minor grammatical variations can be ignored while gauging relevance.
