---
abstract: 'Textual distractors in current multi-choice VQA datasets are not challenging enough for state-of-the-art neural models. To better assess whether well-trained VQA models are vulnerable to potential attack such as more challenging distractors, we introduce a novel task called *textual Distractors Generation for VQA* (DG-VQA). The goal of DG-VQA is to generate the most confusing distractors in multi-choice VQA tasks represented as a tuple of image, question, and the correct answer. Consequently, such distractors expose the vulnerability of neural models. We show that distractor generation can be formulated as a Markov Decision Process, and present a reinforcement learning solution to unsupervised produce distractors. Our solution addresses the lack of large annotated corpus issue in classical distractor generation methods. Our proposed model receives reward signals from well-trained multi-choice VQA models and updates its parameters via policy gradient. The empirical results show that the generated textual distractors can successfully confuse several cutting-edge models with an average $20\%$ accuracy drop from around $64\%$. Furthermore, we conduct extra adversarial training to improve the robustness of VQA models by incorporating the generated distractors. The experiment validates the effectiveness of adversarial training by showing a performance improvement of $27\%$ for the multi-choice VQA task. [^1]'
author:
- |
    Jiaying Lu^1^, Xin Ye^1^, Yi Ren^2^, Yezhou Yang^1^\
    ^1^School of Computing, Informatics, and Decision Systems Engineering\
    ^2^School for the Engineering of Matter, Transport and Energy\
    Arizona State University\
    {jiaying4, xinye1, yiren, yz.yang}@asu.edu\
bibliography:
- 'aaai.bib'
title: 'Good, Better, Best: Multi-Choice VQA with Textual Distractors Generation via Policy Gradient'
---

Introduction {#sec:intro}
============

![An example of DG-VQA task. The well-trained VQA model predicts the right answer choice for the input image and question (\[fig:example\_DGVQA\]a and \[fig:example\_DGVQA\]b). However, it is easy to distinguish correct choice from distractor choices. The model will be fooled when encountering generated distractors (\[fig:example\_DGVQA\]c). []{data-label="fig:example_DGVQA"}](images/Example_DGVQA.pdf){width="0.9\linewidth"}

With advances in deep learning technologies, research in multi-modal learning tasks such as image captioning [@yang2011corpus; @vinyals2015show], text to image synthesis [@reed2016generative], visual question answering [@AntolVQA] that combines natural language processing and computer vision has dramatically increased. Among them, Visual Question Answering (VQA) is considered as a compelling “AI-complete” task. The visual multiple-choice question (visual MCQs) is one type of VQA, which takes an image, a free-form natural language question, and several (typically four) natural language answer choices as input and choose one choice as the output. Recently, the artificial intelligence community has achieved remarkable progress to bridge the gap between human performance (96.6%, [@zhu2016visual7w]) and state-of-the-art neural network model (64.8%, [@jabri2016revisiting]) on this task. However, it has been pointed out that the distracting choices are too simple or biased [@jabri2016revisiting] in the current benchmark datasets, which raises doubt about the trained models’ true discriminative ability (see Figure \[fig:example\_DGVQA\]).

To facilitate visual MCQs better to serve as a “visual Turing test” [@turing2009computing; @geman2015visual], we introduce a novel task as generating challenging distractors, dubbed as DG-VQA: *textual Distractor Generation for VQA*. Given an image-question-answer triplet, the system’s task is to generate plausible distracting choices. The generated distractors should be semantically consistent with the multi-modality context, and be difficult to discriminate from the correct answer even for human beings. Generating such distractors provides a tool for researchers to further figure out whether well-trained VQA models are vulnerable to potential attacks and determine whether they are ready for real-world deployment. Moreover, MCQs are widely used in the education area. And distractor generation is a crucial and time-consuming procedure. There are some previous works [@liang2018distractor; @gao2018generating] focusing on automatic distractor generation (DG) to alleviate instructors’ workload. But none of them involve in visual question domain considering there is a trend to apply multimodal materials on student learning. Last but not least, MCQ is still a significant task even though the community has largely stopped working on this direction. DG-VQA is more cognitively challenging compared to open-ended VQA. We regard DG-VQA as a “teachers’ task” while VQA is a “students’ task”, since teachers are responsible for designing test questions to better assess students’ knowledge and reasoning skills. It showcases DG-VQA requires the counterfactual thinking ability that generated distractors should be seemingly correct alternatives (but wrong indeed).

Many previous works [@gronlund1982constructing; @chen2006fast; @gao2018generating] highly rely on manual evaluation, which is time-consuming and may contain biases and errors. BLEU [@papineni2002bleu] and ROUGE [@lin2004rouge] are used commonly as an automatic measurement. They both calculate the similarity between generated distractors and ground truth. BLEU evaluates average n-gram precision on a set of reference sentences, and penalizes overly long sentences. ROUGE is a set of measurements which includes $ROUGE_{1}$, $ROUGE_{2}$ and $ROUGE_{L}$. $ROUGE_{1}$ and $ROUGE_{2}$ stand for the recall of uni-grams and bi-grams respectively, while $ROUGE_{L}$ refers to the recall of longest common subsequences between the generated and reference. However, these automatic metrics can not reflect if generated distractors would confuse test takers. Owing to the recent progress of deep learning, state-of-art VQA models’ performance is close to humans. Therefore, it is reasonable to hypothesize that existing VQA models can replace human beings at evaluating the quality of distractors. Follow this logic, we propose the performance degradation of the well-trained VQA models due to generated distrators as the metric. Since prior DG works typically apply supervised learning methods, there is a mismatch between training objective and testing metric. In addition, crafting a large multi-choice dataset is demanding and resource-intensive. To address these two issues, we adopt an reinforcement learning (RL) framework. Recent work [@rennie2017self; @li2018learning] showcase the potential of RL techniques for non-differentiable task metric issue and lacking labeled samples issue. In the proposed RL framework, we utilize the score for an input (image, question, answer choice) triplet calculated by well-trained VQA model as the reward. Therefore, a distractor generation model becomes a policy agent, where the input data is the *state* and the resultant distractor to adopt is the *action*. With the trained model serving as the *environment*, we adopt a policy gradient algorithm to optimize the DG model.

In this paper, we conduct extensive experiments on the Visual7W dataset [@zhu2016visual7w]. The quantitative result demonstrates our proposed method is able to generate challenging distractors to confuse well-trained VQA models. We also include a case study to show that the produced distractors are semantically related to the corresponding image and question. Moreover, to justify the hypothesis mentioned above, we carry out an experiment to adversarially train the models by augment the training sample with generated distractors. We observe that the accuracy of adversarially trained model achieves a sharp improvement facing the artificial distracting choices, while remain similar accuracy for the original choices. Our contributions are:

1.  We introduce a new DG-VQA task for the high-level vision and language understanding research, accompanied with a practical metric for evaluating the quality of the generated distractors. We also propose a novel perspective to formulate DG-VQA task as a reinforcement learning task and optimize it with policy gradient.

2.  We show that existing VQA models can be leveraged for learning more accurate and robust models, without the need of large human-labeled data.

3.  We present and show that by incorporating the challenging distractors will improve the performance of VQA under an adversarial learning scenario.

Related Work
============

**Distractor Generation.** Automatic distractors generation (DG) from text is explored in-depth in the Natural Language Processing domain. At the same time, there are only few studies in the visual questions domain. Most prior approaches to DG are based on unsupervised similarity measures. These include n-gram co-occurrence likelihood [@hill2016automatic], word/sentence embedding-based semantic similarities [@kumar2015revup], syntactic homogeneity [@chen2006fast] and ontology-based similarity [@stasaski2017multiple]. Besides, other works utilize supervised learning algorithms for DG. Sakaguchi et al. train a discriminative model to predict distractors, Liang et al. apply learning to rank algorithm, and Gao et al. use an end-to-end framework to produce distractors generatively. Although being successful, multimodality knowledge is still required to produce high-quality distractors.

**Visual Question Answering.** The open-ended answering task and the multiple-choice task are two typical tasks for VQA [@AntolVQA]. Here, we focus on the multiple-choice task. Existing VQA models commonly combine a CNN image encoder and an RNN question encoder which are merged before feeding to an answer decoder. The answer decoder ranges from a softmax classifier [@fukui2016multimodal], an RNN decoder [@malinowski2015ask] to a dot product layer [@jabri2016revisiting].

**Reinforcement Learning.** Reinforcement learning [@sutton2018reinforcement] has been adopted in a variety of vision and language tasks, such as image captioning [@rennie2017self], text to image synthesis [@reed2016generative], VQA [@liu2018ivqa; @fan2018reinforcement] and visual dialogue [@zhang2017asking]. Liu et al.  propose a reinforcement learning based strategy to generate visual questions. Fan et al.  enhance content and linguistic attributes of produced questions by introducing two discriminators in an RL framework.

**Adversarial Examples.** Szegedy et al. [@szegedy2013intriguing] present an initial investigation on the existence of adversarial examples. Many authors then discuss adversarial examples for image classification [@goodfellow2014generative; @nguyen2015deep]. Recent work has demonstrated that adversarial examples widely exist in various fields, including text classification [@wong2017dancin; @alzantot2018generating], autonomous cars [@sitawarin2018darts], and so on. For image, typical approaches are gradient-based methods such as fast gradient sign method (FGSM) [@goodfellow2014generative] and its variants. However, algorithms used for adversarial image samples generation can not easily transfer to the text domain. The discrete nature of natural language makes small perturbations easily perceptible, and sometimes the replacement of one word would change the semantic of the sentence dramatically. Adversarial text example requires the perturbated text to cause misclassification while preserving semantics. The goal of DG-VQA task is similar but at the opposite end. Instead of preserving semantics, distractors should mislead models towards misclassification with different semantic meaning.

![The MLP Reinforce Framework[]{data-label="fig:framework"}](images/MLP_Reinforce_Framework.pdf){width="1.05\columnwidth"}

The Approach {#sec:approach}
============

Figure \[fig:example\_DGVQA\] displays an example of the DG-VQA task. Formally, given an image $i$, a natural language question $q$ and four corresponding multi-choice answers $a$s, which include one correct answer $a^{correct}$ and three wrong answers $a^{wrong}$s in original dataset, the task is to produce three plausible but incorrect distractors $d$s. A DG-VQA is learned so that its generated distractors maximize the expected accuracy drop on a given multi-choices VQA model, where accuracy is measured as the percentage of times the model picks up the correct choice.

DG-VQA as an RL Problem
-----------------------

Inspired by recent advance in reinforcement learning and adversarial training [@moosavi2016deepfool; @yao2019improving], RL methods are ideal approaches to address lacking large scale annotated training data and inconsistency between the training objective and test metrics. So we adopt a policy gradient framework to generate textual distractor (adversarial example) for visual multiple-choice questions. The framework has two major components: an environment model $J_{\phi}$ which is a well-trained VQA model, and an agent model $G_{\theta}$ learns to confuse $J_{\phi}$ by generating high quality distractors $d$s. Here we utilize a well-trained model as the discriminator rather than train a model from scratch. The reason behind it is the concern of local convergence [@Mescheder2018ICML]. We put our approach under a semi white-box attack setting where $G_{\theta}$ can receive feedback signals regarding selected choices from $J_{\phi}$, but can not access $J_{\phi}$ parameters or gradients. From the RL perspective, the well-trained VQA model $J_{\phi}$ serves as the environment, and the generative model $G_{\theta}$ is the policy agent.

We first denote distractors generation as a sequence generation process. The generative model $G_{\theta}$ is trained to produce a sequence $y_{1:T}=(y_1,y_2,...,y_t,...,y_T)$, where $y_t$ is one word in the vocabulary of all candidate tokens. At each timestep $t$, $G_{\theta}$ is given an (image, question, answer sequence until last timestep) triple ($i$, $q$, $y_{1:t-1}$). Since $G_{\theta}$ outputs a probability distribution over each token in produced sequence, we can use decoding algorithms like greedy search or beam search to locate the top-3 distractors. Under the reinforcement learning setting, at timestep $t$ the state $s$ is the current producedly tokens $y_{1:t-1}$ and the action $a$ is the next token $y_t$ to produce. So the state transition is deterministic once an action has been chosen. Following the notation in [@sutton2018reinforcement], the object of the $G_{\theta}$ is to produce a sequence to minimize its negative expected reward:

$$L(\theta)=-\mathbb{E}_{y_{1:T}\sim G_{\theta}}[R(y_{1:T})]$$

where $y_{1:T}$ is the distractor sampled from the model $G_{\theta}$.

Without the loss of generality, we adopt the REINFORCE algorithm [@williams1992simple] to compute the policy gradient and take the predicted likelihood score of being true by the discriminator $J_{\phi}(i, q, y_{1:T})$ as the reward.

$$\begin{aligned}
    y_{1:T} &= G_{\theta}(i,q; y_{1:T-1}) \\ 
    R(y_{1:T}) &= J_{\phi}(i, q, y_{1:T}) \\
    \nabla_{\theta}L(\theta) &= -\mathbb{E}_{y_{1:T}\sim G_{\theta}}[R(y_{1:T})\nabla_{\theta}log G_{\theta}(y_{1:T})].\end{aligned}$$

It is worth mentioning that the discriminator can only output a reward value from a completed sequence. However, in DG-VQA setting and under sequence generation scenario, the model should consider the long-term reward at every timestep. To tackle this challenge, researchers typically use Monte Carlo search to sample the unknown last $T-t$ tokens. For simplicity, we generate the final distractor sequence $d$ for one time step by selecting the output distractor over a distractors pool, and empirical results show that it is already effective. Thus, the distractor generator can be formulated as follows: $$\begin{aligned}
    d = y_{1:T} = G_{\theta}(i, q).
  \end{aligned}$$

The framework is dubbed as *MLP Reinforce* (MLPR) since we adopt a MLP architecture as the agent in a Reinforcement learning setting. Equation \[eqn:reward\] defines the return of sampled distractor answer choices. In general, we take the output of the well-trained multi-choice VQA model as the reward. Furthermore, we punish the distractor $d$ which is semantically equivalent to the correct answer $a^{correct}$ for the given context. The semantic similarity model is trained in the BERT [@devlin2018bert] architecture. We put details about the similarity model in Section \[subsec:baselines\]. $$\label{eqn:reward}
    R(d) = \left\{
      \begin{array}{lr}
      -1 & if\ IsSemEquiv(d, a^{correct});
      \\ J_{\phi}(i, q, d) & otherwise.
      \end{array}
    \right.$$

In practice, the expected gradient can be approximated using several distractors $d^s$ sampled from $G_{\theta}$ for each input image, question and correct answer triplet in a minibatch.

$$\label{eqn:gradient_practical}
    \nabla_{\theta}L(\theta) \approx \sum_{s} -R(d^s)\nabla_{\theta}log G_{\theta}(d^s).$$

The Agent: Multiple-Layer Perceptron
------------------------------------

The architecture of our generative model $G_{\theta}$ is a multiple-layer perceptron. It is widely used in many domain and has shown success in visual question answering by Jabri’s work [@jabri2016revisiting]. Figure \[fig:mlp\_architecture\] depicts the layers of the MLP model. Specifically, the questions are represented by averaging word2vec [@mikolov2013distributed] embedding over all tokens. The images are represented using features computed by a pre-trained deep CNN image encoder. Unless otherwise stated, we use the penultimate layer of Resnet-101 [@he2016deep] as the extracted features. Then, the multi-modality feature sets are concatenated and used to train a classification model that predicts the corresponding distractor label.

![Multiple Layer Perceptron Architecture for DG-VQA[]{data-label="fig:mlp_architecture"}](images/Attack_VQA_MLP.pdf){width="\linewidth"}

A multilayer perceptron (MLP) is adopted as the classification model trained on the concatenated features: 1) The word2vec embedding (300-dimensional), and 2) the image features (2048-dimensional). By default, the MLP has 4,096 hidden units unless otherwise specified. We denote the image and question features as $x_i$ and $x_q$, respectively. By denoting the concatenation as $c = x_i \oplus x_q$, we formulate the models as follows: $$z = (w_2\ ReLu(w_1c + b_1) + b_2).$$ The MLP outputs a distribution over the distractor pool using the softmax function. Then, the system selects the distractors $ds$ with the top-3 highest likelihood from $G_{\theta}(i, q)$. $$\label{d_probabiltiy}
    P(d|i, q) = softmax(z).$$

The Environment: VQA Models {#subsec:environment}
---------------------------

Any multi-choice VQA model which produces a likelihood score of answers for given visual questions can serve as the environment in the MLPR framework. Here, we conduct attacks on three state-of-the-art VQA models:

**TellingVQA**  is a recurrent QA model with spatial attention. It first encodes the image through a pretrain VGG-16 model [@Simonyan15]. Then it uses a one-layer LSTM to read the image encoding and all the question tokens.It continue feed the answer choice tokens into LSTM, and would finally produce the likelihood score.

**RevisitedVQA**  proposes a quite simple architecture for VQA multiple choice task. RevisitedVQA receives an image-question-answer triplet, encodes it and utlizes a MLP to compute whether or not the triplet is correct.

**MCB**  proposes a novel method called Multimodel Compact Biliniear pooling to efficiently and expressively combine language and vision features.

The MLPR can also support generating distractors over a bundle of well-trained models together as the environment by providing a mixed reward.

The baselines we incorporated here seem not up-to-date. It is worth noting that the proposed distractor generation method is not restricted to these models, but generally applicable to any VQA models which can produce confidence scores regarding the answers they choose.

Experimental Results {#sec:results}
====================

Datasets and Evaluation Metrics
-------------------------------

We evaluate our model on **Visual7w** [@zhu2016visual7w], which is a public multiple-choice visual question answering dataset. **Visual7w** consists of 47,300 images from COCO and 327,939 multiple-choice QA pairs collected on Amazon Mechanical Turk.

As mentioned in Section \[sec:intro\], traditional metrics for DG such as reliability and validity highly rely on manual evaluations. Inspired by adversarial attack evaluation, we define the ability of generated distractors to fool well-trained VQA models as the metric, denoted as $\Delta Acc$. $\Delta Acc$ is the difference between VQA model’s performance on the original distractors and on the generated distractors. The learning objective of proposed method is directly related to this metric and thus can eliminate the mismatch between training goal and test measurement. $\Delta Acc$ is an automatic metric. The higher $\Delta Acc$ is, the better generated distractors are.

Baselines {#subsec:baselines}
---------

We implemente the following baselines:

**Per Q-type prior**: We select 3 most popular answers per question type as distractors.

**Adversarial Matching**: Zellers et al.  constructs multiple choice questions by balance two measurements: 1) distractors must be relevant to the context; 2) distractors can not be overly similar to the correct response. The relevance model is trained on Viusal7W train split. We choose the question and correct answer pair as one positive sample, and pick up the same specific question and some 5 answers from the whole answer pool as corresponding negative samples. For similarity model, we use sentences pairs with ‘entailment’ label from SNLI [@snli:emnlp2015] and MultiNLI [@williams2018broad] as training data. We then employ BERT [@devlin2018bert] architecture for both relevance model and similarity model. The similarity model is also used to filter semantic equivalent distractors of correct choice in each baseline and proposed method.

**LSTM Q+I**: The architecture is similar to the two channels neural networks in [@AntolVQA] which the visual features and textual features are fused by point-wise multiplication. We keep the fine-tuned VGG and two-layer LSTM and for encoding input context, while only change the training targets from correct choices to incorrect ones. These incorrect choices are failed predictions from well-trained VQA models.

As we can see, Per Q-type prior is a heuristic method for distractor generation. Adversarial Matching leverages external knowledge to measure the two major characteristics of high quality distractors. LSTM Q+I further takes visual and textual clues. These three baselines tackle DG-VQA from different perspetives.

Experimental Settings
---------------------

We adopt a two-channel vision and language neural network that outputs probabilities over $K$ candidate distractors as the agent. We set the candidate distractor frequency threshold to $20$, to filter the candidate pool size $K$ to $1516$. This set of answer choices covers $2\%$ of the training and validation answer choices. For text preprocessing, we convert the dataset to lowercase and filter out punctuations. The questions are represented by 300-dim averaged word embeddings from a pre-trained fastText [@bojanowski2017enriching] model. We use all words in the training and validation dataset to train the embedding. In the experiment, we set dropout to $0.5$ in each hidden layer with a ReLU activation. We train the MLP for $200$ epochs, which is determined emprically.

For the environment, we adopt the best RevisitedVQA model in  [@jabri2016revisiting]. The well-trained RevisitedVQA model outperforms other state-of-the-art models which are mentioned in \[subsec:environment\], and it achieves $65.8\%$ accuracy on the Visual7W dataset. We evaluate the propose MLP Reinforce model with two ablated versions:

**MLPR**: Here, model parameters are updated only through policy gradient. We train the model with the rewards from the well-trained RevisitedVQA  environments.

**MLPR+ Pre-train**: Reinforce algorithm is known to have large variance. Inspired by the concept of Imitation Learning  and Teacher Forcing , we first pre-train the MLP model with correct answer choice using cross-entropy loss for a small size (80) epochs. The pre-train process is to prevent generating unstable results. Then we trained the model as described before. An interpretation of adopting this practical training strategy is by an analogy with the undercover police: being integrated and then attack.

[l|cc|cc|cc]{} & & &\
& $Acc$ & $\Delta Acc$ & $Acc$ & $\Delta Acc$ & $Acc$ & $\Delta Acc$\
Original & 55.6% & - & 64.8% & - & 62.2% & -\
\
per Q-type prior & 57.3% & -1.7% & 68.7% & -3.9% & 85.7% & -23.5%\
Adversarial Matching & 54.7% & 0.9% & 71.7% & -6.9% & 51.3% & 10.9%\
LSTM Q+I & 41.7% & 13.9% & 68.9% & -4.1% & 85.7% & -23.5%\
\
Reward by RevisitedVQA\
- MLPR & 86.5% & -30.9% & & **64.7%** & & **35.7%**\
- MLPR + Pre-train & & **21.9%** & 49.1% & 15.8% & 37.5% & 24.7%\

Experimental Results {#experimental-results}
--------------------

Table \[tab:attack\] list the attacking results of the generated distractors for Visual7W Dataset. since the three defending models use different architectures, the distractor generation model requires high generalization capability to confuse all three of them. Baseline models yield poor overall $\Delta Acc$. Per Q-type prior fails to fool any defenders. Adversarial matching and LSTM Q+I are lack of generalization capability, which are only able to make one defender’s accuracy flipping. Our proposed MLPR methods yield statistically significant improvements on both defenders. Note that MLPR performs better on RevisitedVQA and MCB than MLPR+Pretrain, while worse on TellingVQA($\Delta Acc$ -30.9%). It indicates that without pre-train MLPR is vulnerable to overfitting, although it is impressive that MLPR successfully confuses RevisitedVQA in almost every question($Acc$ 0.01%)! Table \[tab:case\_study\] provides a case study (in Section \[subsec:casestudy\]) of the distractors with the given context extracted by our models.

Furthermore, the attacking performance boost from baseline models to MLPR is considerable according to automatic metrics and case study. This shows that feedbacks from well-trained VQA models are beneficial to distractor generation. Only receiving rewards from one specific environment, MLPR + Pre-train method produces generalized distractors to fool all three defenders. We speculate that the pre-train process in this case provides a better beginning probability distribution over distractors. Therefore, it prevents the agent from falling into the local minima trap.

Augmenting VQA with DG-VQA
--------------------------

We adopt DG-VQA for training more robust VQA models as a data augmentation process. To validate it, an extra data augmentation experiment is conducted, where we keep the correct alternative of each question and swap the original incorrect choices to the generated distractors. MCB and RevisitedVQA are better suited for this setting since they take both correct and incorrect alternatives into consideration while training, while TellingVQA only takes the correct answer as input. We re-train the two VQA multiple-choice models on two settings: 1) adversarial examples alone, 2) the union of these examples and the original training data. As a control group, we adopt the VQA models which are trained on the original data alone. Here, adversarial examples are produced by our MLPR + Pre-train method, which has been shown the most effective.

[0.9]{}

coordinates [(\[O\],62.2) (\[A\],43.9) (0.5\[O\]+0.5\[A\],61.1)]{}; coordinates [(\[O\],37.5) (\[A\],79.7) (0.5\[O\]+0.5\[A\],65.4)]{}; coordinates [(\[O\],64.8) (\[A\],39.8) (0.5\[O\]+0.5\[A\],55.0)]{}; coordinates [(\[O\],49.1) (\[A\],78.8) (0.5\[O\]+0.5\[A\],75.2)]{};

Figure \[fig:augment\] reports the results, where x-axis indicates on which dataset the models are trained. $[O]$ and $[A]$ refer to the original data and the augmented data respectively. And $0.5[O]+0.5[A]$ denotes $50\%$ of all questions’ incorrect alternatives are replaced by the generated distractors. Different bars indicate the VQA accuracy that models can achieve on a specific dataset. At first glance, we find that data augmentation training improve the models’ performance on adversarial examples. However, it hurts the performance on the original test data. Models trained on the union of augmented and original data achieves the best performance with minimal $Acc@[O]$ drop of $1.1\%$ and highest $Acc@[A]$ improvement by $27.9\%$. These results demonstrate the effectiveness of DG-VQA for training more robust VQA models.

[X X X X X]{} & & & &\
Q:What does the sky look like?& Q:How many black cows are there?& Q:What sport are they playing?& Q:Why is there a piece missing?& Q:What two colors are in the flag directly above the cats head?\
Original Choices\
& & & &\
A: Hazy& A: 9& A: **Baseball** & A: It was removed& A: **Blue and white**\
A: Windy& A: 8& A: Hockey& A: **It was put somewhere else** & A: Black and red\
A: Sunny& A: 7& A: Basketball& A: Someone took it& A: Green and black\
\
A: Stormy& A: 3& A: **Golf** & A: Someone ate some& A: Green and yellow\
A: **Sky** & A: Zero& A: Volleyball& A: Wood& A: Blue and black\
A: Blue& A: 5& A: Playing soccer& A: **Glass** & A: Blue and red\
A: Cloudy& A: **0** & A: Soccer& A: To rest& A: **Blue and white**\
\
A: Stormy& A: 3& A: Golf& A: Someone ate some& A: Green and yellow\
A: Shadows& A: **Shadows** & A: Shadows& A: Shadows& A: **Shadows**\
A: Daylight& A: During daylight& A: **During daylight** & A: **Daylight** & A: Daylight\
A: **Shadow** & A: In the daytime& A: Daylight& A: During daylight& A: In the daytime\
\
A: Stormy& A: 3& A: Golf& A: Someone ate some& A: **Green and yellow**\
A: **Cloudy** & A: Two& A: **Baseball** & A: **To eat** & A: Blue\
A: Blue& A: Four& A: Soccer& A: To cook& A: Legs\
A: Clouds& A: **One** & A: Tennis&A: For display& A: Orange\

Case Study {#subsec:casestudy}
----------

We collect sample adversarial distractor choices generated by baseline and the proposed methods. Table \[tab:case\_study\] showcases these samples and their corresponding images, questions and original choices. Oberseving the predictions made by the defender (RevisitedVQA model), our proposed methods, MLPR + Pretrain, exhibits an ability to learn certain interpretable and practical policies generating adversarial and distracting alternatives:

**Semantic Similarity**: It is not surprising the proposed distractors generator learned the strategy to take advantage of semantically similar tokens to the correct answer. Human beings follow the same strategy when they try to come up with distracting alternatives. As we can see, distractors generated by our method and the correct answer almost belong to the same general concept category. For example, “baseball”, “soccer”, “tennis” and “golf” are sport terms. And all distractors produced for the question “how many black cows are there” are all numbers, which belong to the same category of the correct answer: “3”.

**Context Matters**: The other critical factor is the context. In the first column of Table \[tab:case\_study\], both distractors of the original dataset and of augmented ones are adjectives to describe the weather. However, “cloudy” is a better distractor to “stormy” if we take a look at the image, compared to “hazy”, “windy” and “sunny”. Under the original choice setting, the defender is able to select the correct answer. But once encountering with the generated distractors, it is confused and misleadingly pick “cloudy” as the answer. Tackling vision and language tasks needs multimodal cognitive ability. In the DG-VQA task, a system should comprehensively utilize information from both the given questions and the images.

**Attack the Weaknesses and Improve**: Our architecture is able receive feedback from the defender. It is a common sense to exploit opponents’ weaknesses to defeat them. By analyzing confidence scores of the alternatives, the distractor generator identifies the differences between the hard and the easy ones. Examples of this can be found in distractors generated by MLPR (See Table \[tab:case\_study\]). It seems our system generates easy-to-human distractors like “shadows” or “daylight”, the defender is observed to be defeated in fact. Intuitively speaking, the fatal drawback could be attributed to the trained model’s overly biased objective function. Our method is able to identify and exploits them. Further, by considering these weakness for the next round of training, a model’s robustness is improved. The above case studies support that our method in fact outputs more challenging distractors by considering all together the semantics of correct answer, the information of the context, and the feedback from the trained discriminative model.

Conclusion and Future Work {#sec:conclusion}
==========================

We introduced the textual distractor generation task for visual question answering (DG-VQA). These “hard negative“” distractors are significantly important when deep learning models have been applied in many real-life and safety-sensitive environments. In this paper, we propose a hypothesis that we can replace human evaluation that is widely used in previous work with well-trained models. We then justify the hypothesis through the augmenting experiment. The experimental outcome further validates that generated distractor could be utilized to improve the robustness of VQA models. Moreover, we developed a policy gradient based model dubbed as MLPR to utilize feedback from exisiting models for distractors generation. The generated distractors achieve high successful rates to make well-trained VQA models confuse. The proposed approach is demonstrated to be effective in generating challenging distractors, which address the lacking human-labeled data issue.

We hold the view that the DG-VQA task and the adversarial training towards distractor generation for visual questions pave a new pathway for further research in robust and anti-adversarial VQA. There are several caveats of our method that is worth mentioning. For instance, the alternatives generated are less diverse and the lingering concern of over-fitting by our proposed MPLR and MPLR+Pre-train methods. It sparks future directions such as devising better distractor generation approaches in both visual and textual question answering research fields. The other future research direction that worth exploring is to generate distractors in visual domain.

[^1]: Data and code will be made publicly available.
