Model details
=============

For all datasets, we use [spaCy](https://spacy.io/) for tokenization. We map out of vocabulary words to a special $\texttt{<unk>}$ token and map all words with numeric characters to ‘qqq’. Each word in the vocabulary was initialized to pretrained embeddings. For general domain corpora we used either (i) [FastText Embeddings](https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.simple.vec) (SST, IMDB, 20News, and CNN) trained on Simple English Wikipedia, or, (ii) [GloVe 840B](http://nlp.stanford.edu/data/glove.840B.300d.zip) embeddings (AGNews and SNLI). For the MIMIC dataset, we learned word embeddings using [Gensim](https://radimrehurek.com/gensim/) over all discharge summaries in the corpus. We initialize words not present in the vocabulary using samples from a standard Gaussian $\mathcal{N}$($\mu=0$, $\sigma^2=1$).

BiLSTM
------

We use an embedding size of 300 and hidden size of 128 for all datasets except bAbI (for which we use 50 and 30, respectively). All models were regularized using $\ell_2$ regularization ($\lambda = 10^{-5}$) applied to all parameters. We use a sigmoid activation functions for binary classification tasks, and a softmax for all other outputs. We trained the model using maximum likelihood loss using the Adam Optimizer with default parameters in PyTorch.

CNN
---

We use an embedding size of 300 and 4 kernels of sizes \[1, 3, 5, 7\], each with 64 filters, giving a final hidden size of 256 (for bAbI we use 50 and 8 respectively with same kernel sizes). We use ReLU activation function on the output of the filters. All other configurations remain same as BiLSTM.

Average
-------

We use the embedding size of 300 and a projection size of 256 with ReLU activation on the output of the projection matrix. All other configurations remain same as BiLSTM.

Graphs
======

To provide easy navigation of our (large set of) graphs depicting attention weights on various datasets/tasks under various model configuration we have created an interactive interface to browse these results, accessible at: <https://successar.github.io/AttentionExplanation/docs/>.

Adversarial Heatmaps
====================

**SST**

**Original**:

**Adversarial**: **$\Delta\hat{y}$**: *0.005*

**IMDB**

**Original**:

**Adversarial**: **$\Delta\hat{y}$**: *0.004*

**20 News Group - Sports**

**Original**:

**Adversarial**: **$\Delta\hat{y}$**: *0.001*

**ADR**

**Original**:

**Adversarial**: **$\Delta\hat{y}$**: *0.002*

**AG News**

**Original**: **Adversarial**: **$\Delta\hat{y}$**: *0.006*

**SNLI**

**Hypothesis**:a man is running on foot

**Original Premise Attention**:

**Adversarial Premise Attention**: **$\Delta\hat{y}$**: *0.002*

**Babi Task 1**

**Question**: Where is Sandra ?

**Original Attention**:

**Adversarial Attention**: **$\Delta\hat{y}$**: *0.003*

**CNN-QA**

**Question**:federal education minister @placeholder visited a @entity15 store in @entity17 , saw cameras

**Original**:

**Adversarial**: **$\Delta\hat{y}$**: *0.005*
