### Explainable Metrics - Machine Translation
This repository contains the source code for the experiments conducted in our paper
"Towards Explainable Evaluation Metrics for Natural Language Generation".

### Reference
If you build upon this work, we'd be happy when you cite our paper:

```bib
@misc{https://doi.org/10.48550/arxiv.2203.11131,
  doi = {10.48550/ARXIV.2203.11131},
  url = {https://arxiv.org/abs/2203.11131},
  author = {Leiter, Christoph and Lertvittayakumjorn, Piyawat and Fomicheva, Marina and Zhao, Wei and Gao, Yang and Eger, Steffen},
  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Towards Explainable Evaluation Metrics for Natural Language Generation},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution 4.0 International}
}
```

### Description
This project contains code for experiments regarding
* Hard Reference Free Metrics for Explainability
  * To apply these experiments, follow these steps:
    * Use `xai/ShapExplainer.py` to explain the metrics `TranslationMeteor` and `TranslationBleu`.
    * To apply word by word mode for TranslationMeteor, set `self.use_word_by_word=True` in `metrics/collection/TranslationMeteor.py` 
* Adversarial Attacks on Machine Translation Evaluation Metrics
  * To apply these experiments, follow these steps:
    * Use `metrics/corpora/wmt/download_wmt.py` and follow the instructions in `metrics/corpora/wmt/README.md` to prepare a wmt corpus as pandas df.
    * Use `metrics/evaluation/ProduceBins.py` to apply metrics to the corpus and determine bins for the discretization of adversarial attacks
    * Use `xai/AdversarialAttackExplainer.py` to apply attackers to the generated samples
    * Use `xai/evaluation/evaluate_attacks.py` to apply an automated evaluation
* Inverse Metrics
  * Use `xai/GreedyExplainer.py` to use the algorithm for inverse sample generation

Additionally, this project contains a skeleton to apply different explainability techniques to explain machine translation evaluation metrics.
The project is divided into two main folders *metrics* and *xai*. The *metrics* folder contains wrappers for machine translation
evaluation (MTE) metrics and evaluation loops. The *xai* folder contains a collection of Explainer classes that
explain the metrics. Further, it contains scripts that can check the generated explanations depending on their type.

For detailed information refer to the readme's and comments in the respective folders.

If files are used and edited from another projects, it is indicated in a README file at the top-level of the 
respective folder. E.g. `metrics/collection/metrics_libs/bertscore` contains libraries of BERTScore, so there 
 is a README in that folder indicating this source. Otherwise, the sources of code passages, are specified in the comments of the code.

In this main README we detail the usage of the core functionalities. At some points we explicitly set cuda,
therefore, this code will only run with a gpu.

### Installation
This project was run on Windows, therefore there might be some compatibility issues with linux. To install the packages
use the `requirements.txt` file that was build with `pipreqs`. It is recommended to use a new conda environments.
The compatibility with COMET can be a bit difficult to set up. If this occurs, please have a look at `metrics/collection/Comet.py`.
The root directory of this project can be referred to by importing `from project_root import ROOT_DIR`.


### Running Metrics
Most of the scripts in this project expect their input in terms of a pandas dataframe, where each datapoint describes a 
sample from a dataset for metric evaluation. I.e. the columns include SRC (source), HYP (hypothesis), REF (reference),
LP (language pair), SYSTEM (nmt system). Not all of the columns are required for all methods and some methods use additional
columns. Applying a metric to a dataframe means that a new column is created that contains the scores for each evaluated 
datapoint for a metric. To produce the pandas dataframes, helper scripts can be found in `metrics/corpora`.
The evaluation loop, that applies the metrics to a dataframe is implemented in `metrics/collection/MetricWrapper.py`.
When a new corpus should be used, it can simply be provided as a Pandas dataframe with the respective columns.

### Applying explanations
Explanations use a different loop that is also implemented in `metrics/collection/MetricWrapper.py`. The classes that contain
the Explainers are defined in `xai/`. They also expect a pandas dataframe that holds the input samples (with the same columns).
As a difference, the Explainers do not only evaluate the metrics, but also how the metrics will behave with perturbed input 
(or use other functionalities to explain why a metric produced a certain result). Each Explainer class contains a usage example
in its main block. In general, they can be applied with following utility function:

```python
import os
from project_root import ROOT_DIR
from xai.util.corpus_explainer import explain_corpus

# Initialize the explainer object (Pseudo Code)
E = ExplainerObject()

# Apply the explainer to a corpus
explain_corpus(E,                                   # The explainer object
               recover=False,                       # Whether to recover previous explanations
               from_row=0, to_row=100,              # Which rows of a corpus to consider (default corpus are mlqe-pe et-en rows)
               outfile='attribution.json')          # The name of the output file

# After this, feature importance explanations are saved in the specified output file under xai/outputs/explanations. 
# Adversarial attacks are instead evaluated with the objects that are saved to xai/outputs/explanation_checkpoints.
``` 


### Evaluating plausibility with AUC, AP, RtopK to word-level error annotations
The evaluation with AUC, AP and RtopK is handled and described in `xai/evaluation/`. Extending the previous pseudocode,
a simple evaluation can be performed by calling:
```python
# loads the previous output file that was prepended with its starting and ending row
evaluate_mlqe_auc([
        os.path.join(ROOT_DIR,'xai/output/explanations/0_99_attribution.json')], 
        start=0,                        # The start row
        end=100,                        # The end row
        invert=True,                    # Whether to invert feature importance scores
        f=20)                           # Starting increment for output filenames
```

The results are printed as pandas dataframe. The script `xai/SystemPaperScores.py` reproduces scores of the Eval4NLP system paper.


### Evaluating adversarial attacks
Adversarial attacks on metrics can be applied using the `xai/AdversarialAttackExplainer.py`. For usage, see the comments. They can be evaluated using the file 
`xai/evaluation/evaluate_attacks.py`. 

Sample usage of Bert-Attack (the parameters are described in the respective file):
```python
AE = AdversarialExplainer(binning_mode = 'corpus', bins = 3, attack_types = ['bert-attack'], target = 0,
                              corpus=os.path.join(ROOT_DIR,'metrics/outputs/wmt/wmt20_full_scores_raw_zh_en'))
explain_corpus(AE,
           recover=True,
           from_row=0, to_row=1200,
           outfile='attack_attributions',
           mlqe_pandas_path=os.path.join(ROOT_DIR,'metrics/corpora/pandas_corpora/wmt/wmt20_classes/2_class_zh_en'),
           explanation_path=os.path.join(ROOT_DIR,'xai/output/explanation_checkpoints/adv_attack_zh_en_2_to_0_tf.dill'))
```
This will attack all metrics using the sentences from a corpus passed with `mlqe_pandas_path` and save explanation objects to `explanation_path`.

### Adding new metrics
In order to add a new metric to the evaluations, create a new class in `metrics/collection` and implement the class `MetricClass` (see the README.md of that folder).
You can use the other metrics as an example. Further, add the metric to the `MetricWrapper` class, to enable it for evaluations. Therefore,
add the metric to the dictionary that is defined in the beginning of the `MetricWrapper.py` script. 

### Adding new explainability techniques
In order to add a new explainability technique, create a new class in `xai` that implements `xai/Explainer`. As an example
you can also look at the existing explainbility techniques. 

### Walkthrough
Finally, we present a short walkthrough that shows the parts that were teaserd before in action. Assume we have written a 
new metric `NewScore`, that grades translations based on the length of the shortest word. I.e. if a translation contains a 
word with only one letter, it is a good translation:

```python
from metrics.collection.MetricClass import MetricClass

class NewScore(MetricClass):
    ref_based = True
    name = 'NewScore'


    def __call__(self, ref, hyp):
        res = []
        for h in hyp:
            try:
                res.append(-min([len(a) for a in h.split()]))
            except:
                res.append(0.0)

        return res
``` 

Let us try out out new metric:
```python
n = NewScore()
print(n(['Any Reference Sentence', 'Any Reference Sentence'], ['A sentence with a short word', 'Long wordy sentence']))

# [-1, -4]
```
As you can see our metric assigned a higher score to the sentence with the short word `A`.

Next, let us assume we have a dataset that contains human annotations on translation quality. Then we could transform this
dataset into a pandas dataframe as follows:

```python
import pandas as pd

test_df = pd.DataFrame([['de-en', 'Ein cooler Satz', "A cool sentence", "Even more cool sentences", 'Dummy', 0.9],
                        ['de-en', 'Ein zweiter cooler Satz', "A second cool sentence", "Even more cool sentences two", 'Dummy', 0.3],
                        ['de-en', 'Das ist wirklich der letzte Satz', "This really is the last sentence", "This is not the last sentence", 'Dummy', 0.5]],
                       columns=['LP', 'SRC', 'REF', 'HYP', 'SYSTEM', 'DA'])
```
This dataframe should contain columns for the language pair (here German-English), for the source sentence SRC, the reference sentence REF,
the hypothesis HYP, the System SYSTEM and the human annotations DA. Depending on the metric not all of these values are necessary and 
some can be filled with dummies (for our metric SRC and REF could be dummies as well).

If we do not register our metric manually for the evaluation loops in the source code, we can do so during runtime:
```python
from metrics.collection.MetricWrapper import MetricWrapper

metric_wrapper = MetricWrapper()
metric_wrapper.metrics['NEWSCORE'] = NewScore
``` 

To evaluate our metric for the pearson correlation with human judgements, we can now use the evaluation loop of the MetricWrapper:
```python
from metrics.evaluation.pearson_eval import evaluate_seg
scores = metric_wrapper.evaluate(test_df, metrics = ['NEWSCORE'])

print(evaluate_seg(scores, metrics = ['NEWSCORE'], output=None, da_column='DA'))

# LP	NEWSCORE
# de-en	-0.6546536707079772
```

As you can see, our metric has a rather weak correlation with human judgements ;) 


Next, assume our dataset additionally contains ground-truth word-level error annotations (in this case non-sensical) that we want to compare our 
scores with. Therefore, lets reassign the example from before:
```python
test_df = pd.DataFrame([['de-en', 'Ein cooler Satz', "A cool sentence", "Even more cool sentences", 'Dummy', 0.9, '[0, 1, 0, 0]'],
                        ['de-en', 'Ein zweiter cooler Satz', "A second cool sentence", "Even more cool sentences two", 'Dummy', 0.3, '[0, 0, 0, 0, 1]'],
                        ['de-en', 'Das ist wirklich der letzte Satz', "This really is the last sentence", "This is not the last sentence", 'Dummy', 0.5, '[0, 1, 0, 0, 0, 0]']],
                       columns=['LP', 'SRC', 'REF', 'HYP', 'SYSTEM', 'DA', 'TAGS_HYP'])
```

Then we can evaluate the AUC, AP, RtopK scores as follows:

```python
from xai.util.corpus_explainer import explain_corpus
from xai.evaluation.eval4nlp_evaluate import evaluate_mlqe_auc
# Save our corpus to a file
test_df.to_csv('test_corpus.tsv', sep='\t', index=False)

# Use the explanation utils to explain directly on the file (and save the attributions to JSON)
explain_corpus(LE, metrics=['NEWSCORE'], recover=False,from_row=0, to_row=4,outfile='NewScore_le.json',outpath='',mlqe_pandas_path='test_corpus.tsv',)
explain_corpus(IM, metrics=['NEWSCORE'], recover=False,from_row=0, to_row=4,outfile='NewScore_im.json',outpath='',mlqe_pandas_path='test_corpus.tsv')

# Evaluate the resulting explanation files
evaluate_mlqe_auc(['0_3_NewScore_le.json'], start=0, end=4, invert=True, f=100, corpus_path='test_corpus.tsv')
evaluate_mlqe_auc(['0_3_NewScore_im.json'], start=0, end=4, invert=True, f=100, corpus_path='test_corpus.tsv')

# The last 6 functions will return the following results:
#     Metric  AUC     AP  REC_TOPK  PEARSON
#0  NEWSCORE  0.0  0.206       0.0   -0.655
#     Metric    AUC   AP  REC_TOPK  PEARSON
#0  NEWSCORE  0.444  0.5     0.333   -0.655
```

As a last step lets assume we want to attack our metric on the three presented samples. Therefore
we can use an attack explainer:
```python
from xai.AdversarialAttackExplainer import AdversarialExplainer

AE = AdversarialExplainer(binning_mode = 'loose', bins = 5, loose_span=5, attack_types = ['bert-attack'], target = 0)
AE.MW.metrics['NEWSCORE'] = NewScore
im_explanations = AE.apply_explanation(test_df, metrics=['NEWSCORE'], explanation_path='')

# In this case no attack succeeded
# NEWSCORE : 
# 2 (100%) --> [FAILED]
# Even more cool sentences
# src: Ein cooler Satz
# ref: A cool sentence
# orig_score: -4
# ---------------------------------
# NEWSCORE : 
# 3 (100%) --> [FAILED]
# Even more cool sentences two
# src: Ein zweiter cooler Satz
# ref: A second cool sentence
# orig_score: -3
# ---------------------------------
# NEWSCORE : 
# 4 (100%) --> [FAILED]
# This is not the last sentence
# src: Das ist wirklich der letzte Satz
# ref: This really is the last sentence
# orig_score: -2
```
