<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Text Data Explanation Benchmarking
Step1:  Load Data and Model
Step2:  Class Label Mapping
Step3:  Define Score Function
Step4:  Create Explainer Object
Step5:  Run SHAP Explanation
Step6:  Define Metrics (Sort Order & Perturbation Method)
Step7:  Benchmark Explainer
<ASSISTANT_TASK:>
Python Code:
import copy
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import shap.benchmark as benchmark
import shap
import scipy as sp
import nlp
import torch
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.set_option('max_colwidth', None)
Explanation: Text Data Explanation Benchmarking: Emotion Multiclass Classification
This notebook demonstrates how to use the benchmark utility to benchmark the performance of an explainer for text data. In this demo, we showcase explanation performance for partition explainer on an Emotion Multiclass Classification model. The metrics used to evaluate are "keep positive" and "keep negative". The masker used is Text Masker.
The new benchmark utility uses the new API with MaskedModel as wrapper around user-imported model and evaluates masked values of inputs.
End of explanation
train, test = nlp.load_dataset("emotion", split = ["train", "test"])
data={'text':train['text'],
     'emotion':train['label']}
        
data = pd.DataFrame(data)
tokenizer = AutoTokenizer.from_pretrained("nateraw/bert-base-uncased-emotion",use_fast=True)
model = AutoModelForSequenceClassification.from_pretrained("nateraw/bert-base-uncased-emotion")
Explanation: Load Data and Model
End of explanation
# set mapping between label and id
id2label = model.config.id2label
label2id = model.config.label2id
labels = sorted(label2id, key=label2id.get)
Explanation: Class Label Mapping
End of explanation
def f(x):
    tv = torch.tensor([tokenizer.encode(v, padding='max_length', max_length=128,truncation=True) for v in x])
    attention_mask = (tv!=0).type(torch.int64)
    outputs = model(tv,attention_mask=attention_mask)[0].detach().numpy()
    scores = (np.exp(outputs).T / np.exp(outputs).sum(-1)).T
    val = sp.special.logit(scores) 
    return val
Explanation: Define Score Function
End of explanation
explainer = shap.Explainer(f,tokenizer,output_names=labels)
Explanation: Create Explainer Object
End of explanation
shap_values = explainer(data['text'][0:20])
Explanation: Run SHAP Explanation
End of explanation
sort_order = 'positive'
perturbation = 'keep'
Explanation: Define Metrics (Sort Order & Perturbation Method)
End of explanation
sequential_perturbation = benchmark.perturbation.SequentialPerturbation(explainer.model, explainer.masker, sort_order, perturbation)
xs, ys, auc = sequential_perturbation.model_score(shap_values, data['text'][0:20])
sequential_perturbation.plot(xs, ys, auc)
sort_order = 'negative'
perturbation = 'keep'
sequential_perturbation = benchmark.perturbation.SequentialPerturbation(explainer.model, explainer.masker, sort_order, perturbation)
xs, ys, auc = sequential_perturbation.model_score(shap_values, data['text'][0:20])
sequential_perturbation.plot(xs, ys, auc)
Explanation: Benchmark Explainer
End of explanation
<END_TASK>