<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 LIT Standalone Components
This notebook shows use of the Language Interpretability Tool components on a binary classifier for labelling statement sentiment (0 for negative, 1 for positive).
All LIT backend components (models, datasets, metrics, generators, etc.) are standalone Python classes, and can easily be used from Colab or another Python context without starting a server. This can be handy for development, of if you want to re-use components in an offline workflow.
Copyright 2021 Google LLC.
SPDX-License-Identifier
Step1:  Load data
LIT's Dataset classes are just lists of records, plus spec information to describe each field.
Step2:  You can easily convert this to tabular form, too
Step3:  Load a model and run inference
LIT's Model class defines a predict() function to perform inference. The input_spec() describes the expected inputs (it should be a subset of the dataset fields), and output_spec() describes the output.
Step4:  There's a lot of fields in the output spec, since this model returns embeddings, gradients, attention, and more. We can view it using Pandas to avoid too much clutter
Step5:  If we just want the predicted probabilites for each class, we can look at the probas field
Step6:  Salience methods
We can use different interpretability components as well. Here's an example running LIME to get a salience map. The output has entries for each input field, though here that's just one field named "sentence"
Step7:  Metrics
We can also compute metrics. The metrics components (via the SimpleMetrics API) will automatically detect compatible fields marked by the parent attribute - in this case, our model's probas field that should be scored against label in the input.
Step8:  Generators
We can use counterfactual generators as well. Here's an example with a generator that simply scrambles words in a text segment.
Step9:  Running the LIT UI
Of course, you can always still use these components in the LIT UI, without leaving Colab.
<ASSISTANT_TASK:>
Python Code:
# Install LIT and transformers packages. The transformers package is needed by the model and dataset we are using.
# Replace tensorflow-datasets with the nightly package to get up-to-date dataset paths.
!pip uninstall -y tensorflow-datasets
!pip install lit_nlp tfds-nightly transformers==4.1.1
Explanation: LIT Standalone Components
This notebook shows use of the Language Interpretability Tool components on a binary classifier for labelling statement sentiment (0 for negative, 1 for positive).
All LIT backend components (models, datasets, metrics, generators, etc.) are standalone Python classes, and can easily be used from Colab or another Python context without starting a server. This can be handy for development, of if you want to re-use components in an offline workflow.
Copyright 2021 Google LLC.
SPDX-License-Identifier: Apache-2.0
End of explanation
import attr
import pandas as pd
from lit_nlp import notebook
from lit_nlp.examples.datasets import glue
from lit_nlp.examples.models import glue_models
sst_data = glue.SST2Data('validation')
sst_data.spec()
sst_data.examples[:10]
Explanation: Load data
LIT's Dataset classes are just lists of records, plus spec information to describe each field.
End of explanation
pd.DataFrame(sst_data.examples)
Explanation: You can easily convert this to tabular form, too:
End of explanation
# Fetch the trained model weights and load the model to analyze
!wget https://storage.googleapis.com/what-if-tool-resources/lit-models/sst2_tiny.tar.gz
!mkdir sst2_tiny
!tar -xvf sst2_tiny.tar.gz -C sst2_tiny
sentiment_model = glue_models.SST2Model('./sst2_tiny')
sentiment_model.input_spec(), sentiment_model.output_spec()
Explanation: Load a model and run inference
LIT's Model class defines a predict() function to perform inference. The input_spec() describes the expected inputs (it should be a subset of the dataset fields), and output_spec() describes the output.
End of explanation
preds = list(sentiment_model.predict(sst_data.examples[:10]))
pd.DataFrame(preds)
Explanation: There's a lot of fields in the output spec, since this model returns embeddings, gradients, attention, and more. We can view it using Pandas to avoid too much clutter:
End of explanation
labels = sentiment_model.output_spec()['probas'].vocab
pd.DataFrame([p['probas'] for p in preds], columns=pd.Index(labels, name='label'))
Explanation: If we just want the predicted probabilites for each class, we can look at the probas field:
End of explanation
from lit_nlp.components import lime_explainer
lime = lime_explainer.LIME()
lime_results = lime.run(sst_data.examples[:1], sentiment_model, sst_data)[0]
lime_results
# Again, pretty-print output with Pandas. The SalienceMap object is just a dataclass defined using attr.s.
pd.DataFrame(attr.asdict(lime_results['sentence']))
from lit_nlp.components import gradient_maps
ig = gradient_maps.IntegratedGradients()
ig_results = ig.run(sst_data.examples[:1], sentiment_model, sst_data)[0]
ig_results
# Again, pretty-print output with Pandas. The SalienceMap object is just a dataclass defined using attr.s.
pd.DataFrame(attr.asdict(ig_results['token_grad_sentence']))
Explanation: Salience methods
We can use different interpretability components as well. Here's an example running LIME to get a salience map. The output has entries for each input field, though here that's just one field named "sentence":
End of explanation
from lit_nlp.components import metrics
classification_metrics = metrics.MulticlassMetrics()
classification_metrics.run(sst_data.examples[:100], sentiment_model, sst_data)
Explanation: Metrics
We can also compute metrics. The metrics components (via the SimpleMetrics API) will automatically detect compatible fields marked by the parent attribute - in this case, our model's probas field that should be scored against label in the input.
End of explanation
from lit_nlp.components import scrambler
sc = scrambler.Scrambler()
sc_in = sst_data.examples[:5]
sc_out = sc.generate_all(sc_in, model=None, dataset=sst_data)
# The output is a list-of-lists, generated from each original example.
sc_out
# Format as a flat table for display, including original sentences
import itertools
for ex_in, exs_out in zip(sc_in, sc_out):
  for ex_out in exs_out:
    ex_out['original_sentence'] = ex_in['sentence']
pd.DataFrame(itertools.chain.from_iterable(sc_out), columns=['original_sentence', 'sentence', 'label'])
Explanation: Generators
We can use counterfactual generators as well. Here's an example with a generator that simply scrambles words in a text segment.
End of explanation
widget = notebook.LitWidget(models={'sentiment': sentiment_model}, 
                            datasets={'sst2': sst_data}, 
                            height=800)
widget.render()
Explanation: Running the LIT UI
Of course, you can always still use these components in the LIT UI, without leaving Colab.
End of explanation
<END_TASK>