<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Vector-space models
Step1:  Contents
Overview
General set-up
Loading Transformer models
The basics of tokenizing
The basics of representations
The decontextualized approach
Basic example
Creating a full VSM
The aggregated approach
Some related work
Overview
Can we get good static representations of words from models (like BERT) that supply only contextual representations? On the one hand, contextual models are very successful across a wide range of tasks, in large part because they are trained for a long time on a lot of data. This should be a boon for VSMs as we've designed them so far. On the other hand, the goal of having static representations might seem to be at odds with how these models process examples and represent examples. Part of the point is to obtain different representations for words depending on the context in which they occur, and a hallmark of the training procedure is that it processes sequences rather than individual words.
Bommasani et al. (2020) make a significant step forward in our understanding of these issues. Ultimately, they arrive at a positive answer
Step2:  The transformers library does a lot of logging. To avoid ending up with a cluttered notebook, I am changing the logging level. You might want to skip this as you scale up to building production systems, since the logging is very good – it gives you a lot of insights into what the models and code are doing.
Step3:  Loading Transformer models
To start, let's get a feel for the basic API that transformers provides. The first step is specifying the pretrained parameters we'll be using
Step4:  There are lots other options for pretrained weights. See this Hugging Face directory.
Next, we specify a tokenizer and a model that match both each other and our choice of pretrained weights
Step5:  The basics of tokenizing
It's illuminating to see what the tokenizer does to example texts
Step6:  Simple tokenization
Step7:  The encode method maps individual strings to indices into the underlying embedding used by the model
Step8:  We can get a better feel for what these representations are like by mapping the indices back to "words"
Step9:  Those are all the essential ingredients for working with these parameters in Hugging Face. Of course, the library has a lot of other functionality, but the above suffices for our current application.
The basics of representations
To obtain the representations for a batch of examples, we use the forward method of the model, as follows
Step10:  The return value reps is a special transformers class that holds a lot of representations. If we want just the final output representations for each token, we use last_hidden_state
Step11:  The shape indicates that our batch has 1 example, with 10 tokens, and each token is represented by a vector of dimensionality 768. 
Aside
Step12:  The length 13 corresponds to the initial embedding layer (layer 0) and the 12 layers of this BERT model.
The final layer in hidden_states is identical to last_hidden_state
Step13:  The decontextualized approach
As discussed above, Bommasani et al. (2020) define and explore two general strategies for obtaining static representations for word using a model like BERT. The simpler one involves processing individual words and, where they correspond to multiple tokens, pooling those token representations into a single vector using an operation like mean.
Basic example
To begin to see what this is like in practice, we'll use the method vsm.hf_encode, which maps texts to their ids, taking care to use unk_token for texts that can't otherwise be processed by the model.
Where a word corresponds to just one token in the vocabulary, it will get mapped to a single id
Step14:  As we saw above, some words map to multiple tokens
Step15:  Next, the function vsm.hf_represent will map a batch of ids to their representations in a user-supplied model, at a specified layer in that model
Step16:  The shape here
Step17:  The function vsm.mean_pooling is simple torch.mean with axis=1. There are also predefined functions vsm.max_pooling, vsm.min_pooling, and vsm.last_pooling (representation for the final token).
Creating a full VSM
Now we want to scale the above process to a large vocabulary, so that we can create a full VSM. The function vsm.create_subword_pooling_vsm makes this easy. To start, we get the vocabulary from one of our count VSMs (all of which have the same vocabulary)
Step18:  And then we use vsm.create_subword_pooling_vsm
Step19:  The result, pooled_df, is a pd.DataFrame with its index given by vocab. This can be used directly in the word relatedness evaluations that are central the homework and associated bakeoff.
Step20:  This approach, and the associated code, should work generally for all Hugging Face Transformer-based models. Bommasani et al. (2020) provide a lot of guidance when it comes to how the model, the layer choice, and the pooling function interact.
The aggregated approach
The aggregated is also straightfoward to implement given the above tool. To start, we can create a map from vocabulary items into their sequences of ids
Step21:  Next, let's assume we have a corpus of texts that contain the words of interest
Step22:  The following embeds every corpus example, keeping layer=1 representations
Step23:  Finally, we define a convenience function for finding all the occurrences of a sublist in a larger list
Step24:  For example
Step25:  And here's an example using our vocab_ids and corpus
<ASSISTANT_TASK:>
Python Code:
__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Spring 2022"
Explanation: Vector-space models: Static representations from contextual models
End of explanation
import os
import pandas as pd
import torch
from transformers import BertModel, BertTokenizer
from transformers import RobertaModel, RobertaTokenizer
import utils
import vsm
DATA_HOME = os.path.join('data', 'vsmdata')
# Set all the random seeds for reproducibility:
utils.fix_random_seeds()
Explanation: Contents
Overview
General set-up
Loading Transformer models
The basics of tokenizing
The basics of representations
The decontextualized approach
Basic example
Creating a full VSM
The aggregated approach
Some related work
Overview
Can we get good static representations of words from models (like BERT) that supply only contextual representations? On the one hand, contextual models are very successful across a wide range of tasks, in large part because they are trained for a long time on a lot of data. This should be a boon for VSMs as we've designed them so far. On the other hand, the goal of having static representations might seem to be at odds with how these models process examples and represent examples. Part of the point is to obtain different representations for words depending on the context in which they occur, and a hallmark of the training procedure is that it processes sequences rather than individual words.
Bommasani et al. (2020) make a significant step forward in our understanding of these issues. Ultimately, they arrive at a positive answer: excellent static word representations can be obtained from contextual models. They explore two strategies for achieving this:
The decontextualized approach: just process individual words as though they were isolated texts. Where a word consists of multiple tokens in the model, pool them with a function like mean or max.
The aggregrated approach: process lots and lots of texts containing the words of interest. As before, pool sub-word tokens, and also pool across all the pooled representations.
As Bommasani et al. say, the decontextualized approach "presents an unnatural input" – these models were not trained on individual words, but rather on longer sequences, so the individual words are infrequent kinds of inputs at best (and unattested as far as the model is concerned if the special boundary tokens [CLS] and [SEP] are not included). However, in practice, Bommasani et al. achieve very impressive results with this approach on word similarity/relatedness tasks.
The aggregrated approach is even better, but it requires more work and involves more decisions relating to which texts are processed.
This notebook briefly explores both of these approaches, with the goal of making it easy for you to apply these methods in the associated homework and bakeoff.
General set-up
End of explanation
import logging
logger = logging.getLogger()
logger.level = logging.ERROR
Explanation: The transformers library does a lot of logging. To avoid ending up with a cluttered notebook, I am changing the logging level. You might want to skip this as you scale up to building production systems, since the logging is very good – it gives you a lot of insights into what the models and code are doing.
End of explanation
bert_weights_name = 'bert-base-cased'
Explanation: Loading Transformer models
To start, let's get a feel for the basic API that transformers provides. The first step is specifying the pretrained parameters we'll be using:
End of explanation
bert_tokenizer = BertTokenizer.from_pretrained(bert_weights_name)
bert_model = BertModel.from_pretrained(bert_weights_name)
Explanation: There are lots other options for pretrained weights. See this Hugging Face directory.
Next, we specify a tokenizer and a model that match both each other and our choice of pretrained weights:
End of explanation
example_text = "Bert knows Snuffleupagus"
Explanation: The basics of tokenizing
It's illuminating to see what the tokenizer does to example texts:
End of explanation
bert_tokenizer.tokenize(example_text)
Explanation: Simple tokenization:
End of explanation
ex_ids = bert_tokenizer.encode(example_text, add_special_tokens=True)
ex_ids
Explanation: The encode method maps individual strings to indices into the underlying embedding used by the model:
End of explanation
bert_tokenizer.convert_ids_to_tokens(ex_ids)
Explanation: We can get a better feel for what these representations are like by mapping the indices back to "words":
End of explanation
with torch.no_grad():
    reps = bert_model(torch.tensor([ex_ids]), output_hidden_states=True)
Explanation: Those are all the essential ingredients for working with these parameters in Hugging Face. Of course, the library has a lot of other functionality, but the above suffices for our current application.
The basics of representations
To obtain the representations for a batch of examples, we use the forward method of the model, as follows:
End of explanation
reps.last_hidden_state.shape
Explanation: The return value reps is a special transformers class that holds a lot of representations. If we want just the final output representations for each token, we use last_hidden_state:
End of explanation
len(reps.hidden_states)
Explanation: The shape indicates that our batch has 1 example, with 10 tokens, and each token is represented by a vector of dimensionality 768. 
Aside: Hugging Face transformers models also have a pooler_output value. For BERT, this corresponds to the output representation above the [CLS] token, which is often used as a summary representation for the entire sequence. However, we cannot use pooler_output in the current context, as transformers adds new randomized parameters on top of it, to facilitate fine-tuning. If we want the [CLS] representation, we need to use reps.last_hidden_state[:, 0].
Finally, if we want access to the output representations from each layer of the model, we use hidden_states. This will be None unless we set output_hidden_states=True when using the forward method, as above.
End of explanation
reps.hidden_states[-1].shape
torch.equal(reps.hidden_states[-1], reps.last_hidden_state)
Explanation: The length 13 corresponds to the initial embedding layer (layer 0) and the 12 layers of this BERT model.
The final layer in hidden_states is identical to last_hidden_state:
End of explanation
bert_tokenizer.tokenize('puppy')
vsm.hf_encode("puppy", bert_tokenizer)
Explanation: The decontextualized approach
As discussed above, Bommasani et al. (2020) define and explore two general strategies for obtaining static representations for word using a model like BERT. The simpler one involves processing individual words and, where they correspond to multiple tokens, pooling those token representations into a single vector using an operation like mean.
Basic example
To begin to see what this is like in practice, we'll use the method vsm.hf_encode, which maps texts to their ids, taking care to use unk_token for texts that can't otherwise be processed by the model.
Where a word corresponds to just one token in the vocabulary, it will get mapped to a single id:
End of explanation
bert_tokenizer.tokenize('snuffleupagus')
subtok_ids = vsm.hf_encode("snuffleupagus", bert_tokenizer)
subtok_ids
Explanation: As we saw above, some words map to multiple tokens:
End of explanation
subtok_reps = vsm.hf_represent(subtok_ids, bert_model, layer=-1)
subtok_reps.shape
Explanation: Next, the function vsm.hf_represent will map a batch of ids to their representations in a user-supplied model, at a specified layer in that model:
End of explanation
subtok_pooled = vsm.mean_pooling(subtok_reps)
subtok_pooled.shape
Explanation: The shape here: 1 example containing 6 (sub-word) tokens, each of dimension 768. With layer=-1, we obtain the final output repreentation from the entire model.
The final step is to pool together the 6 tokens. Here, we can use a variety of operations; Bommasani et al. 2020 find that mean is the best overall:
End of explanation
vsm_index = pd.read_csv(
    os.path.join(DATA_HOME, 'yelp_window5-scaled.csv.gz'),
    usecols=[0], index_col=0)
vocab = list(vsm_index.index)
vocab[: 5]
Explanation: The function vsm.mean_pooling is simple torch.mean with axis=1. There are also predefined functions vsm.max_pooling, vsm.min_pooling, and vsm.last_pooling (representation for the final token).
Creating a full VSM
Now we want to scale the above process to a large vocabulary, so that we can create a full VSM. The function vsm.create_subword_pooling_vsm makes this easy. To start, we get the vocabulary from one of our count VSMs (all of which have the same vocabulary):
End of explanation
%%time
pooled_df = vsm.create_subword_pooling_vsm(
    vocab, bert_tokenizer, bert_model, layer=1)
Explanation: And then we use vsm.create_subword_pooling_vsm:
End of explanation
pooled_df.shape
pooled_df.iloc[: 5, :5]
Explanation: The result, pooled_df, is a pd.DataFrame with its index given by vocab. This can be used directly in the word relatedness evaluations that are central the homework and associated bakeoff.
End of explanation
vocab_ids = {w: vsm.hf_encode(w, bert_tokenizer)[0] for w in vocab}
Explanation: This approach, and the associated code, should work generally for all Hugging Face Transformer-based models. Bommasani et al. (2020) provide a lot of guidance when it comes to how the model, the layer choice, and the pooling function interact.
The aggregated approach
The aggregated is also straightfoward to implement given the above tool. To start, we can create a map from vocabulary items into their sequences of ids:
End of explanation
corpus = [
    "This is a sailing example",
    "It's fun to go sailing!",
    "We should go sailing.",
    "I'd like to go sailing and sailing",
    "This is merely an example"]
Explanation: Next, let's assume we have a corpus of texts that contain the words of interest:
End of explanation
corpus_ids = [vsm.hf_encode(text, bert_tokenizer)
              for text in corpus]
corpus_reps = [vsm.hf_represent(ids, bert_model, layer=1)
               for ids in corpus_ids]
Explanation: The following embeds every corpus example, keeping layer=1 representations:
End of explanation
def find_sublist_indices(sublist, mainlist):
    indices = []
    length = len(sublist)
    for i in range(0, len(mainlist)-length+1):
        if mainlist[i: i+length] == sublist:
            indices.append((i, i+length))
    return indices
Explanation: Finally, we define a convenience function for finding all the occurrences of a sublist in a larger list:
End of explanation
find_sublist_indices([1,2], [1, 2, 3, 0, 1, 2, 3])
Explanation: For example:
End of explanation
sailing = vocab_ids['sailing']
sailing_reps = []
for ids, reps in zip(corpus_ids, corpus_reps):
    offsets = find_sublist_indices(sailing, ids.squeeze(0))
    for (start, end) in offsets:
        pooled = vsm.mean_pooling(reps[:, start: end])
        sailing_reps.append(pooled)
sailing_rep = torch.mean(torch.cat(sailing_reps), axis=0).squeeze(0)
sailing_rep.shape
Explanation: And here's an example using our vocab_ids and corpus:
End of explanation
<END_TASK>