<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  Author - @SauravMaheshkar
Install Dependencies
Install the latest version of the Trax Library.
Step2:  Introduction
Named-entity recognition (NER) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.
To evaluate the quality of a NER system's output, several measures have been defined. The usual measures are called Precision, recall, and F1 score. However, several issues remain in just how to calculate those values. State-of-the-art NER systems for English produce near-human performance. For example, the best system entering MUC-7 scored 93.39% of F-measure while human annotators scored 97.60% and 96.95%.
Importing Packages
Step3:  Pre-Processing
Loading the Dataset
Let's load the ner_dataset.csv file into a dataframe and see what it looks like
Step4:  Creating a Vocabulary File
We can see there's a column for the words in each sentence. Thus, we can extract this column using the .loc() and store it into a .txt file using the .savetext() function from numpy.
Step5:  Creating a Dictionary for Vocabulary
Here, we create a Dictionary for our vocabulary by reading through all the sentences in the dataset.
Step6:  Extracting Sentences from the Dataset
For extracting sentences from the dataset and creating (X,y) pairs for training.
Step7:  Making a Batch Generator
Here, we create a batch generator for training.
Step8:  Splitting into Test and Train
Step9:  Building the Model
The Reformer Model
In this notebook, we use the Reformer, which is a more efficient of Transformer that uses reversible layers and locality-sensitive hashing. You can read the original paper here. 
Locality-Sensitive Hashing
The biggest problem that one might encounter while using Transformers, for huge corpora is the handling of the attention layer. Reformer introduces Locality Sensitive Hashing to solve this problem, by computing a hash function that groups similar vectors together. Thus, a input sequence is rearranged to bring elements with the same hash together and then divide into segments(or chunks, buckets) to enable parallel processing. Thus, we can apply Attention to these chunks (rather than the whole input sequence) to reduce the computational load.
Reversible Layers
Using Locality Sensitive Hashing, we were able to solve the problem of computation but still we have a memory issue. Reformer implements a novel approach to solve this problem, by recomputing the input of each layer on-demand during back-propagation, rather than storing it in memory. This is accomplished by using Reversible Layers (activations from last layers are used to recover activations from any intermediate layer). 
Reversible layers store two sets of activations for each layer. 
One follows the standard procedure in which the activations are added as they pass through the network
The other set only captures the changes. Thus, if we run the network in reverse, we simply subtract the activations applied at each layer.
Model Architecture
We will perform the following steps
Step10:  Train the Model
<ASSISTANT_TASK:>
Python Code:
#@title
# Copyright 2020 Google LLC.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: <a href="https://colab.research.google.com/github/SauravMaheshkar/trax/blob/SauravMaheshkar-example-1/examples/NER_using_Reformer.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
End of explanation
!pip install -q -U trax
Explanation: Author - @SauravMaheshkar
Install Dependencies
Install the latest version of the Trax Library.
End of explanation
import trax # Our Main Library
from trax import layers as tl
import os # For os dependent functionalities
import numpy as np # For scientific computing
import pandas as pd # For basic data analysis
import random as rnd # For using random functions
Explanation: Introduction
Named-entity recognition (NER) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.
To evaluate the quality of a NER system's output, several measures have been defined. The usual measures are called Precision, recall, and F1 score. However, several issues remain in just how to calculate those values. State-of-the-art NER systems for English produce near-human performance. For example, the best system entering MUC-7 scored 93.39% of F-measure while human annotators scored 97.60% and 96.95%.
Importing Packages
End of explanation
data = pd.read_csv("/kaggle/input/entity-annotated-corpus/ner_dataset.csv",encoding = 'ISO-8859-1')
data = data.fillna(method = 'ffill')
data.head()
Explanation: Pre-Processing
Loading the Dataset
Let's load the ner_dataset.csv file into a dataframe and see what it looks like
End of explanation
## Extract the 'Word' column from the dataframe
words = data.loc[:, "Word"]
## Convert into a text file using the .savetxt() function
np.savetxt(r'words.txt', words.values, fmt="%s")
Explanation: Creating a Vocabulary File
We can see there's a column for the words in each sentence. Thus, we can extract this column using the .loc() and store it into a .txt file using the .savetext() function from numpy.
End of explanation
vocab = {}
with open('words.txt') as f:
  for i, l in enumerate(f.read().splitlines()):
    vocab[l] = i
  print("Number of words:", len(vocab))
  vocab['<PAD>'] = len(vocab)
Explanation: Creating a Dictionary for Vocabulary
Here, we create a Dictionary for our vocabulary by reading through all the sentences in the dataset.
End of explanation
class Get_sentence(object):
    def __init__(self,data):
        self.n_sent=1
        self.data = data
        agg_func = lambda s:[(w,p,t) for w,p,t in zip(s["Word"].values.tolist(),
                                                     s["POS"].values.tolist(),
                                                     s["Tag"].values.tolist())]
        self.grouped = self.data.groupby("Sentence #").apply(agg_func)
        self.sentences = [s for s in self.grouped]
getter = Get_sentence(data)
sentence = getter.sentences
words = list(set(data["Word"].values))
words_tag = list(set(data["Tag"].values))
word_idx = {w : i+1 for i ,w in enumerate(words)}
tag_idx =  {t : i for i ,t in enumerate(words_tag)}
X = [[word_idx[w[0]] for w in s] for s in sentence]
y = [[tag_idx[w[2]] for w in s] for s in sentence]
Explanation: Extracting Sentences from the Dataset
For extracting sentences from the dataset and creating (X,y) pairs for training.
End of explanation
def data_generator(batch_size, x, y,pad, shuffle=False, verbose=False):
    num_lines = len(x)
    lines_index = [*range(num_lines)]
    if shuffle:
        rnd.shuffle(lines_index)
    
    index = 0 
    while True:
        buffer_x = [0] * batch_size 
        buffer_y = [0] * batch_size 
        max_len = 0
        for i in range(batch_size):
            if index >= num_lines:
                index = 0
                if shuffle:
                    rnd.shuffle(lines_index)
            
            buffer_x[i] = x[lines_index[index]]
            buffer_y[i] = y[lines_index[index]]
            
            lenx = len(x[lines_index[index]])    
            if lenx > max_len:
                max_len = lenx                  
            
            index += 1
        X = np.full((batch_size, max_len), pad)
        Y = np.full((batch_size, max_len), pad)
        for i in range(batch_size):
            x_i = buffer_x[i]
            y_i = buffer_y[i]
            for j in range(len(x_i)):
                X[i, j] = x_i[j]
                Y[i, j] = y_i[j]
        if verbose: print("index=", index)
        yield((X,Y))
Explanation: Making a Batch Generator
Here, we create a batch generator for training.
End of explanation
from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test = train_test_split(X,y,test_size = 0.1,random_state=1)
Explanation: Splitting into Test and Train
End of explanation
def NERmodel(tags, vocab_size=35181, d_model = 50):
  model = tl.Serial(
    # tl.Embedding(vocab_size, d_model),
    trax.models.reformer.Reformer(vocab_size, d_model, ff_activation=tl.LogSoftmax),
    tl.Dense(tags),
    tl.LogSoftmax()
  )
  return model
model = NERmodel(tags = 17)
print(model)
Explanation: Building the Model
The Reformer Model
In this notebook, we use the Reformer, which is a more efficient of Transformer that uses reversible layers and locality-sensitive hashing. You can read the original paper here. 
Locality-Sensitive Hashing
The biggest problem that one might encounter while using Transformers, for huge corpora is the handling of the attention layer. Reformer introduces Locality Sensitive Hashing to solve this problem, by computing a hash function that groups similar vectors together. Thus, a input sequence is rearranged to bring elements with the same hash together and then divide into segments(or chunks, buckets) to enable parallel processing. Thus, we can apply Attention to these chunks (rather than the whole input sequence) to reduce the computational load.
Reversible Layers
Using Locality Sensitive Hashing, we were able to solve the problem of computation but still we have a memory issue. Reformer implements a novel approach to solve this problem, by recomputing the input of each layer on-demand during back-propagation, rather than storing it in memory. This is accomplished by using Reversible Layers (activations from last layers are used to recover activations from any intermediate layer). 
Reversible layers store two sets of activations for each layer. 
One follows the standard procedure in which the activations are added as they pass through the network
The other set only captures the changes. Thus, if we run the network in reverse, we simply subtract the activations applied at each layer.
Model Architecture
We will perform the following steps:
Use input tensors from our data generator
Produce Semantic entries from an Embedding Layer
Feed these into our Reformer Language model
Run the Output through a Linear Layer
Run these through a log softmax layer to get predicted classes
We use the:
tl.Serial(): Combinator that applies layers serially(by function composition). It's commonly used to construct deep networks. It uses stack semantics to manage data for its sublayers
tl.Embedding(): Initializes a trainable embedding layer that maps discrete tokens/ids to vectors
trax.models.reformer.Reformer(): Creates a Reversible Transformer encoder-decoder model.
tl.Dense(): Creates a Dense(fully-connected, affine) layer
tl.LogSoftmax(): Creates a layer that applies log softmax along one tensor axis.
End of explanation
from trax.supervised import training
rnd.seed(33)
batch_size = 64
train_generator = trax.data.inputs.add_loss_weights(
    data_generator(batch_size, x_train, y_train,vocab['<PAD>'], True),
    id_to_mask=vocab['<PAD>'])
eval_generator = trax.data.inputs.add_loss_weights(
    data_generator(batch_size, x_test, y_test,vocab['<PAD>'] ,True),
    id_to_mask=vocab['<PAD>'])
def train_model(model, train_generator, eval_generator, train_steps=1, output_dir='model'):
    train_task = training.TrainTask(
      train_generator,  
      loss_layer = tl.CrossEntropyLoss(), 
      optimizer = trax.optimizers.Adam(0.01), 
      n_steps_per_checkpoint=10
    )
    eval_task = training.EvalTask(
      labeled_data = eval_generator, 
      metrics = [tl.CrossEntropyLoss(), tl.Accuracy()], 
      n_eval_batches = 10 
    )
    training_loop = training.Loop(
        model, 
        train_task, 
        eval_tasks = eval_task, 
        output_dir = output_dir) 
    training_loop.run(n_steps = train_steps)
    return training_loop
train_steps = 100
training_loop = train_model(model, train_generator, eval_generator, train_steps)
Explanation: Train the Model
End of explanation
<END_TASK>