<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Build your own NER Tagger
Named Entity Recognition (NER) , also known as entity chunking/extraction , is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.
There are various off the shelf solutions which offer capabilites to perform named entity extraction (some of which we discussed in the previous units). Yet there are times when the requirements are beyond the capabilities of off-the-shelf classifiers.
In this notebook, we will go through an exercise to build our own NER using Conditional Random Fields.
We would be utilizing sklearn_crfsuite to develop our NER.
Load Dataset
Named Entity Recognition is a sequence modeling problem at it's core. It is more related to classification class of problems where in we need a labeled dataset to train a classifier. 
There are various labeled datasets for NER class of problems. We would be utilizing a pre-processed version of GMB(Groningen Meaning Bank) corpus for this notebook. The preprocessed version is availble at the following link 
Step1:  We have 47959 sentences that contain 35178 unique words.
These sentences have a total of 42 unique POS tags and 17 unique NER tags in total.
Tag Distribution
The GMB dataset utilizes IOB tagging or Inside, Outside Beginning. IOB is a common tagging format for tagging tokens which we have discussed earlier. To refresh your memory
Step2:  Conditional Random Fields
As mentioned above, NER belongs to sequence modeling class of problems. There are different algorithms to tackle sequence modeling, CRF or Conditional Random Fields are one such example. CRFs are proven to perform extremely well on NER and related domains. In this notebook, we will attempt at developing our own NER based on CRFs.
Question
Step3:  Prepare Train and Test Datasets
Step4:  Building Models with sklearn-crfsuite
sklearn-crfsuite is a thin CRFsuite (python-crfsuite) wrapper which provides scikit-learn-compatible sklearn_crfsuite.CRF estimator
Step5:  Train the model!
Train the model using the default configurations mentioned in the sklearn-crfsuite API docs
algorithm
Step6:  Use the following to load our pre-trained model if training above takes a lot of time
Step7:  Model Evaluation
Let's evaluate our model performance for NER Tagging on the test data now!
Try playing around with the following cells and observe the overall model performance.
We use standard classification metrics like precision, recall and f1-score
Step9:  We have intentially left out the Others tag to understand the performance of model on the remaining tags. The above evaluation statistics showcase a model which seems to have learnt the transitions quite well giving us an overall F1-score of 85%!
We can achieve even better results by fine tuning the feature engineering step along with hyper-parameter tuning.
End-to-End NER Tagger with trained NER Model
There is no fun (or value!) if we cannot use our model to tag new sentences in the future assuming we would want to put this model in production. Let's try and build an end-to-end workflow to perform NER Tagging on our sample document. First we perform NER tagging with SpaCy to remind you how it looks like.
Prepare Sample Document
Step10:  NER Tagging with SpaCy
Step11:  Pipeline Step 1
Tokenize Text
POS Tagging
Step12:  Pipeline Step 2
Extract Features from the POS tagged text document
Hint
Step13:  Pipeline Step 3
Use the CRF Model crf to predict on the features
Step14:  Pipeline Step 4
Combine text tokens with NER Tags
Retrieve relevant named entities from NER Tags
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
df = pd.read_csv('ner_dataset.csv.gz', compression='gzip', encoding='ISO-8859-1')
df.info()
df.T
df = df.fillna(method='ffill')
df.info()
df.T
df['Sentence #'].nunique(), df.Word.nunique(), df.POS.nunique(), df.Tag.nunique()
Explanation: Build your own NER Tagger
Named Entity Recognition (NER) , also known as entity chunking/extraction , is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.
There are various off the shelf solutions which offer capabilites to perform named entity extraction (some of which we discussed in the previous units). Yet there are times when the requirements are beyond the capabilities of off-the-shelf classifiers.
In this notebook, we will go through an exercise to build our own NER using Conditional Random Fields.
We would be utilizing sklearn_crfsuite to develop our NER.
Load Dataset
Named Entity Recognition is a sequence modeling problem at it's core. It is more related to classification class of problems where in we need a labeled dataset to train a classifier. 
There are various labeled datasets for NER class of problems. We would be utilizing a pre-processed version of GMB(Groningen Meaning Bank) corpus for this notebook. The preprocessed version is availble at the following link : kaggle/ner
We have provided the dataset in the code repository itself using some intelligent compression and you can access it directly from pandas as follows.
End of explanation
df.Tag.value_counts()
Explanation: We have 47959 sentences that contain 35178 unique words.
These sentences have a total of 42 unique POS tags and 17 unique NER tags in total.
Tag Distribution
The GMB dataset utilizes IOB tagging or Inside, Outside Beginning. IOB is a common tagging format for tagging tokens which we have discussed earlier. To refresh your memory:
I- prefix before a tag indicates that the tag is inside a chunk.
B- prefix before a tag indicates that the tag is the beginning of a chunk.
O-  tag indicates that a token belongs to no chunk (outside).
The tags in this dataset are explained as follows:
geo = Geographical Entity
org = Organization
per = Person
gpe = Geopolitical Entity
tim = Time indicator
art = Artifact
eve = Event
nat = Natural Phenomenon
Anything outside these classes is termed as other, denoted as O. 
The following output shows the unbalanced distribution of different tags in the dataset
End of explanation
def word2features(sent, i):
    word = sent[i][0]
    postag = sent[i][1]
    features = {
        'bias': 1.0,
        'word.lower()': word.lower(),
        'word[-3:]': word[-3:],
        'word[-2:]': word[-2:],
        'word.isupper()': word.isupper(),
        'word.istitle()': word.istitle(),
        'word.isdigit()': word.isdigit(),
        'postag': postag,
        'postag[:2]': postag[:2],
    }
    if i > 0:
        word1 = sent[i-1][0]
        postag1 = sent[i-1][1]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.istitle()': word1.istitle(),
            '-1:word.isupper()': word1.isupper(),
            '-1:postag': postag1,
            '-1:postag[:2]': postag1[:2],
        })
    else:
        features['BOS'] = True
    if i < len(sent)-1:
        word1 = sent[i+1][0]
        postag1 = sent[i+1][1]
        features.update({
            '+1:word.lower()': word1.lower(),
            '+1:word.istitle()': word1.istitle(),
            '+1:word.isupper()': word1.isupper(),
            '+1:postag': postag1,
            '+1:postag[:2]': postag1[:2],
        })
    else:
        features['EOS'] = True
    return features
def sent2features(sent):
    return [word2features(sent, i) for i in range(len(sent))]
def sent2labels(sent):
    return [label for token, postag, label in sent]
agg_func = lambda s: [(w, p, t) for w, p, t in zip(s['Word'].values.tolist(), 
                                                   s['POS'].values.tolist(), 
                                                   s['Tag'].values.tolist())]
grouped_df = df.groupby('Sentence #').apply(agg_func)
print(grouped_df[grouped_df.index == 'Sentence: 1'].values)
grouped_df.shape
sentences = [s for s in grouped_df]
sentences[0]
sent2features(sentences[0][5:7])
sent2labels(sentences[0][5:7])
Explanation: Conditional Random Fields
As mentioned above, NER belongs to sequence modeling class of problems. There are different algorithms to tackle sequence modeling, CRF or Conditional Random Fields are one such example. CRFs are proven to perform extremely well on NER and related domains. In this notebook, we will attempt at developing our own NER based on CRFs.
Question: What is a CRF and how does it work?
Wikipedia :  CRF is an undirected graphical model whose nodes can be divided into exactly two disjoint sets $X$ and $Y$, the observed and output variables, respectively; the conditional distribution $p(Y|X)$ is then modeled.
For more details, checkout the paper Conditional Random Fields: Probabilistic Models
for Segmenting and Labeling Sequence Data
Prepare Data
CRF trains upon sequence of input data to learn transitions from one state (label) to another. 
To enable such an algorithm, we need to define features which take into account different transitions. 
In the function word2features() below, we transform each word into a feature dictionary depicting the following attributes or features:
lower case of word
suffix containing last 3 characters
suffix containing last 2 characters
flags to determine upper-case, title-case, numeric data and POS tag
We also attach attributes related to previous and next words or tags to determine beginning of sentence (BOS) or end of sentence (EOS)
End of explanation
from sklearn.model_selection import train_test_split
import numpy as np
X = np.array([sent2features(s) for s in sentences])
y = np.array([sent2labels(s) for s in sentences])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
X_train.shape, X_test.shape
Explanation: Prepare Train and Test Datasets
End of explanation
!pip install sklearn-crfsuite
Explanation: Building Models with sklearn-crfsuite
sklearn-crfsuite is a thin CRFsuite (python-crfsuite) wrapper which provides scikit-learn-compatible sklearn_crfsuite.CRF estimator: you can use e.g. scikit-learn model selection utilities (cross-validation, hyperparameter optimization) with it, or save/load CRF models using joblib.
End of explanation
import sklearn_crfsuite
crf = sklearn_crfsuite.CRF(algorithm='lbfgs',
                           c1=0.1,
                           c2=0.1,
                           max_iterations=100,
                           all_possible_transitions=True,
                           verbose=True)
crf.fit(X_train, y_train)
Explanation: Train the model!
Train the model using the default configurations mentioned in the sklearn-crfsuite API docs
algorithm: the training algorithm. We use L-BFGS for gradient descent for optimization and getting model parameters
c1: Coefficient for Lasso (L1) regularization
c2: Coefficient for Ridge (L2) regularization
all_possible_transitions: Specify whether CRFsuite generates transition features that do not even occur in the training data
Note: If the model is taking too long to train, you can load up the pre-trained model using the code after the training cells and use that for predictions.
End of explanation
from sklearn.externals import joblib
#joblib.dump(crf, 'ner_model.pkl')
crf = joblib.load('ner_model.pkl')
Explanation: Use the following to load our pre-trained model if training above takes a lot of time
End of explanation
y_pred = crf.predict(X_test)
print(y_pred[0])
print(y_test[0])
from sklearn_crfsuite import metrics as crf_metrics
labels = list(crf.classes_)
labels.remove('O')
print(crf_metrics.flat_classification_report(y_test, y_pred, labels=labels))
Explanation: Model Evaluation
Let's evaluate our model performance for NER Tagging on the test data now!
Try playing around with the following cells and observe the overall model performance.
We use standard classification metrics like precision, recall and f1-score
End of explanation
import re
text = Three more countries have joined an “international grand committee” of parliaments, adding to calls for 
Facebook’s boss, Mark Zuckerberg, to give evidence on misinformation to the coalition. Brazil, Latvia and Singapore 
bring the total to eight different parliaments across the world, with plans to send representatives to London on 27 
November with the intention of hearing from Zuckerberg. Since the Cambridge Analytica scandal broke, the Facebook chief 
has only appeared in front of two legislatures: the American Senate and House of Representatives, and the European parliament. 
Facebook has consistently rebuffed attempts from others, including the UK and Canadian parliaments, to hear from Zuckerberg. 
He added that an article in the New York Times on Thursday, in which the paper alleged a pattern of behaviour from Facebook 
to “delay, deny and deflect” negative news stories, “raises further questions about how recent data breaches were allegedly 
dealt with within Facebook.”
text = re.sub(r'\n', '', text)
text
Explanation: We have intentially left out the Others tag to understand the performance of model on the remaining tags. The above evaluation statistics showcase a model which seems to have learnt the transitions quite well giving us an overall F1-score of 85%!
We can achieve even better results by fine tuning the feature engineering step along with hyper-parameter tuning.
End-to-End NER Tagger with trained NER Model
There is no fun (or value!) if we cannot use our model to tag new sentences in the future assuming we would want to put this model in production. Let's try and build an end-to-end workflow to perform NER Tagging on our sample document. First we perform NER tagging with SpaCy to remind you how it looks like.
Prepare Sample Document
End of explanation
import spacy
from spacy import displacy
nlp = spacy.load('en')
text_nlp = nlp(text)
displacy.render(text_nlp, style='ent', jupyter=True)
Explanation: NER Tagging with SpaCy
End of explanation
import nltk
text_tokens = nltk.word_tokenize(text)
text_pos = nltk.pos_tag(text_tokens)
text_pos[:10]
Explanation: Pipeline Step 1
Tokenize Text
POS Tagging
End of explanation
features = [sent2features(text_pos)]
features[0][0]
Explanation: Pipeline Step 2
Extract Features from the POS tagged text document
Hint: Use sent2features
End of explanation
labels = crf.predict(features)
doc_labels = labels[0]
doc_labels[10:20]
Explanation: Pipeline Step 3
Use the CRF Model crf to predict on the features
End of explanation
text_ner = [(token, tag) for token, tag in zip(text_tokens, doc_labels)]
print(text_ner)
named_entities = []
temp_entity_name = ''
temp_named_entity = None
for term, tag in text_ner:
    if tag != 'O':
        temp_entity_name = ' '.join([temp_entity_name, term]).strip()
        temp_named_entity = (temp_entity_name, tag)
    else:
        if temp_named_entity:
            named_entities.append(temp_named_entity)
            temp_entity_name = ''
            temp_named_entity = None
import pandas as pd
pd.DataFrame(named_entities, columns=['Entity', 'Tag'])
Explanation: Pipeline Step 4
Combine text tokens with NER Tags
Retrieve relevant named entities from NER Tags
End of explanation
<END_TASK>