<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Using spaCy for Text Preprocessing
Date
Step1:  1. What is spaCy
spaCy is a free, open-source library for NLP in Python
Providing optimized pipelines for taking models to production, i.e., facilitating integration with other components, and scalability
Current version (spaCy v3, released in Feb 2021) comes with pre-trained deep learning models, including state-of-the-art transformers, trained over huge data sets of documents
Available models can be fine-tuned to better fit specific document collections characteristics
SpaCy is intended to be used as a component of a more complex system, not as final application itself, i.e., it cannot be directly used to implement a chatbot system, a sentiment analyzer, etc ... but it provides a lot of tools that are easy to integrate for taking such systems into production.
1.1. SpaCy Features
spaCy provides a lot of features similar to those we have already discussed for the NLTK library.
spaCy makes it very easy to concatenate several of these operations
Step2:  In order to use a specific model you need to download it first. If working locally, you will need to download the model just once; however, in the cloud your environment resets and you will need to download the model on each session.
For this tutorial we will use an English model of medium size, the smallest model that incorporates word embeddings. For a complete list of available models, please refer to the spaCy website.
Step3:  2.2. Obtaining Model Info
You can retrieve the most relevant information about available language models using the following command
spacy.info('model_name')
Note that you can only apply this command on models that have already been downloades. Otherwise, an exception is thrown.
Exercise 1
Step4:  3. Spacy Data Structures and Processing Pipelines
3.1. Introduction and basic usage
Processing texts with spaCy is really easy. You just need to load the model, and pass any text you wish to process. SpaCy will execute a series of transformations (a pipeline) and return a Doc object. The returned object has all information extracted from the original text, and provides a number of features to facilitate accessing the desired information.
<figure>
<center>
<img src='https
Step5:  Note how in the example we could easily access all lemmas and entities found by iterating over the document (variable doc) itself or over its entitities (doc.ents)
3.2. Architecture
Central data structures
Step6:  Exercise 3
Step7:  3.3. Usual Pipelines Components and Annotations
4. Linguistic Features
In this Section we will review a set of liguistic features provided by most spaCy pretrained pipelines. We will focus mainly on pipeline components that are relevant to build Bag of Words (BoW) representations of text.
4.1. Tokenizer
4.1.1. Word Tokenization
The Tokenizer is always the first component of the spaCy pretrained pipelines.
It has the important role of producing a Doc object out of a text string
It first splits the string using blank spaces
Then tokens are processed sequentially from left to right performing two operations
First, language-specific rules and exceptions are applied (e.g., in English "don't" is splitted into two separate tokens, but U.K. is kept as one token)
Second, prefixes or suffixes are identified. This is relevant to separate punctuation marks from the main tokens
It is important to note that tokenization rules, as well as exceptions are language specific. This means you need to make sure that the languages of the text and the selected Tokenizer match, otherwise you could get unexpected results.
Once the Doc object has been created, you can easily iterate over the identified tokens. Note also that the original text is preserved. You can access the string representation of Doc, Span, Token and even Lexeme objects by using the text attribute.
Step8:  Unlike other spaCy components, the Tokenizer is not a statistical model. A finite set of rules and exceptions are encoded. If you wish to modify its behavior, you cannot retrain the component using labeled data. Instead, you would need to extend the list of rules and exceptions.
The following example adds an exception to expand word MUSD into tokens MUSD. Newly added exceptions are always applied after previous rules. Note also that exceptions must preserve the original text. Otherwise, an exception will be raised.
Step9:  4.1.2. Sentence Tokenization
Note that with the Doc object you can also iterate over sentences
Step10:  However, be aware that sentences are not identified by the Tokenizer element we have just described, but sentence tokenization is carried out instead as a subproduct of the dependency extraction component, that we will shortly review.
This can be a problem for multilingual documents, since all components of the previously used pipeline assumes an input in English language. In this case, what we normally do is
Step11:  Example
Step12:  4.2. Part of Speech Tagging
The next component available in most spaCy pipelines is a POS tagger. The role of this component is to predict the part of speech of every detected token. This is where the statistical models come in. These components have been trained using machine learning methods over a large set of annotated data. If necessary, the models can be fine-tuned providing additional training data. This can be sometimes helpful when working in specialized domains.
The attributes calculated by all pipeline components after the Tokenizer are added as additional attributes to each of the patterns.
POS tags can be accessed as Token.pos_ (POS strings are hashed, and the underscore facilitates accessing the readable format of the variable).
Finer tags can be accessed as Token.tag_
The following code fragment allows us to represent the calculated POS for each token in the provided text.
Step13:  Exercise 5
Step14:  4.3. Dependency Parser
The dependency parser aims at syntatic analysis of the text. It is the component that identifies the relation among the tokens in the given text, e.g., noun-chunks, verb objects, dependent clauses, etc.
Since our goal here is to use the pipeline to obtain BoW representation of the documents, we will not go deeper in the description of this component. If you are interested in learning more about dependy parsing in spaCy, you can check the official documentation
Since we will not be using this information, we can disable the component in the pipeline. This will also speed up document preprocessing.
Step15:  4.4. Named Entity Recognition
According to spaCy documentation
Step16:  Discussion
Step17:  4.5. Lemmatization
English lemmatizer in spaCy consists of the following elements
Step18:  4.6. Other Annotations
Exercise 6
<ASSISTANT_TASK:>
Python Code:
# Common imports 
import numpy as np
import pandas as pd
import zipfile as zp
from termcolor import colored
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina' 
#To wrap long text lines
from IPython.display import HTML, display
def set_css():
  display(HTML('''
  <style>
    pre {
        white-space: pre-wrap;
    }
  </style>
  '''))
get_ipython().events.register('pre_run_cell', set_css)
#For fancy table Display
%load_ext google.colab.data_table
Explanation: Using spaCy for Text Preprocessing
Date: Mar 16, 2021
Author: Jerónimo Arenas-García (jeronimo.arenas@uc3m.es)
Version 1.0
This notebook is based on the spaCy 101 course and documentation available at the spaCy website.
Our goal here is to present a basic overview of spacy that covers the elements necessary for implementing the preprocessing pipelines that we will need for obtaining the Bag of Word Representation of the document.
A more Advanced Tutorial by Ines Montani, one of the main developers of the library, is proposed for further study of interested students. In that tutorial, you can learn how to use spaCy matching functionalities, or how to retrain neural network models using your own training data.
End of explanation
!pip install --upgrade spacy
import spacy
Explanation: 1. What is spaCy
spaCy is a free, open-source library for NLP in Python
Providing optimized pipelines for taking models to production, i.e., facilitating integration with other components, and scalability
Current version (spaCy v3, released in Feb 2021) comes with pre-trained deep learning models, including state-of-the-art transformers, trained over huge data sets of documents
Available models can be fine-tuned to better fit specific document collections characteristics
SpaCy is intended to be used as a component of a more complex system, not as final application itself, i.e., it cannot be directly used to implement a chatbot system, a sentiment analyzer, etc ... but it provides a lot of tools that are easy to integrate for taking such systems into production.
1.1. SpaCy Features
spaCy provides a lot of features similar to those we have already discussed for the NLTK library.
spaCy makes it very easy to concatenate several of these operations:
Pipelines allow to concatenate a number of components to carry out the desired preprocessing tasks
Specific components can be enabled or disabled if necessary
It is possible to add ad-hoc components
Other developers are providing specific components ready to use in spaCy, e.g., spaCy langdetect is a wrapper for the langdetect library for language classification.
1.2. Language and Models
spaCy v3 comes with 55 pre-trained models for 17 languages. Details and installation instructions can be found here.
For most of these languages, three models are available, e.g.:
   - en_core_web_sm
   - en_core_web_md
   - en_core_web_lg
[Convention for the model name is language_core_source_size]
These models are optimized for CPU usage, but they still incorporate neural networks for certain components. 
Medium and Large models come with word-embeddings available, while small model does not
The larger the model, the higher the accuracy, but also the longer it takes to analyze a text fragment. I.e., accuracy comes at the cost of larger networks and, therefore, more computation
Accuracy of pipeline components are provided for specific annotated datasets
For English, Spanish, French, German, and Chinese, a fourth model (e.g. en_core_web_trf) based on transformers is also provided. These models are optimized to run over a GPU
1.3. Performance
--- WPS: Words per second
2. Using spaCy in Google Colab
2.1. Installing spaCy and loading language models
You can check that Google Colab already comes with spaCy v2 preinstalled. However, since in this notebook we will be using the new v3 release, you will need to upgrade to the latest available version
End of explanation
!python -m spacy download en_core_web_md
Explanation: In order to use a specific model you need to download it first. If working locally, you will need to download the model just once; however, in the cloud your environment resets and you will need to download the model on each session.
For this tutorial we will use an English model of medium size, the smallest model that incorporates word embeddings. For a complete list of available models, please refer to the spaCy website.
End of explanation
spacy.explain('HYPH')
spacy.info('en_core_web_md')
Explanation: 2.2. Obtaining Model Info
You can retrieve the most relevant information about available language models using the following command
spacy.info('model_name')
Note that you can only apply this command on models that have already been downloades. Otherwise, an exception is thrown.
Exercise 1: Run the following command and find the information related to
   - Components included in the pipeline
   - Are all components enabled?
   - How many types of entities can be recognized by the corresponding component?
   - What Part-of-Speech elements can you recognize?
   - What is the dimension of the word-embeddings incorporated in the model?
Detailed information about some specific components of the pipeline, as well as how they can be used, will be studied in the next sections.
End of explanation
text = 'Modern condensed matter physics research has produced novel materials with fundamental properties that underpin a remarkable number of cutting-edge technologies. It is now generally accepted that novel materials are necessary for critical advances in technologies and whoever discovers novel materials generally controls the science and technology of the future. Transition metal oxides have attracted enormous interest within both the basic and applied science communities.  However, for many decades, the overwhelming balance of effort was focused on the 3d-elements (such as iron, copper, etc.) and their compounds; the heavier 4d- and 5d-elements (such as ruthenium, iridium, etc., which constitute two thirds of the d-elements listed in the Periodic Table) and their compounds have been largely ignored until recently. The principal investigator seeks to discover novel materials containing 4d- and/or 5d-elements and understand how they offer wide-ranging opportunities for the discovery of new physics and, ultimately, new device paradigms. This project also provides rigorous training to all students involved, focusing on synthesis and characterization techniques covering a broad spectrum of materials and experimental probes available in the principal investigator\'s laboratory. Technical Abstract: Physics driven by spin-orbit interactions is among the most important topics in contemporary condensed matter physics. Since the spin-orbit interaction is comparable to the on-site Coulomb and other relevant interactions, it creates a unique balance between competing interactions that drive complex behaviors and exotic states not observed in other materials. The project encompasses a systematic effort to elucidate physics of novel phenomena in spin-orbit-coupled and correlated materials and a rigorous search for new materials having exotic ground states. This project focuses on the following areas: (1) Novel phenomena at high pressures and high magnetic fields, (2) Unusual correlations between the insulating gap and magnetic transition in iridates and ruthenates, (3) Exotic metallic and superconducting states in iridates, (4) Mott insulators with "intermediate-strength" spin-orbit interaction and other competing energies, and (5) Single-crystal synthesis and search for novel materials. The principal investigator is one of a few key pioneers who have initiated seminal studies on iridates and, before that, ruthenates, and has comprehensive facilities and proven expertise for single-crystal synthesis and wide-ranging studies of structural, transport, magnetic, thermal and dielectric properties as functions of temperature, magnetic field, pressure and doping.'
print(text)
nlp = spacy.load('en_core_web_md')
doc = nlp(text)
print(colored('============= Original Text =============', 'blue'))
print(doc)
print(colored('\n============= Lemmatized Text =============', 'red'))
print(' '.join([tk.lemma_ for tk in doc]))
print(colored('\n============= Entities Found =============', 'green'))
print('\n'.join([ent.text for ent in doc.ents]))
doc[3:10].vector
Explanation: 3. Spacy Data Structures and Processing Pipelines
3.1. Introduction and basic usage
Processing texts with spaCy is really easy. You just need to load the model, and pass any text you wish to process. SpaCy will execute a series of transformations (a pipeline) and return a Doc object. The returned object has all information extracted from the original text, and provides a number of features to facilitate accessing the desired information.
<figure>
<center>
<img src='https://spacy.io/pipeline-fde48da9b43661abcdf62ab70a546d71.svg' width="800"></img>
<figcaption>Source: https://spacy.io/pipeline-fde48da9b43661abcdf62ab70a546d71.svg</figcaption></center>
</figure>
End of explanation
#<SOL>
vec1 = nlp('structural, transport, magnetic, thermal and dielectric properties').vector
vec2 = nlp('temperature, magnetic field, pressure and doping').vector
vec3 = nlp('This project also provides rigorous training to all students involved').vector
print(np.linalg.norm(vec1-vec2))
print(np.linalg.norm(vec1-vec3))
print(np.linalg.norm(vec2-vec3))
#</SOL>
Explanation: Note how in the example we could easily access all lemmas and entities found by iterating over the document (variable doc) itself or over its entitities (doc.ents)
3.2. Architecture
Central data structures:
Language: is instantiated when loading the model, and contains the pipeline. Transforms text into spaCy documents.
Doc: Sequence of tokens with annotations. We can iterate over tokens, access individual tokens (doc[3]) or a span of tokens (doc[5:15]).
Vocab: Unique vocabulary associated to the language. Vocabulary is composed of Lexemes that are hashed and stored in the vocabulary with word vectors and attributes. This is memory efficient and assures a unique ground truth.
<figure>
<center>
<img src='https://spacy.io/architecture-415624fc7d149ec03f2736c4aa8b8f3c.svg' width="600"></img>
<figcaption>Source: https://spacy.io/architecture-415624fc7d149ec03f2736c4aa8b8f3c.svg</figcaption></center>
</figure>
The Tokenizer component of the pipeline is special, since this is where the Doc object is generated from the text. Subsequent pipeline components perform operations in place, obtanining new attributes that are stored as annotations in the tokens.
Pipeline components can be fine-tuned using annotated data
New components can be easily implemented and added to the Pipeline
Exercise 2:
   - Find the Spans associated to the following text fragments contained in the original text:
      * structural, transport, magnetic, thermal and dielectric properties
      * temperature, magnetic field, pressure and doping
      * This project also provides rigorous training to all students involved
   - Use command dir to examine what are the different methods and attributes of the Span object
   - Recover the vector representation associated to each of the previous strings
   - Compute the Euclidean distances between the selected Spans
--Hint: To compute Euclidean distances at this point, it can be convenient to use numpy function np.linalg.norm. Later in the notebook you will find that spaCy provides functions to carry out these calculations.
End of explanation
#<SOL>
vocab = []
for wd in nlp.vocab:
    vocab.append([wd.text, wd.orth, wd.is_stop])
vocab = pd.DataFrame(vocab, columns=['Term', 'Hash', 'Stopword'])
vocab
#</SOL>
#<SOL>
stopw = vocab[vocab['Stopword']==True]
stopw
#</SOL>
#<SOL>
print(len(nlp.vocab))
doc_aux = nlp('This is just a catalogue of some weird nomenclatures to increase the vocabulary size: USA, OMG, teeth, tooth. We can also include words which are in a different language or just typos: mecachis, guau, hause')
print(len(nlp.vocab))
#</SOL>
Explanation: Exercise 3: You can access all vocab elements as nlp.vocab. Each element of the vocabulary is known as a Lexeme
   - Use command dir to examine what are the different methods and attributes of Lexeme objects.
   - For each element in the vocabulary, print the text representation, the hash representation, and whether the term should be considered as a stopword or not.
   - Find all stopwords in the Vocabulary
   - Which is the current size of your vocabulary? Create an additional doc object from a text with words that have not been previously used, and check the new size of the vocabulary after processing the new document.
--Hint: For displaying the vocabulary in a convenient format, you can store the requested information in a Pandas DataFrame, and print the DataFrame instead
End of explanation
shortext = 'Natural Language Processing is a key component of many relevant Artificial Intelligence Applications.' \
           ' Libraries such as spaCy v3 make it simple to benefit from statistical NLP models based on neural networks.' \
           ' It is estimated that NLP market in the U.S. will grow to around 30000 MUSD during the next five years.' \
           ' I don\'t know how accurate this is, but a solid growth is guaranteed'
shortdoc = nlp(shortext)
print(colored('============= The original text information is still kept in the Doc object =============', 'blue'))
print(shortdoc)
print(colored('\n============= Identified Tokens =============', 'red'))
for token in shortdoc:
    print(token.text, end='\t\t')
#print('\t\t'.join([token.text for token in shortdoc]))
Explanation: 3.3. Usual Pipelines Components and Annotations
4. Linguistic Features
In this Section we will review a set of liguistic features provided by most spaCy pretrained pipelines. We will focus mainly on pipeline components that are relevant to build Bag of Words (BoW) representations of text.
4.1. Tokenizer
4.1.1. Word Tokenization
The Tokenizer is always the first component of the spaCy pretrained pipelines.
It has the important role of producing a Doc object out of a text string
It first splits the string using blank spaces
Then tokens are processed sequentially from left to right performing two operations
First, language-specific rules and exceptions are applied (e.g., in English "don't" is splitted into two separate tokens, but U.K. is kept as one token)
Second, prefixes or suffixes are identified. This is relevant to separate punctuation marks from the main tokens
It is important to note that tokenization rules, as well as exceptions are language specific. This means you need to make sure that the languages of the text and the selected Tokenizer match, otherwise you could get unexpected results.
Once the Doc object has been created, you can easily iterate over the identified tokens. Note also that the original text is preserved. You can access the string representation of Doc, Span, Token and even Lexeme objects by using the text attribute.
End of explanation
# Add special case rule
from spacy.symbols import ORTH
special_case = [{ORTH: "M"}, {ORTH: "USD"}]
nlp.tokenizer.add_special_case("MUSD", special_case)
shortdoc = nlp(shortext)
print(colored('============= The original text information is still kept in the Doc object =============', 'blue'))
print(shortdoc)
print(colored('\n============= Identified Tokens =============', 'red'))
for token in shortdoc:
    print(token.text, end='\t\t')
#print('\t\t'.join([token.text for token in shortdoc]))
Explanation: Unlike other spaCy components, the Tokenizer is not a statistical model. A finite set of rules and exceptions are encoded. If you wish to modify its behavior, you cannot retrain the component using labeled data. Instead, you would need to extend the list of rules and exceptions.
The following example adds an exception to expand word MUSD into tokens MUSD. Newly added exceptions are always applied after previous rules. Note also that exceptions must preserve the original text. Otherwise, an exception will be raised.
End of explanation
for sentence in shortdoc.sents:
    print(sentence.text)
Explanation: 4.1.2. Sentence Tokenization
Note that with the Doc object you can also iterate over sentences
End of explanation
!python -m spacy download xx_sent_ud_sm
!pip install --upgrade spacy_langdetect
multilingualtext = 'Natural Language Processing is a key component of many relevant Artificial Intelligence Applications.' \
           ' El Procesamiento de Lenguaje Natural es un componente de gran importancia en multitud de aplicaciones de la Inteligencia Artificial.' \
           ' Libraries such as spaCy v3 make it simple to benefit from statistical NLP models based on neural networks.' \
           ' SpaCy v3 y otras librerías similares hacen posible emplear métodos de NLP basados en redes neuronales de manera sencilla.' \
           ' It is estimated that NLP market in the U.S. will grow to around 30000 MUSD during the next five years.' \
           ' Se estima que el mercado del NLP en USA será de alrededor de 30.000 millones de dolares en cinco años.'
           
#<SOL>
english_text = 
spanish_text = 
#</SOL>
Explanation: However, be aware that sentences are not identified by the Tokenizer element we have just described, but sentence tokenization is carried out instead as a subproduct of the dependency extraction component, that we will shortly review.
This can be a problem for multilingual documents, since all components of the previously used pipeline assumes an input in English language. In this case, what we normally do is:
   - Split the document in sentences using a multilingual sentence tokenizer
   - Detect the language of each sentence
   - Use the appropriate pipeline for each sentence depending on its language
Exercise 4: Split the following paragraph into two variables english_text and spanish_text using multilingual sentence tokenizers and language detection libraries.
Sentence tokenization: If you opt to use spaCy, you can use the multilingual pipeline xx_sent_ud_sm which provides just a basic (rule-based) sentence tokenizer. You may also use NLTK library (from nltk.tokenize import sent_tokenize)
Language detection: You can use python library langdetect, or the pipeline component spacy-langdetect for spaCy, which is just a wrapper for the previous library
End of explanation
from spacy.language import Language
from spacy_langdetect import LanguageDetector
# Add LanguageDetector and assign it a string name
@Language.factory("language_detector")
def create_language_detector(nlp, name):
    return LanguageDetector(language_detection_function=None)
mult_nlp = spacy.load('xx_sent_ud_sm')
mult_nlp.add_pipe('language_detector', last=True)
mult_doc = mult_nlp(multilingualtext)
# document level language detection. Think of it like average language of the document!
print(colored('============= Document level language detection =============', 'blue'))
print(mult_doc._.language)
# sentence level language detection
print(colored('\n============= Sentence level language detection =============', 'red'))
for sent in mult_doc.sents:
    print(sent, sent._.language)
# English and Spanish Texts
print(colored('\n============= English sentences =============', 'green'))
english_text = ' '.join([sent.text for sent in mult_doc.sents if sent._.language['language']=='en'])
print(english_text)
print(colored('\n============= Spanish sentences =============', 'green'))
spanish_text = ' '.join([sent.text for sent in mult_doc.sents if sent._.language['language']=='es'])
print(spanish_text)
Explanation: Example: The following code fragment adapts the example provided in the documenation for spacy-langdetect to construct a new pipeline that concatenates xx_sent_ud_sm and spacy-langdetect. 
The new pipeline is then used to calculate variables english_text and spanish_text
End of explanation
text = 'Modern condensed matter physics research has produced novel materials with fundamental properties that underpin a remarkable number of cutting-edge technologies. It is now generally accepted that novel materials are necessary for critical advances in technologies and whoever discovers novel materials generally controls the science and technology of the future. Transition metal oxides have attracted enormous interest within both the basic and applied science communities.  However, for many decades, the overwhelming balance of effort was focused on the 3d-elements (such as iron, copper, etc.) and their compounds; the heavier 4d- and 5d-elements (such as ruthenium, iridium, etc., which constitute two thirds of the d-elements listed in the Periodic Table) and their compounds have been largely ignored until recently. The principal investigator seeks to discover novel materials containing 4d- and/or 5d-elements and understand how they offer wide-ranging opportunities for the discovery of new physics and, ultimately, new device paradigms. This project also provides rigorous training to all students involved, focusing on synthesis and characterization techniques covering a broad spectrum of materials and experimental probes available in the principal investigator\'s laboratory. Technical Abstract: Physics driven by spin-orbit interactions is among the most important topics in contemporary condensed matter physics. Since the spin-orbit interaction is comparable to the on-site Coulomb and other relevant interactions, it creates a unique balance between competing interactions that drive complex behaviors and exotic states not observed in other materials. The project encompasses a systematic effort to elucidate physics of novel phenomena in spin-orbit-coupled and correlated materials and a rigorous search for new materials having exotic ground states. This project focuses on the following areas: (1) Novel phenomena at high pressures and high magnetic fields, (2) Unusual correlations between the insulating gap and magnetic transition in iridates and ruthenates, (3) Exotic metallic and superconducting states in iridates, (4) Mott insulators with "intermediate-strength" spin-orbit interaction and other competing energies, and (5) Single-crystal synthesis and search for novel materials. The principal investigator is one of a few key pioneers who have initiated seminal studies on iridates and, before that, ruthenates, and has comprehensive facilities and proven expertise for single-crystal synthesis and wide-ranging studies of structural, transport, magnetic, thermal and dielectric properties as functions of temperature, magnetic field, pressure and doping.'
nlp = spacy.load('en_core_web_md')
doc = nlp(text)
df = pd.DataFrame([[token.text, token.pos_, token.tag_] for token in doc],
                  columns = ['Token', 'POS', 'TAG'])
df
Explanation: 4.2. Part of Speech Tagging
The next component available in most spaCy pipelines is a POS tagger. The role of this component is to predict the part of speech of every detected token. This is where the statistical models come in. These components have been trained using machine learning methods over a large set of annotated data. If necessary, the models can be fine-tuned providing additional training data. This can be sometimes helpful when working in specialized domains.
The attributes calculated by all pipeline components after the Tokenizer are added as additional attributes to each of the patterns.
POS tags can be accessed as Token.pos_ (POS strings are hashed, and the underscore facilitates accessing the readable format of the variable).
Finer tags can be accessed as Token.tag_
The following code fragment allows us to represent the calculated POS for each token in the provided text.
End of explanation
# Descriptions for POS values
#<SOL>
all_pos = set(df.POS.values.tolist())
pos_df = pd.DataFrame([[el, spacy.explain(el)] for el in all_pos], columns=['POS', 'Description'])
pos_df
#</SOL>
# Descriptions for TAGS values
#<SOL>
all_tag = set(df.TAG.values.tolist())
pos_df = pd.DataFrame([[el, spacy.explain(el)] for el in all_tag], columns=['TAG', 'Description'])
pos_df
#</SOL>
Explanation: Exercise 5: Use spaCy command spacy.explain() to obtain the descriptions of all POS and TAG values that you got for the previous text fragment. Avoid repetitions.
End of explanation
nlp.disable_pipe("parser")
#If you wish to completely remove the component from the pipeline, you can use the following command
#nlp.remove_pipe("parser")
Explanation: 4.3. Dependency Parser
The dependency parser aims at syntatic analysis of the text. It is the component that identifies the relation among the tokens in the given text, e.g., noun-chunks, verb objects, dependent clauses, etc.
Since our goal here is to use the pipeline to obtain BoW representation of the documents, we will not go deeper in the description of this component. If you are interested in learning more about dependy parsing in spaCy, you can check the official documentation
Since we will not be using this information, we can disable the component in the pipeline. This will also speed up document preprocessing.
End of explanation
doc = nlp(english_text)
df_ents = pd.DataFrame([[ent.text, ent.label_, spacy.explain(ent.label_)] for ent in doc.ents], columns=['Entity', 'Type', 'Description'])
df_ents
Explanation: 4.4. Named Entity Recognition
According to spaCy documentation:
spaCy features an extremely fast statistical entity recognition system, that assigns labels to contiguous spans of tokens. The default trained pipelines can indentify a variety of named and numeric entities, including companies, locations, organizations and products. You can add arbitrary classes to the entity recognition system, and update the model with new examples.
A named entity is a “real-world object” that’s assigned a name – for example, a person, a country, a product or a book title. spaCy can recognize various types of named entities in a document, by asking the model for a prediction.
Because models are statistical and strongly depend on the examples they were trained on, this doesn’t always work perfectly and might need some tuning later, depending on your use case
You can iterate over the entities found in the text using doc.ents, as illustrated in the following example that displays also the types of the entities found.
End of explanation
from spacy import displacy
wiki_text = 'Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist, widely acknowledged to be one of the greatest physicists of all time.' \
            ' Einstein is known widely for developing the theory of relativity, but he also made important contributions to the development of the theory of quantum mechanics.' \
            ' He received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect".' \
            ' Einstein was born in the German Empire, but moved to Switzerland in 1895, forsaking his German citizenship the following year.' \
            ' Einstein was awarded a PhD by the University of Zürich.' \
            ' On the eve of World War II, he endorsed a letter to President Franklin D. Roosevelt.'
wiki_doc = nlp(wiki_text)
displacy.render(wiki_doc, style="ent", jupyter=True, options={'distance': 90})
entypes = set([ent.label_ for ent in wiki_doc.ents])
df_ent = pd.DataFrame([[enttyp, spacy.explain(enttyp)] for enttyp in entypes], columns=['Entity type', 'Description'])
df_ent
Explanation: Discussion: You can check that the NER algorithm is not always very accurate, both with respect to detection of entities and entity type classification. Note, however, that for some other texts performance can be much better. The following example contains an excerpt from the Albert Einstein wikipedia web page.
How does NER accuracy in this example compares to the previous case?
What do you believe is the reason for this?
End of explanation
doc = nlp(text)
print(colored('============= Original text =============', 'blue'))
print(doc.text)
print(colored('\n============= Lemmas =============', 'red'))
print(' '.join([token.lemma_ for token in doc]))
Explanation: 4.5. Lemmatization
English lemmatizer in spaCy consists of the following elements:
Lookup tables
Rule-based lemmatizer, that exploits POS information
List-based exceptions aquired from WordNet
The annotation attribute can be easily accessed as Token.lemma_
End of explanation
mult_nlp = spacy.load('xx_sent_ud_sm')
mult_nlp.add_pipe('language_detector', last=True)
nlp = spacy.load('en_core_web_md')
nlp.disable_pipe('parser')
nlp.disable_pipe('ner')
valid_POS = set(['VERB', 'NOUN', 'ADJ', 'PROPN'])
specific_stw = set(['relevant', 'simple', 'base'])
def text_preprocessing(rawtext):
    #<SOL>
    mult_doc = mult_nlp(rawtext)
    english_text = ' '.join([sent.text for sent in mult_doc.sents if sent._.language['language']=='en'])
    doc = nlp(english_text)
    lemmatized = ' '.join([token.lemma_ for token in doc 
                           if token.is_alpha
                           and token.pos_ in valid_POS
                           and not token.is_stop
                           and token.lemma_ not in specific_stw])
    return lemmatized
    #</SOL>
print(colored('============= Original text =============', 'blue'))
print(multilingualtext)
print(colored('\n============= Lemmatized text =============', 'red'))
print(text_preprocessing(multilingualtext))
Explanation: 4.6. Other Annotations
Exercise 6: Have a look at the available attributes and functions of spaCy tokens using Python dir command. 
Find out the significance of the following attributes: is_stop, is_alpha, is_digit, like_url, like_email, like_num, vector, and test your findings using text examples of your own. 
5. Final Implementation of a pre-processing pipeline
Exercise 7: Implement a function that takes a string object and outputs the lemmatized text, ready for calculating BoW representation. The function should carry out the following steps:
Sentence tokenization and filtering of non-English sentences
Tokenization
POS
Lemmatization
Keep only alphanumeric tokens
Keep nouns, verbs, and adjectives
Generic Stopword removal
Specific Stopword removal
End of explanation
<END_TASK>