<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to Python Jupyter
Welcome to Jupyter, through this interface I will be showing you the following
Step1:  Cleaning Up The Full Text
In order for better results from our analysis later we need to clean up the full text. 
This could be a project within itself and will differ item to item so for the intial run I have just set the full text to get lowered so 'Canada' and 'canada' aren't considered two different words.
I've also provided a basic regex that you can uncomment to strip everything other than words from the full text
Step2:  Basic Analysis
So now we have the item's full text we are going to use the Natural Language Toolkit to perform some analysis on it using NLTK.
NLTK is a Python Library for working with written language data. It is free and very well documented. Many areas we'll be covering are treated in more detail in the NLTK Book, available for free online from here.
Note
Step3:  Exploring Vocabulary
NLTK makes it really easy to get basic information about the size of a text and the complexity of its vocabulary.
len gives the number of symbols or 'tokens' in your text. This is the total number of words and items of punctuation.
set gives you a list of all the tokens in the text, without the duplicates.
Hence, len(set(fullText)) will give you the total number unique tokens. Remember this still includes punctuation. 
sorted() places items in the list into alphabetical order, with punctuation symbols and capitalised words first.
Number of characters
Step4:  Number of unique characters
Step5:  List of unique characters
Step6:  Get token count (words + symbols)
For our analysis, we want to break up the full text into words and punctuation, this step is called tokenization
Step7:  Unique Token Count
Step8:  Average number of times a word is used
We can investigate the lexical richness of a text. For example, by dividing the total number of words by the number of unique words, we can see the average number of times each word is used.
Step9:  Number of times a specific word is used
Step10:  Percentage of text that is a specific word
Step11:  Exploring Text
Concordance
Step12:  Words used similarly??
Step13:  Common contexts
Common contexts allow us to examine just the contexts that are shared by two or more words, such as valley and river.
Step14:  Longest words in the text
It is possible to select the longest words in a text, which may tell you something about its vocabulary and style
Step15:  Collocations
We can also find words that typically occur together, which tend to be very specific to a text or genre of texts.
Step16:  Graphing Data
Single Dispersion Plot
Step17:  Multiple Dispersion Plot
Step18:  Frequency distributions
<ASSISTANT_TASK:>
Python Code:
import json
import requests
apiResponse = requests.get('https://oc-index.library.ubc.ca/collections/bcbooks/items/1.0059569').json()
item = apiResponse['data']
fullText = item['FullText'][0]['value']
print(fullText)
Explanation: Introduction to Python Jupyter
Welcome to Jupyter, through this interface I will be showing you the following:
Python - A programming language that lets you work quickly. - Documentation
NLTK - Natural Languge Toolkit - a Python Library for working with written language data. -  Documentation
Open Collections API - Our "Application Programming Interface" which will allow you to import full text. - Documentation
Python is a great language for data analysis, more experienced programmers might want to use R, but Python is a nice entry point for everyone.
If you don't know Python, or any programming for that matter, please remain calm you won't need to do any programming throughout this talk, however if you do know Python you can feel free to edit any of the code and have your notebook update accordingly.
Getting the Full Text
To begin with we are just going to get one item from the Open Collections API and perform some analysis on that. Later on we will look at getting entire collections, and performing searches via the API. 
For our first item I have chosen:
https://open.library.ubc.ca/collections/bcbooks/items/1.0059569
The Open Collections API URL is: 
https://oc-index.library.ubc.ca 
So to access the item I have chosen via the API we would need to GET the data from:
https://oc-index.library.ubc.ca/collections/bcbooks/items/1.0059569
End of explanation
import re, string;
fullTextLower = fullText.lower()
cleanFullText = fullTextLower
### To strip everything other than words uncomment below ###
pattern = re.compile('[\W_]+')
cleanFullText = pattern.sub(' ', cleanFullText)
print(cleanFullText)
Explanation: Cleaning Up The Full Text
In order for better results from our analysis later we need to clean up the full text. 
This could be a project within itself and will differ item to item so for the intial run I have just set the full text to get lowered so 'Canada' and 'canada' aren't considered two different words.
I've also provided a basic regex that you can uncomment to strip everything other than words from the full text
End of explanation
import nltk # imports all the nltk basics
nltk.download("punkt") # Word tokenizer
nltk.download("stopwords") # Stop words
from nltk import word_tokenize
Explanation: Basic Analysis
So now we have the item's full text we are going to use the Natural Language Toolkit to perform some analysis on it using NLTK.
NLTK is a Python Library for working with written language data. It is free and very well documented. Many areas we'll be covering are treated in more detail in the NLTK Book, available for free online from here.
Note: NLTK provides tools for tasks ranging from very simple (counting words in a text) to very complex (writing and training parsers, etc.). Many advanced tasks are beyond the scope of this talk, but by the time we're done, you should understand Python and NLTK well enough to perform these tasks on your own!
Firstly, we will need to import NLTK.
End of explanation
len(fullText)
Explanation: Exploring Vocabulary
NLTK makes it really easy to get basic information about the size of a text and the complexity of its vocabulary.
len gives the number of symbols or 'tokens' in your text. This is the total number of words and items of punctuation.
set gives you a list of all the tokens in the text, without the duplicates.
Hence, len(set(fullText)) will give you the total number unique tokens. Remember this still includes punctuation. 
sorted() places items in the list into alphabetical order, with punctuation symbols and capitalised words first.
Number of characters
End of explanation
len(set(fullText))
Explanation: Number of unique characters
End of explanation
sorted(set(fullText))[:50] # Limited to 50
Explanation: List of unique characters
End of explanation
tokens = word_tokenize(cleanFullText)
len(tokens)
Explanation: Get token count (words + symbols)
For our analysis, we want to break up the full text into words and punctuation, this step is called tokenization
End of explanation
len(set(tokens))
Explanation: Unique Token Count
End of explanation
len(tokens)/len(set(tokens))
Explanation: Average number of times a word is used
We can investigate the lexical richness of a text. For example, by dividing the total number of words by the number of unique words, we can see the average number of times each word is used.
End of explanation
cleanFullText.count("vancouver")
Explanation: Number of times a specific word is used
End of explanation
100.0*fullText.count("and")/len(fullText) 
Explanation: Percentage of text that is a specific word
End of explanation
text = nltk.Text(tokens)
text.concordance("vancouver")
Explanation: Exploring Text
Concordance
End of explanation
text.similar("miles")
Explanation: Words used similarly??
End of explanation
text.common_contexts(["valley", "river"])
Explanation: Common contexts
Common contexts allow us to examine just the contexts that are shared by two or more words, such as valley and river.
End of explanation
v = set(text)
long_words = [word for word in v if len(word) > 15]
sorted(long_words)
Explanation: Longest words in the text
It is possible to select the longest words in a text, which may tell you something about its vocabulary and style
End of explanation
text.collocations()
Explanation: Collocations
We can also find words that typically occur together, which tend to be very specific to a text or genre of texts.
End of explanation
import numpy
# allow visuals to show up in this interface-
% matplotlib inline 
text.dispersion_plot(["river"])
Explanation: Graphing Data
Single Dispersion Plot
End of explanation
text.dispersion_plot(["miles", "sea", "lake", "land", "rest"])
Explanation: Multiple Dispersion Plot
End of explanation
from nltk import FreqDist
fdist = FreqDist(text)
fdist.most_common(50)
fdist.plot(25)
Explanation: Frequency distributions
End of explanation
<END_TASK>