<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Who Am I?
<img src="./images/google_map.png" align="right" style="width
Step1:  The notebook understand that I was asking for the execution of a command. OK, with that concept of what you are looking at out of the way, let's move on.
<hr>
What Does a Data Scientist Do?
Computer programming
<img src="./images/mr_robot.jpg" align="right" style="width
Step2:  Comparing with Cosine Similarity
<img src="./images/cosine_similarity.png" align="right" style="width
Step3:  As we would hope, cheeseburgers and hamburgers are pretty similar, while cheeseburgers and corvettes are not.
Finding the Average Word Embedding
Of course, for our problem, we need need to do more than analyze a single word. We want to find analyze the "Tweets" from the class. How do do this? We can find the average vector of all the words in the Tweet. That should give us the best understanding of what the Tweet was about.
<br>
How do you find an average vector? Just like you would find the average (or mean) with numbers. Take the sum of the vectors and divide by the number of vectors. Check out this simple example.
Step4:  I've written some code that will find the average word vector (or "embedding") for a particular Tweet with our 300-D vectors.
Compare Two Tweets
Once we've converted our Tweets into average word vectors, we want to see how similar they are, right? So to make this process a little more efficient, I wrote some code that would check in a particular class who had the most similar Tweet in the data set. See, if we searched through all of Twitter's history of 200 billion Tweets per year, we would likely find a nice match for each Tweet. However, with a data set this small (90 Tweets), the matches won't be quite as easy to find, so I'm just looking for those of you that really had the best matches.
<br>
Let's find an example for each class period
Step5:  Our model does not have the vocabulary of Google News, but it has clearly learned a few interesting relationships.
We now have 2 models. 
One model has been trained with billions of words (over a million unique) with Google News data.
The other model has been trained with over a million words using Twitter data.
This is an interesting case to compare the amount of training data vs. the domain of the training data (also worth noting is that I'm sure the Google data scientists tuned the heck out of their model, where I took some "off the shelf" tuning parameters, so there's that).
Step6:  First, Let's Look at Our Model Build From Twitter Data
Step7:  Now Let's Look at the Model Trained with Google News Data
<ASSISTANT_TASK:>
Python Code:
2+2
Explanation: Who Am I?
<img src="./images/google_map.png" align="right" style="width: 500px; margin:0px 50px;" />
Small Town Living
I grew up in a town not far from Morton enjoying math and science classes, scholastic bowl, band, and sports. While my experience was a little more rural than the average MHS student, our high school experiences would have a lot in common. 
<br clear="all" />
<img src="./images/altgeld_hall.png" align="right" style="width: 500px; margin:0px 50px;"/>
University of Illinois at Urbana-Champaign
Path to finding a major:
<br>
* Industrial Engineering 
* Mechanical Engineering
* Computer Science
* <b>Mathematics</b>
<br>
<br>
Master's Degree:
<br>
* Mathematics Education
<br clear="all" />
<img src="./images/the-potterdome.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
Started Coaching Volleyball & Teaching
After a few years, in 2004, I ended up in a little place known as the "Home of the Potters." I was hired as a math teacher, volleyball coach, and soon become acquainted with my favorite student activity of all time...
<br clear="all" />
<img src="./images/math_team_sign.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
Math Team is the Bomb
Loved a lot of things about my time at MHS, but math team takes the cake. MHS has some smart cookies and we had quite a run.
<br clear="all" />
<img src="./images/coursera.png" align="right" style="width: 500px; margin:0px 50px;"/>
MOOCS Are Changing Education
<br>
<br>
I have this thing where I have to keep reinventing myself. I think it'd be hard to make a big change like this with only an online education, an online education can allow you to pivot careers quickly as a supplement to a 4-year or > degree. I spent all of my 2014 summer break in the library completing an intensive MOOCs program in data science.
<hr>
Topics to Cover Today
<img src="./images/big_ideas.png" align="right" style="width: 500px; margin:0px 50px;"/>
Big ideas:
What does a data scientist do?
What are the families of problems?
Reflection: The biggest disservice I did to my students a math teacher
<br>
Our problem:
<br>Can we use data science to identify students who make similar Tweets?</br>
This Presentation
I've prepared this presentation in a "Jupyter Notebook." This is a web based computation environment that allows me to mix and match web syntax like markdown/html (for text, images, bullet points, etc) and code that can be executed right in the browers from a general purpose computing language, in this case Python. 
<br>
I won't be showing much Python until the close to the end of the end of this presentation, but just to give you an idea of what I mean, if I type "2+2" into the next cell of this notebook and type enter...
End of explanation
from gensim.models import Word2Vec
import gensim
model = gensim.models.KeyedVectors.load_word2vec_format(
    './private_data/GoogleNews-vectors-negative300.bin', binary=True)
# What does a word vector look like?
model.word_vec("cheeseburger")
Explanation: The notebook understand that I was asking for the execution of a command. OK, with that concept of what you are looking at out of the way, let's move on.
<hr>
What Does a Data Scientist Do?
Computer programming
<img src="./images/mr_robot.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
What kind of programming?
Open source
Languages that have great statistics libraries (R, Python, Scala)
<br clear="all" />
<br clear="all" />
<img src="./images/big_data.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
Analyze large data sets for business insights
What is big?
"Big Data" means different thing to different people. You might take it to mean:
* Too big for your graphing calculator
* Too big to open in a spreadsheet program
* Bigger than the available memory (RAM) on your computer
* Bigger than the available disk space on your computer
* Repeat the last two statements for a large server, meaning we have to move to a cluster of servers
<br clear="all" />
<br clear="all" />
<br clear="all" />
<img src="./images/target_model.png" align="right" style="width: 500px; margin:0px 50px;"/>
Develop models using data
Key Idea:
Examples of events that happened in the past can be useful to train a computer model
<hr>
Supervised vs. Unsupervised Learning
Supervised Learning
This is the task of inferring a function from labeled training data. From your classes, think of linear regression, where the function is a line, and the labels are the y values assigned to each observation in your data.
<img src="./images/titanic_data.png" align="center" style="width: 1000px;"/>
Some differences about how data scientists perform supervised learning from a high school linear regression problem:
* Not limited to a single dimension
* Many methods other than linear regression exist
* Classification (What class is something?) is just as important as regression
<br clear="all" />
<br clear="all" />
<img src="./images/kaggle_sea_lions.png" align="right" style="width: 500px; margin:0px 50px;"/>
Data Science as Sport
Supervised learning competitions have become increasingly popular in recent years. Google recently purchased the most well known such competition site, Kaggle.
<br clear="all" />
<br clear="all" />
<img src="./images/clustering_dbscan.png" align="right" style="width: 500px; margin:0px 50px;"/>
Unsupervised Learning
This is task of inferring a function to describe hidden structure from "unlabeled" data. Clustering, or the task of grouping similar things together, is a common example of unsupervised learning
<hr>
Structured Data vs. Unstructured Data
<img src="./images/mtcars_structured.png" align="right" style="width: 500px; margin:0px 50px;"/>
Structured data
This is data organized into a row and column structure. You may think of words like:
* Matrices 
* Arrays
* Spreadsheets
<br clear="all" />
<br clear="all" />
<img src="./images/puppy.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
Unstructured data
This is data without an identifiable internal structure. Examples include:
* Images
* Audio
* Text 
* Video
The image pixel data of a cuddly puppy can be transformed into a format recognizable to a computer something like what you see below.
<img src="./images/rgb_data.gif" align="right" style="width: 500px; margin:0px 50px;"/>
<hr>
Iterative Approximation vs. Direct Solutions
<img src="./images/holding_my_nose.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
A Teacher's Confession
While I always tried to have the same level of enthusiasum regardless of what I was teaching, the reality was that some topics were less appealing. In these cases I held my nose and charged ahead.
<br>
<br>
Probably like most students, the reasons for the lack of appeal typically had to do with me not seeing the application of the mathematics. 
<br clear="all" />
<br clear="all" />
<img src="./images/newtons_method.gif" align="right" style="width: 500px; margin:0px 50px;"/>
Examples of "Holding my nose"
Typically, I'm talking about examples where there were two methods of solving a problem, one that involved a more iterative, step-by-step approach, and one that involved solving directly for an answer with some clever tricks,
* Empirical probability vs. Theoretical probability
* Newton's method vs. Calculating derivative algebraically
* Solving a system of equations with matrices vs Subtitution
<br>
<br>
I like clever tricks. In retrospect, why do things the "hard way"? Because oftentimes, what seems hard to us is what a computer understands.
<br clear="all" />
<br clear="all" />
<img src="./images/gradient_descent.png" align="right" style="width: 500px; margin:0px 50px;"/>
<br>
What is the solving the types of problems we encounter?
Linear Algebra & Calculus
 * Matrix multiplication 
 * Dot products
 * Gradients (derivatives in higher dimensions) 
 * In short, optimization methods
 <br clear="all" />
<br clear="all" />
Let's Talk Twitter Profiles
<br clear="all" />
<img src="./images/tweet.png" align="right" style="width: 500px; margin:0px 50px;"/>
OK, now that we've covered a few basics of what a data scientist does, let's talk about our problem. To summarize the problem we were given:
<b>Given a "Tweet" from every student in the class, we would like to be able to find the Tweets that are most similar to any particular student that we choose.</b>
Before we solve the problem, we'll want to think about a few of the concepts we discussed earlier.
* Is our data structured or unstructured?
* Is this a supervised or unsupervised learning problem?
<br clear="all" />
<br clear="all" />
Vectors
<img src="./images/2d_position_vector.png" align="right" style="width: 500px; margin:0px 50px;"/>
Remember from your classes that vectors are simply a direction and a magnitude. We data scientists love using vectors to represent data. 
<br>
<br>
Perhaps, for example, the x component could tell us how often the word "pizza" appears in your Twitter profile and the y component could tell us how often the word "party" appears. This profile would have two appearances of pizza and 3 of party. You can quickly see that this 2D vector representation would have serious limitations...
<br clear="all" />
<br clear="all" />
Word Embeddings
<img src="./images/3d_position_vector.png" align="right" style="width: 500px; margin:0px 50px;"/>
One important limitation is the number of dimensions at just 2. Of course, we can expand that. Here's a picture with 3. While it's hard to visualize > 3 dimensions, there's no reason we have to stop with 3 in our vector representation for the computer.
<br>
<br>
It would also be nice if we had a method of converting words to vector space that could:
* Automatically recognize similar words have similar meanings (i.e. "bad" and "terrible")
* Give all of the profiles the same vector length
<br>
<br>
One tool for doing this is Word2Vec, an open source tool released by Google. It uses neural networks to train the computer what the meanings of words (or documents) are, and then convert those into a vector. Words/documents with similar meanings have vectors that are "embedded" in a high dimensional space near each other.  I realize I'm glossing over the details of how this is done here, but those details go beyond the scope of this lesson.
Cosine Similarity
<img src="./images/cosine_similarity.png" align="right" style="width: 500px; margin:0px 50px;"/>
Once we have the vector for each profile, how can we know if they are similar? We want vectors that are close together to be be ranked as similar, those that are far apart to be ranked as dissimilar. A popular method is <b>cosine similarity</b>. 
<br>
<br>
Many of you already know the exact formula for cosine similarity from your Pre-Calculus class. Since it is used to compare angles between two vectors, it'll give an answer close to 1 for vectors that are close, -1 for those that are not. What's the difference between here and your Pre-Calculus class? Very little, except that we are not limited to two dimensions
Let's Talk Data
<img src="./images/landry_bender.jpg" align="right" style="width: 500px; margin:0px 50px;"/>
OK, so we've talked a little bit about what we'd like to do. Where are we going to get the data to do to train Word2Vec how to understand how the kids talk nowadays. I gathered it from two places.
Google was nice enough to provide the public an already trained model of Word2Vec that had learned the meanings of words from many thousands of Google News articles.
Considering that people (especially teens) may speak differently than reporters, I gathered about 100,000 Tweets from these celebrities and in the case of one popular teen, from her followers. I then created my own version of a Word2Vec model to include that language. It won't know nearly as many words, but it should know about how some workds are used specifically on Twitter.
So who did these Tweets come from? I'm betting you've heard of at least some of these.
Kim Kardashian 
Adam Savage
Bill Nye
Neil deGrasse Tyson
Donald Trump
Hillary Clinton 
Richard Dawkins
Commander Scott Kelly
Barack Obama
NASA
The Onion
Landry Bender (followers only)
Using Word2Vec to take Words to Vectors
Word2Vec literally (yes, literally, not figuratively) does what its name says. Let's take a look at a Word2Vec converting a word to a vector. Let's start with the version of the model that Google has trained for us using Google News.
End of explanation
cheeseburger = model.word_vec("cheeseburger")
hamburger = model.word_vec("hamburger")
corvette = model.word_vec("corvette")
# Numpy is a linear algebra library that we can use to help us efficiently find the dot product and
# vector magnitude (much like the distance formula or Pythagorean theorem)
import numpy as np
# How similar are the words "cheeseburger" and "hamburger"?
numerator = np.dot(cheeseburger,hamburger)
denominator = np.sqrt(np.sum(cheeseburger**2)) * np.sqrt(np.sum(hamburger**2))
print(numerator/denominator)
# How similar are the words "cheeseburger" and "corvette"?
numerator = np.dot(cheeseburger,corvette)
denominator = np.sqrt(np.sum(cheeseburger**2)) * np.sqrt(np.sum(corvette**2))
print(numerator/denominator)
Explanation: Comparing with Cosine Similarity
<img src="./images/cosine_similarity.png" align="right" style="width: 500px; margin:0px 50px;"/>
So that's pretty nice. However, it is going to be impossible to really do a visual comparison of two vectors to see how similar they are. We need an objective method for comparing, and that's where the cosine similarity comes in. Remember our formular from Pre-Calculus (if you've taken the class)? Here it is again in case you forgot. 
<br>
Let's use Python to work this out and see how similar a couple of words are based on their vector representations.
What would we expect a cheeseburger to be more like if our model is working properly, a cheeseburger or a Corvette?
End of explanation
vector_a = np.array([1,0])
vector_b = np.array([0,.5])
average_a_b = (vector_a+vector_b )/2
print(average_a_b)
Explanation: As we would hope, cheeseburgers and hamburgers are pretty similar, while cheeseburgers and corvettes are not.
Finding the Average Word Embedding
Of course, for our problem, we need need to do more than analyze a single word. We want to find analyze the "Tweets" from the class. How do do this? We can find the average vector of all the words in the Tweet. That should give us the best understanding of what the Tweet was about.
<br>
How do you find an average vector? Just like you would find the average (or mean) with numbers. Take the sum of the vectors and divide by the number of vectors. Check out this simple example.
End of explanation
import pandas as pd
tweets = pd.read_csv('./private_data/no_names.csv')
tweets.head()
from gensim.models import Word2Vec
import gensim
morton_model = gensim.models.Word2Vec.load('./private_data/morton_model')
# Just for fun. What has our model learned from Twitter?
morton_model.wv.most_similar(['donald'])
morton_model.wv.most_similar(['rt'])
morton_model.wv.most_similar(['crazy'])
Explanation: I've written some code that will find the average word vector (or "embedding") for a particular Tweet with our 300-D vectors.
Compare Two Tweets
Once we've converted our Tweets into average word vectors, we want to see how similar they are, right? So to make this process a little more efficient, I wrote some code that would check in a particular class who had the most similar Tweet in the data set. See, if we searched through all of Twitter's history of 200 billion Tweets per year, we would likely find a nice match for each Tweet. However, with a data set this small (90 Tweets), the matches won't be quite as easy to find, so I'm just looking for those of you that really had the best matches.
<br>
Let's find an example for each class period:
End of explanation
from embeddings import embeddings
Explanation: Our model does not have the vocabulary of Google News, but it has clearly learned a few interesting relationships.
We now have 2 models. 
One model has been trained with billions of words (over a million unique) with Google News data.
The other model has been trained with over a million words using Twitter data.
This is an interesting case to compare the amount of training data vs. the domain of the training data (also worth noting is that I'm sure the Google data scientists tuned the heck out of their model, where I took some "off the shelf" tuning parameters, so there's that).
End of explanation
# Let's start by adding a 300-D word vector for every Tweet you guys sent out.
tweets1 = embeddings.append_word_vector_cols(tweets,morton_model,text_col='text')
tweets1.head()
embeddings.most_similar_one_class(tweets1,"1st")
embeddings.most_similar_one_class(tweets1,"2nd")
embeddings.most_similar_one_class(tweets1,"3rd")
embeddings.most_similar_one_class(tweets1,"4th")
embeddings.most_similar_one_class(tweets1,"6th")
embeddings.most_similar_one_class(tweets1,"7th")
Explanation: First, Let's Look at Our Model Build From Twitter Data
End of explanation
tweets2 = embeddings.append_word_vector_cols(tweets, model, keyed_vec=True, text_col='text')
tweets2.head()
embeddings.most_similar_one_class(tweets2,"1st")
embeddings.most_similar_one_class(tweets2,"2nd")
embeddings.most_similar_one_class(tweets2,"3rd")
embeddings.most_similar_one_class(tweets2,"4th")
embeddings.most_similar_one_class(tweets2,"6th")
embeddings.most_similar_one_class(tweets2,"7th")
Explanation: Now Let's Look at the Model Trained with Google News Data
End of explanation
<END_TASK>