<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data Exploration
Goals
Introduction to Sentiment Analysis use case
How to quickly understand a real world data set
Introduction
A useful application of machine learning is "sentiment analysis".  Here we are trying to determine if a person feels positively or negatively about what they're writing about.  One important application of sentiment analysis is for marketing departments to understand what people are saying about them on social media.  Nearly every medium or large company with any sort of social media presence does some sort of sentiment analysis like the task we are about to do.
Here we have a collection of tweets from the tech conference SXSW talking about apple brands.  These tweet are hand labeled by humans using a tool I built called CrowdFlower.  Our goal is to build a classifier that can generalize the human labels to more tweets.
The labels are what's known as training data, and we're going to use it to teach our classifier what text is positive sentiment and what text is negative sentiment.
Let's take a look at our data.  Machine learning classes tend to talk mostly about algroithms, but in practice, machine learning practicioners usually spend most of their time looking at their data.
This is a real data set, not a toy one, and I've left it uncleanup up so you will have to work through a few of the messy issues that almost always happen in the real world.
Step1:  Ok, that looks good - if a little messy.  Let's open the file with some python
Loading Data
Step2:  Data frames are pretty cool, for example I can index the column by name.
Step3:  Check for understanding
Some questions that I immediately asked myself (and you should too)
1. How many rows are in our data set?
2. How many different types of labels are there?  What are they?
3. What year was this data collected?
If you were my student and you were sitting in front of me, I would make you actually do this.  Unfortunately I can't force you to answer these questions yourself, but you will have more fun and learn more if you do.
You will probably need to google around a little to figure out how to use the dataframe to answer these questions.  You can check out the cool pandas tutorial at https
Step4:  Looks like there are 9093 rows in our dataset
Question 2
Step5:  Hmmm... looks like there are 4 values for the sentiment of the tweets with "No emotion toward brand or product" being the most common.
Step6:  Interesting, there is a label "I can't tell" along with "Positive emotion", "Negative emotion" and "No emotion toward brand or product"
Question 3
Step7:  Hm, it's a 3G iphone, when was that? 2010?
Step8:  Ok - the ipad2 was released in 2011, these tweets must be from 2011.
Data Cleanup
If we dig into the data set one thing we'll notice is that some of the tweets are actually empty.
Step9:  It is best practice to not change the input data.  It's better to clearly show the ways that you've modified your data in your code.  In this case, we can use pandas to easily pull out the rows where the tweets are empty.  Here we are indexing into our data frame with the results of a pd.notnull function - this notation is really convenient.
Step10:  We also need to remove the same rows of labels so that our "tweets" and "target" lists have the same length.
<ASSISTANT_TASK:>
Python Code:
# Our data file is in ../scikit/tweet.csv
# in a Comma Separated Values format
# this command uses the shell to print out the first ten lines
!head ../scikit/tweets.csv
Explanation: Data Exploration
Goals
Introduction to Sentiment Analysis use case
How to quickly understand a real world data set
Introduction
A useful application of machine learning is "sentiment analysis".  Here we are trying to determine if a person feels positively or negatively about what they're writing about.  One important application of sentiment analysis is for marketing departments to understand what people are saying about them on social media.  Nearly every medium or large company with any sort of social media presence does some sort of sentiment analysis like the task we are about to do.
Here we have a collection of tweets from the tech conference SXSW talking about apple brands.  These tweet are hand labeled by humans using a tool I built called CrowdFlower.  Our goal is to build a classifier that can generalize the human labels to more tweets.
The labels are what's known as training data, and we're going to use it to teach our classifier what text is positive sentiment and what text is negative sentiment.
Let's take a look at our data.  Machine learning classes tend to talk mostly about algroithms, but in practice, machine learning practicioners usually spend most of their time looking at their data.
This is a real data set, not a toy one, and I've left it uncleanup up so you will have to work through a few of the messy issues that almost always happen in the real world.
End of explanation
import pandas as pd    # this loads the pandas library, a very useful data exploration library
import numpy as np     # this loads numpy, a very useful numerical computing library
# Puts tweets into a data frame
df = pd.read_csv('../scikit/tweets.csv') # read the file into a pandas data frame
print(df.head())       # print the first few rows of the data frame
Explanation: Ok, that looks good - if a little messy.  Let's open the file with some python
Loading Data
End of explanation
tweets = df['tweet_text'] # sets tweets to be the first column, titled 'tweet_text'
print(tweets.head())
Explanation: Data frames are pretty cool, for example I can index the column by name.
End of explanation
print(tweets.shape) # print the shape of the variable tweets
Explanation: Check for understanding
Some questions that I immediately asked myself (and you should too)
1. How many rows are in our data set?
2. How many different types of labels are there?  What are they?
3. What year was this data collected?
If you were my student and you were sitting in front of me, I would make you actually do this.  Unfortunately I can't force you to answer these questions yourself, but you will have more fun and learn more if you do.
You will probably need to google around a little to figure out how to use the dataframe to answer these questions.  You can check out the cool pandas tutorial at https://pandas.pydata.org/pandas-docs/stable/10min.html - it will be useful for many things besides this tutorial!
Question 1: How many rows in the dataset?
End of explanation
# we make target the list of labels from the third column
target = df['is_there_an_emotion_directed_at_a_brand_or_product']
# describe is a cool function for quick data exploration
target.describe()
Explanation: Looks like there are 9093 rows in our dataset
Question 2: How many different types of labels are there?  What are they?
End of explanation
target.value_counts()
Explanation: Hmmm... looks like there are 4 values for the sentiment of the tweets with "No emotion toward brand or product" being the most common.
End of explanation
tweets[0]
Explanation: Interesting, there is a label "I can't tell" along with "Positive emotion", "Negative emotion" and "No emotion toward brand or product"
Question 3: What year was this data collected?
End of explanation
tweets[200]
Explanation: Hm, it's a 3G iphone, when was that? 2010?
End of explanation
print(tweets[6])
Explanation: Ok - the ipad2 was released in 2011, these tweets must be from 2011.
Data Cleanup
If we dig into the data set one thing we'll notice is that some of the tweets are actually empty.
End of explanation
fixed_tweets = tweets[pd.notnull(tweets)]
Explanation: It is best practice to not change the input data.  It's better to clearly show the ways that you've modified your data in your code.  In this case, we can use pandas to easily pull out the rows where the tweets are empty.  Here we are indexing into our data frame with the results of a pd.notnull function - this notation is really convenient.
End of explanation
fixed_target = target[pd.notnull(tweets)]
Explanation: We also need to remove the same rows of labels so that our "tweets" and "target" lists have the same length.
End of explanation
<END_TASK>