<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Twitter in Politics
Step1:  Creating and Prepping the Datasets
Twitter
Step2:  Once we were able to confirm that we have access to the Twitter API to access both Hillary Clinton's and Bernie Sanders' tweets, we ran a query for a single candidate's tweet. This allowed us to see what sort of data parameters we can access for a single Tweet. 
A list of all the parameteres is also available here
Step3:  A single tweet holds a large amount of information!
Now we define a function to pull all tweets from a particular screen_name
Step4:  Pull tweets for Hillary Clinton
Step5:  Add in Bernie Sanders Tweets
Step6:  Next, we create a function that converts the given list of tweets into a Pandas DataFrame. Note
Step7:  We now are going to create a few additional data sets to later highlight different aspects of the Twitter data in our visuals.
Step8:  Now import and read the 2016 Democratic Primary Results csv and assign it to a new DataFrame.
Step9:  In many cases, multiple primaries take place on the same date, the most well known example being Super Tuesday with 11 state primaries taking place on  single day (March 1st). It is important to look at the data not only sorted geographically, but by dates as well.
Step10:  The General Twitter Landscape
For the purposes of this project, we will focus on the tweets during the 1st half of the democratic primary (2/1/16 - 4/28/2016). 
| Bernie Sanders | Hillary Clinton
--- | --- | ---
Followers | 2MM | 6MM
Tweets Favorited | 6,794,300 | 3,796,642
Tweets Retweeted | 4,079,894 | 2,025,247
Looking at some numbers around both candidates' Twitter accounts, we see that Hillary has 3x the followers that Bernie Sanders has, however Bernie has had his tweets favorited and retweeted almost double the amount Hillary has.
What does this mean for either candidate?
1. Hillary's tweets may reach a larger audience
2. Bernies's audience may be more actively interested in his campaign OR is simply more active on Twitter in general
The Democratic Primary Landscape
For the purposes of this project, we are focusing on the delegates won by Hillary Clinton and Bernie Sanders from February 1st - April 26th.
| Bernie Sanders | Hillary Clinton
--- | --- | ---
States Won | 17 | 23
Delegates Won | 1,305 | 1,633
Let's break down these results by state
Step11:  Here, we see that Hillary has won all of the largest delegate states (Texas, Florida, New York, Pennsylvania). Because of these major wins, Bernie has only remained in the running by winning many of the much smaller delegate states (New Hampshire, Vermont, Oklahoma...his 7 state winning streak of Idaho, Utah, Washington, Hawaii, Alaska, Wisconsin, Wyoming, among many others).
Now in a more visually appealing fashion...in a map of good old 'Merica!
Step12:  In this format, we see that Hillary has completely won the southeast quarter of the US, continuing up through much of the northeast. Bernie's wins are more spread out, with small pockets of wins in New England and part of the Mid-West.
Tweets Over Time
The general volume of tweets over time is pretty consistent throughout this period, however there are a few peaks of interest the data reveals when graphed. Many of the spikes occur earlier on in the primaries (building momentum), but a few clearly coincide with later dates of import to the candidates
Step13:  To compare the above Twitter activity to the election results, below shows the number of delegates won by each candidate over time as well as the Tweets broken out side by side as well.
<ASSISTANT_TASK:>
Python Code:
import sys                             # system module 
import pandas as pd                    # data package
import matplotlib.pyplot as plt        # graphics module  
import datetime as dt                  # date and time module
import numpy as np                     # foundation for Pandas 
import tweepy                          # Twitter API wrapper 
import json
%matplotlib inline   
from plotly.offline import iplot, iplot_mpl                   # plotly imports
import plotly.graph_objs as go               
import plotly                             
import cufflinks as cf                      
cf.set_config_file(offline=True, offline_show_link=False)            
plotly.offline.init_notebook_mode()
pd.set_option('display.mpl_style', 'default')                 # Make the graphs prettier
Explanation: Twitter in Politics: Active Player or Passive Chronicler?
What sort of role does Twitter play during the Democratic Primary?
May 2016
Created by: Samantha Hoffmann & Ewa Maciukiewicz
Ewa Maciukiewicz has been working in digital advertising, specializing in programmatic, since 2009. She is a Product Manager currently working in machine learning and data optimization. She focuses on operational efficiencies through developing tools that streamline planning and execution within her organization. 
Samantha Hoffmann has been working in media sales & marketing for 3 years. She is currently an Account Executive who provides strategic solutions for top tier clients and develops efficient internal processes across multiple departments.
Why Twitter and politics?
We chose to focus our project on the timely cross-section of Twitter and American Politics due to how intrinsic social media is to everyday life, let alone a major cultural event such as this year's election. We wanted to see what (if any) impact a candidate's twitter presence has on election results during the primary, and how a canidate's performance may then also be reflected through their Twitter behavior. 
In such a high stakes game as the US Presidential Election... 
- are candidates connecting with the American people to foster more 'I'm with her' sentiment versus 'Who does she think she is?!' 
- are they tapping into the power of social media to truly make America great again?
In 2016, when such a large percentage of voters have spent most of their lives connected to social media...
- can a candidate afford to be less than averagely active on these platforms? 
- is it quantity or quality of engagements that wins out during a heated primary election season? 
Below, we will utilize Twitter data and Democratic Primary election data to see whether Hillary CLinton and Bernie Sanders fair successfully or end up crashing and feeling the bern...
Importing Packages
First, import all packages and modules needed to work with the data.
End of explanation
consumer_key = '9QBuUD4uvjy8FDBcLSuUK9I2I'
consumer_secret = 'NYsobcxgUE6Gv5UyusRE3fj8lrmsI5OsIDpUrf7fzVC3uIugGr'
access_token = '29009929-3efFzAePX2CFTUTrrwV5J3WbU0LWwyzGnbET4xJso'
access_secret = 'KDQD725blNsSBNsddeEo3AE3wIinRPEUD29BClPEiw0Ap'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)
hilz = api.get_user('HillaryClinton') # Get the User object for Hillary Clinton
bern = api.get_user('berniesanders') # Get the User object for Bernie Sanders
print (hilz.screen_name,'has', hilz.followers_count, 'followers')
print(bern.screen_name,'has', bern.followers_count, 'followers')
Explanation: Creating and Prepping the Datasets
Twitter:
In order to access data via the Twitter API, we had to register for an "application" that can integrate with Twitter. We are not actually integrating any applications, however, we still need to go through this process as Twitter considers all users as potential Twitter application developers.
Instructions on registering for credentials can be found on the twitter for developers: https://dev.twitter.com/oauth/overview/application-owner-access-tokens
We will be using Tweepy, a python Twitter API wrapper, to interact with the Twitter data.
Democratic Primary Election:
We had to create a csv with this data offline, and will include steps below to access said file, "2016 Democratic Primary Results.csv" (emailed along with this workbook).
End of explanation
result = api.search(q='from:HillaryClinton')      #Perform a basic search query where we 
                                                  #search for the candidate's tweets
len(result)                                       #Print the number of items returned by the search query to 
                                                  #verify our query ran (15 by default).
tweet = result[0]                                #Get the first tweet in the result
                                                
for param in dir(tweet):           #Analyze the data in one tweet to see what elements we can pull into our dataset
                                   #key names beginning with an '_' are hidden and usually not required, so we'll skip
    if not param.startswith("_"):
        print ("%s : %s\n" % (param, eval('tweet.'+param)))
Explanation: Once we were able to confirm that we have access to the Twitter API to access both Hillary Clinton's and Bernie Sanders' tweets, we ran a query for a single candidate's tweet. This allowed us to see what sort of data parameters we can access for a single Tweet. 
A list of all the parameteres is also available here: https://dev.twitter.com/overview/api/users
End of explanation
alltweets = []
                                                                 #Twitter has a restriction on how 
def get_all_tweets(screen_name):                                 #many tweets we can pull: approx.3200
                                                                 #make initial request for most recent tweets 
                                                                 #(200 is the maximum allowed count)
    new_tweets = api.user_timeline(
        screen_name = screen_name,count=200)
    
    alltweets.extend(new_tweets)                                  #save most recent tweets
    
    oldest = alltweets[-1].id - 1                                 #save the id of the oldest tweet less one
    while len(new_tweets) > 0:                                    #keep grabbing tweets until there are 
        print ("getting tweets before %s" % (oldest))             #no tweets left to grab
        new_tweets = api.user_timeline(
            screen_name = screen_name,count=200,max_id=oldest)    #all subsequent requests 
                                                                  #use the max_id param to prevent duplicates
        alltweets.extend(new_tweets)                              #save most recent tweets again
        oldest = alltweets[-1].id - 1
        print ("...%s tweets downloaded so far" % (len(alltweets)))
Explanation: A single tweet holds a large amount of information!
Now we define a function to pull all tweets from a particular screen_name:
End of explanation
if __name__ == '__main__':
        get_all_tweets("HillaryClinton")
Explanation: Pull tweets for Hillary Clinton:
End of explanation
if __name__ == '__main__':
        get_all_tweets("BernieSanders")
Explanation: Add in Bernie Sanders Tweets
End of explanation
def toDataFrame(tweets):
    DataSet = pd.DataFrame()
    DataSet['tweetID'] = [tweet.id for tweet in tweets]
    DataSet['tweetText'] = [tweet.text for tweet in tweets]
    DataSet['tweetRetweetCt'] = [tweet.retweet_count for tweet in tweets]
    DataSet['tweetFavoriteCt'] = [tweet.favorite_count for tweet in tweets]
    DataSet['tweetCreated'] = [tweet.created_at for tweet in tweets]
    DataSet['userID'] = [tweet.user.id for tweet in tweets]
    DataSet['userScreen'] = [tweet.user.screen_name for tweet in tweets]
    DataSet['userName'] = [tweet.user.name for tweet in tweets]
    DataSet['userFollowerCt'] = [tweet.user.followers_count for tweet in tweets]
    DataSet['userFriendsCt'] = [tweet.user.friends_count for tweet in tweets]
    return DataSet
DataSet = toDataFrame(alltweets)                    
from datetime import datetime as dt, date, time
DataSet['tweetDate'] = DataSet['tweetCreated']
DataSet['tweetDate'] = DataSet['tweetDate'].dt.strftime('%m/%d/%Y')
DataSet['tweetDate'] = pd.to_datetime(DataSet['tweetDate'])
DataSet['tweetDates'] = pd.to_datetime(DataSet['tweetDate'])
DataSet.head(5)
Explanation: Next, we create a function that converts the given list of tweets into a Pandas DataFrame. Note: we only chose a subset of tweet and user attributes to include that we think will be useful for our analysis.
End of explanation
Primary = DataSet[(DataSet['tweetDate'] > '2016-01-31') & (DataSet['tweetDate'] <= '2016-04-27')] 
                #selecting tweets in desired timeframe
Primary.groupby('userScreen').tweetFavoriteCt.sum() #pulling sum of tweet favorites by candidate
Primary.groupby('userScreen').tweetRetweetCt.sum()  #pulling sum of tweet retweets by candidate
number_tweets=Primary.tweetDate.dt.strftime('%m/%d/%Y').value_counts().sort_index() 
     #creating set of amount of tweets per day in total
TweetsbyCandidate = Primary.groupby('userScreen').tweetDate.value_counts().sort_index()
AllCandidateTweets = TweetsbyCandidate.unstack()
    #creating set of amount of tweets per day separated by candidate
Explanation: We now are going to create a few additional data sets to later highlight different aspects of the Twitter data in our visuals.
End of explanation
#import democratic primary results csv
url = '/Users/Samantha/Desktop/Data_Bootcamp/2016 Democratic Primary Results.csv'
dpr = pd.read_csv(url, nrows=40)     # read file and assign it to dpr
dpr.head(5)                          #check file has been stored correctly
#create a second set with States as the index and sorted on date
dpr1=dpr.set_index(['State'])
dpr1=dpr1.sort_values(['Date'])
dpr1.head(5)
Explanation: Now import and read the 2016 Democratic Primary Results csv and assign it to a new DataFrame.
End of explanation
#group results by DATE
DemsbyDate = dpr[['Date', 'Clinton Delegates', 'Sanders Delegates']].groupby('Date')
Sum = DemsbyDate.sum()
Sum
Explanation: In many cases, multiple primaries take place on the same date, the most well known example being Super Tuesday with 11 state primaries taking place on  single day (March 1st). It is important to look at the data not only sorted geographically, but by dates as well.
End of explanation
#bar chart for primary results by STATE
Sanders = dict(type="bar",                                      # trace type
           orientation="h",                                 # make bars horizontal
           name="Sanders",                                      # legend entry
           x=dpr1["Sanders Delegates"],                               # x data
           y=dpr1.index,                                # y data
           marker={"color": "#0048BA"}                         #bar color
          )
Clinton = dict(type="bar",                                    # trace type
             orientation="h",                               # horizontal bars
             name="Clinton",                                  # legend entry
             x=dpr1["Clinton Delegates"],                           # x data
             y=dpr1.index,                              # y data
             marker={"color": "#DE5D83"}                       # bar color
            )
layout = go.Layout(width=850, height=950, 
        title = 'Democratic Primary Results by State',
        xaxis=dict(
        title='Delegates',
        titlefont=dict(
            family='Arial, sans-serif',
            size=18,
            color='grey'
        ),
        showticklabels=True,
        tickangle=45,
        tickfont=dict(
            family='Arial, sans-serif',
            size=14,
            color='black'
        ),
        exponentformat='e',
        showexponent='All'
    ),
    yaxis=dict(
        title='State',
        titlefont=dict(
            family='Arial, sans-serif',
            size=18,
            color='grey'
        ),
        showticklabels=True,
        tickangle=45,
        tickfont=dict(
            family='Arial, sans-serif',
            size=14,
            color='black'
        ),
        exponentformat='e',
        showexponent='All'
    ),
)                   
    
 
iplot(go.Figure(data=[Sanders, Clinton], layout=layout))
Explanation: The General Twitter Landscape
For the purposes of this project, we will focus on the tweets during the 1st half of the democratic primary (2/1/16 - 4/28/2016). 
| Bernie Sanders | Hillary Clinton
--- | --- | ---
Followers | 2MM | 6MM
Tweets Favorited | 6,794,300 | 3,796,642
Tweets Retweeted | 4,079,894 | 2,025,247
Looking at some numbers around both candidates' Twitter accounts, we see that Hillary has 3x the followers that Bernie Sanders has, however Bernie has had his tweets favorited and retweeted almost double the amount Hillary has.
What does this mean for either candidate?
1. Hillary's tweets may reach a larger audience
2. Bernies's audience may be more actively interested in his campaign OR is simply more active on Twitter in general
The Democratic Primary Landscape
For the purposes of this project, we are focusing on the delegates won by Hillary Clinton and Bernie Sanders from February 1st - April 26th.
| Bernie Sanders | Hillary Clinton
--- | --- | ---
States Won | 17 | 23
Delegates Won | 1,305 | 1,633
Let's break down these results by state:
End of explanation
for col in dpr.columns:
    dpr[col] = dpr[col].astype('str')
scl = [[0.0, '#0048BA'],[0.495,'#0048BA'],[0.505, '#DE5D83'],[1.0, '#DE5D83']]
dpr['text'] = dpr['State'] + '<br>' +\
    dpr['Date']+ '<br>' +\
    'Clinton Delegates: '+dpr['Clinton Delegates']+'<br>'+\
    'Sanders Delegates: '+dpr['Sanders Delegates']+ '<br>' +\
    'Winner: ' + dpr['Winner']
data = [ dict(
        type='choropleth',
        colorscale = scl,
        showscale = False,
        autocolorscale = False,
        hoverinfo = 'location+text',
        locations = dpr['Code'],
        z = dpr['% Clinton'].astype(float),
        locationmode = 'USA-states',
        text = dpr['text'],
        marker = dict(
            line = dict (
                color = 'rgb(255,255,255)',
                width = 2
            )
        ),
    
            )]
layout = dict(
        title = "'"+'Merica: 2016 Democratic Primary Results',
        showlegend = True,
        
        geo = dict(
            scope='usa',
            projection=dict( type='albers usa' ),
            showlakes = True,
            lakecolor = 'rgb(255, 255, 255)',
        ),
    )
 
    
    
iplot(go.Figure(data=data, layout=layout))
Explanation: Here, we see that Hillary has won all of the largest delegate states (Texas, Florida, New York, Pennsylvania). Because of these major wins, Bernie has only remained in the running by winning many of the much smaller delegate states (New Hampshire, Vermont, Oklahoma...his 7 state winning streak of Idaho, Utah, Washington, Hawaii, Alaska, Wisconsin, Wyoming, among many others).
Now in a more visually appealing fashion...in a map of good old 'Merica!
End of explanation
#Total Tweets overtime for both candidates
plt.rcParams['figure.figsize'] = (35, 10)
# Plot the Time Zone data as a bar-graph
number_tweets.plot(kind='bar')
plt.xlabel('Dates', fontsize=25)
plt.ylabel('Tweet Count', fontsize=25)
plt.title('Total Tweet Timeline', fontsize=30, fontweight='bold')
fig, ax = plt.subplots()
fig.suptitle('Total Tweet Timeline by Candidate', fontsize=30, fontweight='bold')
AllCandidateTweets.T.plot(ax=ax, 
        kind='bar',                 # line plot 
        color=['mediumslateblue', 'orchid'],)   # line color 
ax.set_xlabel('Dates', fontsize=25)
ax.set_ylabel('Tweet Count', fontsize=25)
ax.annotate('Coming up to NYC Primary', color= 'black', xy=(74, 55), xytext=(72, 59),weight='bold',
            arrowprops=dict(facecolor='black', shrink=0.1), size=20)
ax.annotate('Coming up to Super Tuesday', color = 'black', xy=(18, 55), xytext=(20, 59), weight='bold',
            arrowprops=dict(facecolor='black', shrink=0.1), size=20)
Explanation: In this format, we see that Hillary has completely won the southeast quarter of the US, continuing up through much of the northeast. Bernie's wins are more spread out, with small pockets of wins in New England and part of the Mid-West.
Tweets Over Time
The general volume of tweets over time is pretty consistent throughout this period, however there are a few peaks of interest the data reveals when graphed. Many of the spikes occur earlier on in the primaries (building momentum), but a few clearly coincide with later dates of import to the candidates: 
May 1st, Super Tuesday: Appropriately named, on this one day alone, over 850 Democratic delegates were up for grabs. Looking at the Twitter data, Bernie tweeted 54 tweets that day (the 4th highest amount in a single day during this entire period). Ironically, Hillary tweeted only 17 tweets that day, one of her mid-low amounts.
Regardless of the social media buzz (or lackthereof), Hillary walked away with 511 delegates from Super Tuesday's 11 state primaries. Bernie successfully locked in 348.
April 19th, New York Primary: With both candidates calling this state home, New York holding the key to a total of 247 delegates, and the race up to this point being incredibly close, it is no surprise that we see spikes in the Twitter data before and after this event. Just a couple days prior (April 15th), Hillary sent out a personal high of 54 tweets in one day and Bernie spiked to 31 - both no doubt in response to the heated debate that took place the prior night in Brooklyn,NY. After such a heated forum, both camps turned to Twitter (among other channels) to set their records straight and keep their messaging strong leading in to the primary election. Hillary was also no doubt calling on all of her beloved New Yorkers to forget about the prior week's 'Subway Swipe-gate' (I'm with her... those turnstiles can be tricky).
End of explanation
Sum.iplot(kind='bar',
          subplots=True,
          color=['#DE5D83','#0048BA'],
          subplot_titles=True, 
          legend=False,
         layout = dict(
    autosize=False,
    title='Delegate Wins Over Time',
    showlegend=False,
    width=950,
    height=500,
    margin=dict(
        l=50,
        r=50,
        b=100,
        t=100,
        pad=4)))
AllCandidateTweets.T.iplot(kind='bar', subplots=True, subplot_titles=True, layout = dict(
    autosize=False,
    title='Candidate Tweets Over Time',
    width=975,
    height=500))
Explanation: To compare the above Twitter activity to the election results, below shows the number of delegates won by each candidate over time as well as the Tweets broken out side by side as well.
End of explanation
<END_TASK>