<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <img src="../../img/logo_white_bkg_small.png" align="left" /> 
Worksheet 3
Step1:  Exercise 1
Step2:  Exercise 2
Step3:  Exercise 3
Step4:  Exercise 4
Step5:  Part 1
Step6:  Part 2
Step7:  Part 3
Step8:  Part 4
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
%pylab inline
Explanation: <img src="../../img/logo_white_bkg_small.png" align="left" /> 
Worksheet 3:  EDA Worksheet
This worksheet covers concepts covered in the first half of Module 1 - Exploratory Data Analysis in One Dimension.  It should take no more than 20-30 minutes to complete.  Please raise your hand if you get stuck.  
There are many ways to accomplish the tasks that you are presented with, however you will find that by using the techniques covered in class, the exercises should be relatively simple. 
Import the Libraries
For this exercise, we will be using:
* Pandas (http://pandas.pydata.org/pandas-docs/stable/)
* Numpy (https://docs.scipy.org/doc/numpy/reference/)
* Matplotlib (http://matplotlib.org/api/pyplot_api.html)
End of explanation
#Generate a series of random numbers between 1 and 100. 
random_numbers = pd.Series( np.random.randint(1, 100, 50) )
random_numbers[random_numbers > 10].sort_values().describe()
random_numbers = pd.Series( np.random.randint(1, 100, 50) )
random_numbers.sort_values().unique()[0:10]
# even and odd numbers
even_numbers = random_numbers[random_numbers % 2 == 0].count()
Explanation: Exercise 1: Summarize the Data
For this exercise, you are given a Series of random numbers creatively names random_numbers.  For the first exercise please do the following:
Remove all the numbers less than 10
Sort the series
Calculate the Tukey 5 number summary for this dataset
Count the number of even and odd numbers
Find the five largest and 5 smallest numbers in the series
End of explanation
random_numbers = pd.Series( np.random.randint(1, 100, 50) )
random_numbers.hist(bins=10)
Explanation: Exercise 2: Creating a Histogram
Using the random number Series create a histogram with 10 bins.
End of explanation
phone_numbers = [
'(833) 759-6854',
'(811) 268-9951',
'(855) 449-4648',
'(833) 212-2929',
'(833) 893-7475',
'(822) 346-3086',
'(844) 259-9074',
'(855) 975-8945',
'(811) 385-8515',
'(811) 523-5090',
'(844) 593-5677',
'(833) 534-5793',
'(899) 898-3043',
'(833) 662-7621',
'(899) 146-8244',
'(822) 793-4965',
'(822) 641-7853',
'(833) 153-7848',
'(811) 958-2930',
'(822) 332-3070',
'(833) 223-1776',
'(811) 397-1451',
'(844) 096-0377',
'(822) 000-0717',
'(899) 311-1880']
phone_records = pd.Series(phone_numbers)
phone_records.apply(lambda x: int(x.split(' ')[0][1:-1])).unique()
# or
area_codes = phone_number_series.str.slice(1,4)
area_codes = phone_number_series.str.extract('\((\d{3}))', expand=False)
Explanation: Exercise 3:  Counting Values
You have been given a list of US phone numbers.  The area code is the first three digits.  Your task is to produce a summary of how many times each area code appears in the list. To do this you will need to:
1. Extract the area code from each phone number
2. Count the unique occurances.
End of explanation
data = pd.read_csv( '../../data/dailybots.csv' )
data.describe()
data.info() # useful to see columns
data.columns()
data['botfam'].value_counts()
Explanation: Exercise 4:  Putting it all together: Bot Analysis
First you're going to want to create a data frame from the dailybots.csv file which can be found in the data directory.  You should be able to do this with the pd.read_csv() function.  Take a minute to look at the dataframe because we are going to be using it for to answer several different questions.
End of explanation
grouped_df = data[data.botfam == 'Ramnit'].groupby(['industry'])
grouped_df.sum()
Explanation: Part 1: Which industry sees the most Ramnit infections? Least?
Count the number of infected days for "Ramnit" in each industry industry. 
How: 
1.  First filter the data to remove all the infections we don't care about
2.  Aggregate the data on the column of interest. HINT:  You might want to use the groupby() function
3.  Add up the results
End of explanation
group2 = data[['botfam', 'orgs']].groupby(['botfam'])
summary = group2.agg([np.min, np.max, np.mean, np.dedian, np.std])
summary.sort_values(['orgs', 'median'], ascending=False)
Explanation: Part 2: Calculate the min, max, median and mean infected orgs for each bot family, sort by median
In this exercise, you are asked to calculate the min, max, median and mean of infected orgs for each bot family sorted by median.  HINT:
1.  Using the groupby() function, create a grouped data frame
2.  You can do this one metric at a time OR you can use the .agg() function. You might want to refer to the documentation here:  http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once
3.  Sort the values (HINT HINT) by the median column
End of explanation
df3 = data[['date', 'hosts']].groupby('date').sum()
df3.sort_values(by='hosts')
Explanation: Part 3: Which date had the total most bot infections and how many infections on that day?
For the next step, aggregate and sum the number of infections (hosts) by date.  Once you've done that, the next step is to sort in descending order.
End of explanation
#Plot the data
groupedFilteredData.unstack(level=1).plot(kind='line', subplots=False)
Explanation: Part 4: Plot the daily infected hosts for Necurs, Ramnit and PushDo
For the final step, you're going to plot the daily infected hosts for three infection types.  In order to do this, you'll need to do the following steps:
1.  Filter the data to remove the botfamilies we don't care about. 
2.  Use groupby() to aggregate the data by date and family, then sum up the hosts in each group
3.  Plot the data.  Hint:  You might want to use the unstack() function to prepare the data for plotting. 
4.  Use the plot() method to plot the results.
End of explanation
<END_TASK>