<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Hmwk #1
Step1:  Represent the following table using a data structure of your choice
Step2:  Calculate the mean temperature and mean humidity
Step3:  Print outlook and play for those days where the temperature is greater than the average temperature
Step4:  Print outlook and play for those days where the humidity is greater than the average humidity
Step5:  Convert the temperature to Celsius and add a new column therefore in the table.
Step6:  #1
How often do you play tennis independent of the other attributes?
From the out put we can see that we played tennis 9 days w/ a probibilty of 9/14
Step7:  #2
How often do you play tennis when it is "sunny"?
From the output we can see that we played when it was sunny on 2 days or 2/14
Step8:  #3
*Compare the average, minimum and maximum temperature when you play tennis? *
Step9:  #4
Compare the average, minimum and maximum humidity when you play tennis?
Step10:  #5
Plot the an scatter plot (x,y diagramm) of humidity (x) and temperature (y) when you play tennis compared to when you do not play tennis.
Step11:  The only inferences I can make from the scatter plot above, is that you always play when the humidity is between 70 and 85. Temperature seems to play no part of the decision process when you go out to play as from teh scatter plot the plays and no play poionts are evenly distributed across the y axis (Temperature).
#2
We have a set of 8 files where the first 7 and the last 2 have a different format. First I removed the header information from the files and removed any superfolous line breaks, I then read them into pandas in two respective groups. I then had to normalize the dates of teh second dataset to match the dates of the first. Also I had to normalize the values of the first dataset b/c they were in units of 1000, so I made it in units of 1.
Cleaning & Normalization
Step12:  Merging
Step13:  Plot CA vs AK
Step14:  New England vs Southwest
In order to plot these values I have to do some feature engineering to create columns for the respective regions that are not in the original dataset. 
For New England I used
Step15:  Greatest Change in Population
We can quantify population growth in direct terms or relativly using percentages
Step16:  As you can see from teh table above, CA had the largest growth in terms of raw numbers for the time period. However, we can gain additional insites by looking at percentatge growth.
Relative Growth
We can also measure growth as a percenagte difference
Step17:  Some states had no net growth and some had negative growth
Step18:  3
4
We will use a Decision Tree to build a model that will allow us to classify wines into one of three categories.
Step19:  Test/Train Split
Split the dat set into 75% for training and 25% for testing
Step20:  Train Model
I used the defaults for the DecisionTreeClasifier package in scikit
Step21:  Evaluation
Evaluate based on confusion matrix how well the model performed on training vs. testing.
Step22:  As you can see from the confusion matrix, inputs of Class 1 & 2 were perfectly classified. There were only 2 mistakes on Class 3.
5
Step23:  What are the statistical distributions of variables using no class?
Step24:  How much missing data is there?
Step25:  How do distributions differ by each gender?
Step26:  Describe summary statistics for each attribute.
Step27:  Visualize potential difference via the scatter plots.
Are there any ‘high’ correlations between variables?
We can see a correlation between height and weight
Step28:  Create a new variable for the weight in lbs
Creating a new column for pounds obviously does not create any new correlations b/c it simply a linear combination w/ the kg weight.
Step29:  Add new variable w​eight + height.
Step30:  BMI
There appears to be obese males and females in the dataset
Step31:  Split Data By Sport
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
%pylab inline
Explanation: Hmwk #1
End of explanation
df = pd.read_csv("weather.csv", header=0, index_col=0)
df
Explanation: Represent the following table using a data structure of your choice
End of explanation
mean_temp = df["temperature"].mean()
mean_temp
mean_humidity = df["humidity"].mean()
mean_humidity
Explanation: Calculate the mean temperature and mean humidity
End of explanation
temp_selector = df['temperature'] > mean_temp
df[temp_selector][["outlook", "play"]]
Explanation: Print outlook and play for those days where the temperature is greater than the average temperature
End of explanation
humidity_selector = df['humidity'] > mean_humidity
df[humidity_selector][["outlook", "play"]]
Explanation: Print outlook and play for those days where the humidity is greater than the average humidity
End of explanation
df["temp_C"] = ( df["temperature"] - 32 ) * (5/9.0)
df
Explanation: Convert the temperature to Celsius and add a new column therefore in the table.
End of explanation
play_selector = df["play"]=="yes"
play_days = df[play_selector]
len(play_days)
Explanation: #1
How often do you play tennis independent of the other attributes?
From the out put we can see that we played tennis 9 days w/ a probibilty of 9/14
End of explanation
sunny_selector = df["outlook"]=="sunny"
sunny_play_days = df[sunny_selector & play_selector]
len(sunny_play_days)
Explanation: #2
How often do you play tennis when it is "sunny"?
From the output we can see that we played when it was sunny on 2 days or 2/14
End of explanation
print play_days["temperature"].mean()
print play_days["temperature"].min()
print play_days["temperature"].max()
Explanation: #3
*Compare the average, minimum and maximum temperature when you play tennis? *
End of explanation
print play_days["humidity"].mean()
print play_days["humidity"].min()
print play_days["humidity"].max()
Explanation: #4
Compare the average, minimum and maximum humidity when you play tennis?
End of explanation
pyplot.ylabel('Temperature')
pyplot.xlabel("Humidity")
pyplot.scatter(x=play_days["humidity"], y=play_days["temperature"], c='green')
no_play_days = df[df["play"]=="no"]
pyplot.scatter(x=no_play_days["humidity"], y=no_play_days["temperature"], c='red', marker="x")
pyplot.legend(['Play', "No Play"])
Explanation: #5
Plot the an scatter plot (x,y diagramm) of humidity (x) and temperature (y) when you play tennis compared to when you do not play tennis.
End of explanation
#these are in units of thousands, need to scale
df1 = pd.read_fwf("processed/st0009ts.txt", header=0, index_col=0, thousands=",").transpose()
df2 = pd.read_fwf("processed/st1019ts.txt", header=0, index_col=0, thousands=",").transpose()
df3 = pd.read_fwf("processed/st2029ts.txt", header=0, index_col=0, thousands=",").transpose()
df4 = pd.read_fwf("processed/st3039ts.txt", header=0, index_col=0, thousands=",").transpose()
df5 = pd.read_fwf("processed/st4049ts.txt", header=0, index_col=0, thousands=",").transpose()
df6 = pd.read_fwf("processed/st5060ts.txt", header=0, index_col=0, thousands=",").transpose()
df7 = pd.read_fwf("processed/st6070ts.txt", header=0, index_col=0, thousands=",").transpose()
df = pd.concat([df1, df2, df3, df4, df5, df6, df7])
#scale up to unit of 1
df = df.apply(lambda x: x*1000)
#for some reason, this dataset format uses '.'s in U.S. but doesn't for anything else. We'll normalize it here
df[["U.S."]]
df.rename(columns={'U.S.': 'US'}, inplace=True)
#the file format changes here
transform = lambda x: "19"+x[2:4]
df_9 = pd.read_fwf("processed/st7080ts.txt", header=0, index_col=0, thousands=",").transpose()
df_9.index = df_9.index.map(transform)
df_10 = pd.read_fwf("processed/st8090ts.txt", header=0, index_col=0, thousands=",").transpose()
df_10.index = df_10.index.map(transform)
df_10
df_2 = pd.concat([df_9, df_10])
Explanation: The only inferences I can make from the scatter plot above, is that you always play when the humidity is between 70 and 85. Temperature seems to play no part of the decision process when you go out to play as from teh scatter plot the plays and no play poionts are evenly distributed across the y axis (Temperature).
#2
We have a set of 8 files where the first 7 and the last 2 have a different format. First I removed the header information from the files and removed any superfolous line breaks, I then read them into pandas in two respective groups. I then had to normalize the dates of teh second dataset to match the dates of the first. Also I had to normalize the values of the first dataset b/c they were in units of 1000, so I made it in units of 1.
Cleaning & Normalization
End of explanation
# now merge the two together to get the compleete mergered df
df = pd.concat([df, df_2])
df=df.sort_index() #sort
Explanation: Merging
End of explanation
df[["CA", "AK"]].plot()
Explanation: Plot CA vs AK
End of explanation
df["New England"] = df[["CT", "ME", "MA", "NH", "RI", "VT"]].sum(axis=1)
df["Southwest"] = df[["AZ", "CA", "CO", "NV", "NM", "TX", "UT"]].sum(axis=1) 
df[["New England", "Southwest"]].plot()
Explanation: New England vs Southwest
In order to plot these values I have to do some feature engineering to create columns for the respective regions that are not in the original dataset. 
For New England I used: CT, ME, MA, NH, RI, VT
For the Southwest, I used: AZ, CA, CO, NV, NM, TX, UT
Feature Engineering
End of explanation
#remove a few composite columns:
df.drop('US', axis=1, inplace=True)
df.drop('Southwest', axis=1, inplace=True)
df.drop('New England', axis=1, inplace=True)
delta = {}
rel_delta={}
for state in df.columns:
    delta[state]=df[state].iloc[-1] - df[state].iloc[50]
    rel_delta[state] = (df[state].iloc[-1] - df[state].iloc[50]) / df[state].iloc[50]*1. * 100
    
ddf=pd.DataFrame(delta, index=["delta"]).transpose()
ddf = ddf.sort(["delta"], ascending=False)
ddf.head()
Explanation: Greatest Change in Population
We can quantify population growth in direct terms or relativly using percentages:
Magnitude Delta
We don't have measurements for Alaska  until 1950, so if we compare growth from 1950 in terms of pure magnitude, the top states are shown below:
End of explanation
ddp=pd.DataFrame(rel_delta, index=["% change"]).transpose()
ddp = ddp.sort(["% change"], ascending=False)
ddp.head()
Explanation: As you can see from teh table above, CA had the largest growth in terms of raw numbers for the time period. However, we can gain additional insites by looking at percentatge growth.
Relative Growth
We can also measure growth as a percenagte difference: As you can see Nevada had the largest percent growth from 1950 to 1900
End of explanation
ddp.tail(n=10)
Explanation: Some states had no net growth and some had negative growth:
End of explanation
from sklearn import tree
import numpy as np
wine = np.loadtxt("wine.data", delimiter=',')
#Get the targets (first column of file)
Y = wine[:, 0]
#Remove targets from input data
X = wine[:, 1:]
Explanation: 3
4
We will use a Decision Tree to build a model that will allow us to classify wines into one of three categories.
End of explanation
#lets split into a test and training set
from sklearn.cross_validation import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.25, random_state=9)
Explanation: Test/Train Split
Split the dat set into 75% for training and 25% for testing
End of explanation
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X_train, Y_train)
clf.score(X_test, Y_test)
Explanation: Train Model
I used the defaults for the DecisionTreeClasifier package in scikit
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(3)
    plt.xticks(tick_marks, ["1", "2", "3"], rotation=45)
    plt.yticks(tick_marks, ["1", "2", "3"])
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
# Compute confusion matrix
from sklearn.metrics import confusion_matrix
y_true = Y_test
y_pred = clf.predict(X_test)
cm = confusion_matrix(y_true, y_pred)
np.set_printoptions(precision=2)
print('Confusion matrix, without normalization')
print(cm)
plt.figure()
plot_confusion_matrix(cm)
plt.show()
Explanation: Evaluation
Evaluate based on confusion matrix how well the model performed on training vs. testing.
End of explanation
odf = pd.read_csv("hmwk_1_data/AHW_1.csv")
odf.head()
Explanation: As you can see from the confusion matrix, inputs of Class 1 & 2 were perfectly classified. There were only 2 mistakes on Class 3.
5
End of explanation
odf["Age"].plot(kind="hist")
odf["Age"].describe()
odf["Weight"].plot(kind="hist")
odf["Weight"].describe()
odf["Height"].plot(kind="hist")
odf["Height"].describe()
Explanation: What are the statistical distributions of variables using no class?
End of explanation
odf.isnull().sum()
Explanation: How much missing data is there?
End of explanation
male = odf["Sex"]=="M"
female = odf["Sex"]=="F"
odf[male]["Age"].plot(kind="hist")
odf[female]["Age"].plot(kind="hist")
odf[male]["Weight"].plot(kind="hist")
odf[female]["Weight"].plot(kind="hist")
odf[male]["Height"].plot(kind="hist")
odf[female]["Height"].plot(kind="hist")
Explanation: How do distributions differ by each gender?
End of explanation
odf.describe()
Explanation: Describe summary statistics for each attribute.
End of explanation
from pandas.tools.plotting import scatter_matrix
pd.scatter_matrix(odf, alpha=0.2, figsize=(10, 10), diagonal='kde')
Explanation: Visualize potential difference via the scatter plots.
Are there any ‘high’ correlations between variables?
We can see a correlation between height and weight
End of explanation
odf["lbs"] = odf["Weight"] * 2.20462
odf.head()
pd.scatter_matrix(odf, alpha=0.2, figsize=(10, 10), diagonal='kde')
Explanation: Create a new variable for the weight in lbs
Creating a new column for pounds obviously does not create any new correlations b/c it simply a linear combination w/ the kg weight.
End of explanation
odf["w+h"] = odf["Weight"] + odf["Height"]
odf.drop('lbs', axis=1, inplace=True)
odf.head()
pd.scatter_matrix(odf, alpha=0.2, figsize=(10, 10), diagonal='kde')
Explanation: Add new variable w​eight + height.
End of explanation
odf["BMI"] = odf["Weight"] / ((odf["Height"]*0.01)**2)
odf.head()
odf[male]["BMI"].plot(kind="hist")
odf[female]["BMI"].plot(kind="hist")
print odf[male]["BMI"].describe()
print
print odf[female]["BMI"].describe()
Explanation: BMI
There appears to be obese males and females in the dataset
End of explanation
sports = list(set(odf["Sport"]))
sports
# choose 3 random sports
sports
import random
random_sports = random.sample(sports, 3)
for sport in random_sports:
    sport_selector = odf["Sport"] == sport
    odf[sport_selector].plot(kind="scatter", x="Height", y="Weight", marker='x')
Explanation: Split Data By Sport
End of explanation
<END_TASK>