<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Intro to Data Manipulation and Pandas
Dan Tamayo
Material draws from a blog working through the Titanic dataset by Dave Novelli, as well as the Pandas Cookbook by Julia Evans
Follow Along
In a terminal, navigate to the directory where you want to add the MachineLearning folder.  Then
cd /path/to/MachineLearning
git pull
cd Day2
source activate ml
jupyter notebook TitanicPandas.ipynb
Loading a Dataset
Step1:  Magic Pandas
Selecting Columns
Step2:  Data at a glance
Step3:  Plotting
Step4:  Filtering
Step5:  Boolean Masks
Step6:  Filtering Dataframes
Step7:  Memory Issues
Step8:  Selecting specific columns
df.loc[criterion, columns]
Step9:  Combining criteria / columns
Step10:  Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
How to Find Valid Values
Step11:  Wrong way
Step12:  Right Way
Step13:  Approach 1
Step14:  How much data is left?
Step15:  Approach 2
Step16:  Approach 3
Step17:  Categorical Variables
Step18:  Approach 4
Step19:  Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Step20:  Approach 1
Step21:  Approach 2
Step22:  What are the classes?
Step23:  Continuous Features
Step24:  Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Underneath, Pandas series are numpy arrays
Step25:  Getting a pipeline ready for sklearn
Combining Data Frames
Step26:  Indexing
Step27:  Putting it all together
Step28:  Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Step29:  Feature Engineering
Parsing Alphanumeric Features
Step30:  Regular Expressions
Step31:  Apply a function to all rows to generate a new feature
Step32:  What to do with the name?
Number of names
Step33:  Title
Step34:  Final Data Processing Pipeline
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
df = pd.read_csv('data/train.csv')
df.head()
df.tail()
df.shape
Explanation: Intro to Data Manipulation and Pandas
Dan Tamayo
Material draws from a blog working through the Titanic dataset by Dave Novelli, as well as the Pandas Cookbook by Julia Evans
Follow Along
In a terminal, navigate to the directory where you want to add the MachineLearning folder.  Then
cd /path/to/MachineLearning
git pull
cd Day2
source activate ml
jupyter notebook TitanicPandas.ipynb
Loading a Dataset
End of explanation
df['Fare'].head()
df[['Fare', 'Sex']].head()
Explanation: Magic Pandas
Selecting Columns
End of explanation
df['Sex'].value_counts()
df['Age'].median()
Explanation: Data at a glance
End of explanation
%matplotlib inline
import seaborn
fig = df['Pclass'].hist()
fig = df.hist(figsize=(15,5))
Explanation: Plotting
End of explanation
df.head()
Explanation: Filtering
End of explanation
mask = df['Embarked'] == 'C'
mask.head()
Explanation: Boolean Masks
End of explanation
df_filter = df[mask] # df_filter = df[df['Embarked'] == 'C']
df_filter.head()
Explanation: Filtering Dataframes
End of explanation
df_filter = df.loc[df['Embarked'] == 'C']
df_filter.head()
Explanation: Memory Issues
End of explanation
test = df.loc[df['Age'] > 30., ['Age', 'Fare', 'Sex']]
test.head()
Explanation: Selecting specific columns
df.loc[criterion, columns]
End of explanation
test = df.loc[(df['Age'] > 30.) & (df['Fare'] < 50.), 'Age':'Fare']
test.head()
Explanation: Combining criteria / columns
End of explanation
df.head()
Explanation: Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
How to Find Valid Values
End of explanation
import numpy as np
df[df['Cabin']!=np.nan]
Explanation: Wrong way:
End of explanation
df.loc[df['Cabin'].notnull()]
Explanation: Right Way:
End of explanation
df.notnull().head()
df_filter = df.loc[df.notnull().all(axis=1)]
Explanation: Approach 1:  Throw out any rows with missing data values
End of explanation
df_filter.shape
Explanation: How much data is left?
End of explanation
df.loc[df['Cabin'].isnull(), 'Cabin'] = 'U0'
df.head()
Explanation: Approach 2:  Assign missing identifier
End of explanation
df.loc[df['Fare'].isnull()].shape
df['Fare'].median()
df.loc[df['Fare'].isnull(), 'Fare'] = df['Fare'].median()
Explanation: Approach 3: Assign the average/median/mode value
End of explanation
df.loc[df['Embarked'].isnull()].shape
df['Embarked'].mode()
df['Embarked'].mode()[0]
df.loc[df['Embarked'].isnull(), 'Embarked'] = df['Embarked'].dropna().mode()[0]
Explanation: Categorical Variables
End of explanation
for col in df.columns:
    print("NaNs in column {0} = {1}".format(col, df.loc[df[col].isnull()].shape[0]))
df.loc[df['Age'].isnull(), 'Age'] = df['Age'].mean()
Explanation: Approach 4: Fit a regression model to predict missing values
End of explanation
df.head()
Explanation: Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
End of explanation
print(df['Embarked'].unique())
dummies_df = pd.get_dummies(df['Embarked'])
dummies_df.head()
def addEmbarked(name):
    return 'Embarked_' + name
dummies_df = dummies_df.rename(columns=addEmbarked)
dummies_df.head()
df = pd.concat([df, dummies_df], axis=1)
df.head()
Explanation: Approach 1:  Creating Dummy (Binary) Variables
End of explanation
df['EmbarkedNum'] = pd.factorize(df['Embarked'])[0]
df.head(6)
Explanation: Approach 2: Factorizing (Make single multi-class feature)
End of explanation
pd.factorize(df['Sex'])
df['Female'] = pd.factorize(df['Sex'])[0]
df.head()
Explanation: What are the classes?
End of explanation
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(df[['Age', 'Fare']]) # .fit(df) to scale all numerical columns
print("Means = {0}".format(scaler.mean_))
print("Stdevs = {0}".format(scaler.scale_))
df[['Age', 'Fare']] = scaler.transform(df[['Age', 'Fare']])
df.head()
Explanation: Continuous Features: Feature Scaling
End of explanation
fares = df['Fare'].values
type(fares)
Explanation: Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
Underneath, Pandas series are numpy arrays
End of explanation
input_df = pd.read_csv('data/train.csv')
submit_df = pd.read_csv('data/test.csv')
print(input_df.shape)
print(submit_df.shape)
submit_df.head()
input_df.head()
Explanation: Getting a pipeline ready for sklearn
Combining Data Frames
End of explanation
submit_df = pd.read_csv('data/test.csv', index_col=0)
submit_df.head()
input_df = pd.read_csv('data/train.csv', index_col=0)
submit_df = pd.read_csv('data/test.csv', index_col=0)
df = pd.concat([input_df, submit_df])
df.tail()
print(df.shape[1], "columns")
print(df.shape[0], "rows")
print(df.columns.values)
Explanation: Indexing
End of explanation
def process_data(df):
    df['Female'] = pd.factorize(df['Sex'])[0]
    df.loc[df['Age'].isnull(), 'Age'] = df['Age'].mean()
    df.loc[df['Fare'].isnull(), 'Fare'] = df['Fare'].median()
    df.loc[df['Cabin'].isnull(), 'Cabin'] = 'U0'
    df.loc[df['Embarked'].isnull(), 'Embarked'] = df['Embarked'].dropna().mode()[0]
    
    dummies_df = pd.get_dummies(df['Embarked'])
    def addEmbarked(name):
        return 'Embarked_' + name
    dummies_df = dummies_df.rename(columns=addEmbarked)
    df = pd.concat([df, dummies_df], axis=1)
    df['EmbarkedNum'] = pd.factorize(df['Embarked'])[0]
    
    return df
df = process_data(df)
df.tail()
features = ['Age', 'Fare', 'Parch', 'Pclass', 'SibSp', 'Female', 'EmbarkedNum']
df_test = df.loc[df['Survived'].isnull(), features]
df_train = df.loc[df['Survived'].notnull(), features+['Survived']]
df_train.head()
Explanation: Putting it all together
End of explanation
X_train = df_train[features].values
y_train = df_train['Survived'].values
print(X_train[0:5])
print(y_train[0:5])
print("X has {0} rows".format(X_train.shape[0]))
print("y has {0} rows".format(y_train.shape[0]))
Explanation: Requirements for working with data in scikit-learn
Features should not have missing values
Features and response should be numeric
Features and response should be NumPy arrays
Features and response are separate objects
Features and response should have specific shapes
End of explanation
df['Cabin'].unique()
Explanation: Feature Engineering
Parsing Alphanumeric Features
End of explanation
import re
def getDeck(cabin):
    match = re.search("([A-Z])", cabin)
    return match.group(1) if match is not None else None
def getCabinNum(cabin):
    match = re.search("([0-9]+)", cabin)
    return match.group(1) if match is not None else None
print(getDeck('C237'))
print(getCabinNum('C237'))
Explanation: Regular Expressions
End of explanation
df['Deck'] = df['Cabin'].map(getDeck)
df['CabinNum'] = df['Cabin'].map(getCabinNum)
df.head()
df['CabinNum'].isnull().value_counts()
df.loc[df['CabinNum'].isnull(), 'CabinNum'] = 0
df['Deck'].isnull().value_counts()
df['DeckNum'] = pd.factorize(df['Deck'])[0]
Explanation: Apply a function to all rows to generate a new feature
End of explanation
testname = df.loc[1, 'Name']
print(testname)
re.split(' ', testname)
def numNames(name):
    return len(re.split(' ', name))
df['NumNames'] = df['Name'].map(numNames)
df.head()
Explanation: What to do with the name?
Number of names
End of explanation
testname
re.search(", (.+?)\.", testname).group(1)
def getTitle(name):
    match = re.search(", (.*?)\.", name)
    return match.group(1) if match is not None else None
df['Title'] = df['Name'].map(getTitle)
df.head()
df.loc[df['Title'] == 'Jonkheer', 'Title'] = 'Master'
df.loc[df['Title'].isin(['Ms', 'Mlle']), 'Title'] = 'Miss'
df.loc[df['Title'] == 'Mme', 'Title'] = 'Mrs'
df.loc[df['Title'].isin(['Capt', 'Don', 'Major', 'Col', 'Sir']), 'Title'] = 'Sir'
df.loc[df['Title'].isin(['Dona', 'Lady', 'the Countess']), 'Title'] = 'Lady'
df['Title'].value_counts()
df = pd.concat([df, pd.get_dummies(df['Title']).rename(columns=lambda x: 'Title_' + str(x))], axis=1)
df.head()
Explanation: Title
End of explanation
def process_data(df):
    df['Female'] = pd.factorize(df['Sex'])[0]
    df.loc[df['Age'].isnull(), 'Age'] = df['Age'].mean()
    df.loc[df['Fare'].isnull(), 'Fare'] = df['Fare'].median()
    df.loc[df['Cabin'].isnull(), 'Cabin'] = 'U0'
    df.loc[df['Embarked'].isnull(), 'Embarked'] = df['Embarked'].dropna().mode()[0]
    
    dummies_df = pd.get_dummies(df['Embarked'])
    def addEmbarked(name):
        return 'Embarked_' + name
    dummies_df = dummies_df.rename(columns=addEmbarked)
    df = pd.concat([df, dummies_df], axis=1)
    df['EmbarkedNum'] = pd.factorize(df['Embarked'])[0]
    
    import re
    def getDeck(cabin):
        match = re.search("([A-Z])", cabin)
        return match.group(1) if match is not None else None
    def getCabinNum(cabin):
        match = re.search("([0-9]+)", cabin)
        return match.group(1) if match is not None else None
    
    df['Deck'] = df['Cabin'].map(getDeck)
    df['DeckNum'] = pd.factorize(df['Deck'])[0]
    df['CabinNum'] = df['Cabin'].map(getCabinNum)
    df.loc[df['CabinNum'].isnull(), 'CabinNum'] = 0
    
    def numNames(name):
        return len(re.split(' ', name))
    df['NumNames'] = df['Name'].map(numNames)
    
    def getTitle(name):
        match = re.search(", (.*?)\.", name)
        return match.group(1) if match is not None else None
    df['Title'] = df['Name'].map(getTitle)
    
    df.loc[df['Title'] == 'Jonkheer', 'Title'] = 'Master'
    df.loc[df['Title'].isin(['Ms', 'Mlle']), 'Title'] = 'Miss'
    df.loc[df['Title'] == 'Mme', 'Title'] = 'Mrs'
    df.loc[df['Title'].isin(['Capt', 'Don', 'Major', 'Col', 'Sir']), 'Title'] = 'Sir'
    df.loc[df['Title'].isin(['Dona', 'Lady', 'the Countess']), 'Title'] = 'Lady'
    df = pd.concat([df, pd.get_dummies(df['Title']).rename(columns=lambda x: 'Title_' + str(x))], axis=1)
    
    return df
input_df = pd.read_csv('data/train.csv', index_col=0)
submit_df = pd.read_csv('data/test.csv', index_col=0)
df = pd.concat([input_df, submit_df])
df = process_data(df)
features = ['Age', 'Fare', 'Parch', 'Pclass', 'SibSp', 'Female', 'EmbarkedNum', 'DeckNum', 'CabinNum', 'NumNames', 'Title_Dr','Title_Lady','Title_Master','Title_Miss','Title_Mr','Title_Mrs','Title_Rev','Title_Sir']
df_test = df.loc[df['Survived'].isnull(), features]
df_train = df.loc[df['Survived'].notnull(), features+['Survived']]
df_train.to_csv('data/train_processed.csv') # NOT FEATURE SCALED!
df_test.to_csv('data/test_processed.csv')
Explanation: Final Data Processing Pipeline
End of explanation
<END_TASK>