<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 PyConUK 2016
Ian Ozsvald, 2016-09-13
Goal
Step1:  Build a first classifier with a single input
Step2:  Confusion Matrix
Step3:  Create a Dummy Classifier
If we can't outperform this then our machine learning either isn't doing anything useful or we've made a mistake.
Step4:  Create a RandomForest Classifier
This is a fairly powerful classifier that operates with few data-preparation requirements.
Step5:  Try Pclass on its own
Step6:  Add Pclass
Actually this hardly helps things at all
Step7:  Replace NaN for Age with a sentinel value
If we replace NaN values with a sentinel value that's chosen to be 'a long way away' from the next nearest real value, we'll be able to spot the sentinel when we're debugging our data and the RandomForest classifier will be able to use this column of data as a feature.
Step8:  Add new features
Step9:  Steps you might take to improve things
Add more features (do this first) such as
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
# https://www.kaggle.com/c/titanic/data
df = pd.read_csv("titanic_train.csv")
confusion_matrix_labels = ['Died', 'Survived']
df.head()
print("Survived isnull:", df['Survived'].isnull().sum())
print("Sex isnull:", df['Sex'].isnull().sum())
print("Pclass isnull:", df['Pclass'].isnull().sum())
print("Age isnull:", df['Age'].isnull().sum())
print("Fare isnull:", df['Fare'].isnull().sum())
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
_=sns.countplot(df['Survived'])
pc_survived = (1.0-(df['Survived'].sum() / df.shape[0]))*100
_=plt.title("{:2.1f}% of passengers died".format(pc_survived))
df['Sex'].value_counts()
# convert 'male' 'female' to boolean indicator
df['is_female'] = df['Sex']=="female"
sns.countplot(df['is_female']) 
_=plt.title("Approx. twice as many men as women")
sns.pointplot(data=df, x='Sex', y='Survived', ci=None)
_=plt.title("Pointplot for mean survival vs Sex")
sns.pointplot(data=df, x='Pclass', y='Survived', ci=None)
_=plt.title("Pointplot of Pclass vs Survival")
sns.pointplot(data=df, x='Sex', y='Survived', hue='Pclass', ci=None)
_=plt.title("Pointplot of Pclass and Sex vs Survival")
_=sns.boxplot(data=df, x="Survived", y="Age", hue="Sex")
_=plt.title("Boxplots for Age and Sex against Survival")
# note NaN Ages are dropped by seaborn
# first, second (median) and third quartiles for the box
# extents to 1.5 IQR and any outliers
_=sns.stripplot(data=df, x="Survived", y="Age", hue="Sex", jitter=True, alpha=0.5)
_=plt.title("Stripplots for Sex vs Survived")
# first, second (median) and third quartiles for the box
# extents to 1.5 IQR and any outliers
Explanation: PyConUK 2016
Ian Ozsvald, 2016-09-13
Goal: Introduce the basic concepts of 2-class classification using scikit-learn and a well-documented problem, also providing a workflow so you can go and do similar things on your own data.
Locations: https://github.com/ianozsvald/pyconuk_using_sklearn_classification
Blog write-up: (on http://ianozsvald.com/)
Follow-up: See great 2 hour tutorial with many Notebooks from PyCon 2014: https://github.com/savarin/pyconuk-introtutorial
End of explanation
X = df[['is_female']]
y = df['Survived']
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    train_size=0.7, 
                                                    random_state=0)
print("Training and test set sizes:", X_train.shape, X_test.shape)
Explanation: Build a first classifier with a single input
End of explanation
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
#np.set_printoptions(precision=2)
def plot_confusion_matrix_with_annots(cm, labels, cmap=plt.cm.Blues, title="Confusion matrix", ax=None):
    '''Plot a normalised confusion matrix, annotated with classification counts'''
    cm_df = pd.DataFrame(data=cm, columns=labels, index=labels).sort_index(ascending=False)
    cmn_df = cm_df.apply(lambda x: x/x.sum(), axis=1)
    if ax is None:
        fig, ax = plt.subplots()
    fmt = "" # disable string formatting, display the raw counts
    sns.heatmap(cmn_df, annot=cm_df.values, vmin=0, vmax=1, cmap=cmap, ax=ax, fmt=fmt)
    plt.ylabel('True label')
    plt.yticks(rotation=90)
    plt.xlabel('Predicted label')
    plt.xticks(rotation=45)
    plt.title(title)
Explanation: Confusion Matrix
End of explanation
from sklearn.dummy import DummyClassifier
clf_dummy = DummyClassifier(strategy="most_frequent")
clf_dummy.fit(X_train, y_train)
print("Scoring on testing data:", clf_dummy.score(X_test, y_test))
cm = confusion_matrix(y_test, clf_dummy.predict(X_test))
plot_confusion_matrix_with_annots(cm, labels=confusion_matrix_labels)
Explanation: Create a Dummy Classifier
If we can't outperform this then our machine learning either isn't doing anything useful or we've made a mistake.
End of explanation
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
y_test[:10].values # get numpy array from Series
y_predicted = clf.predict(X_test) # numpy array
y_predicted[:10]
print("Scoring on training data:", clf.score(X_train, y_train))
print("Scoring on testing data:", clf.score(X_test, y_test))
cm = confusion_matrix(y_test, clf.predict(X_test))
plot_confusion_matrix_with_annots(cm, labels=confusion_matrix_labels)
from sklearn.cross_validation import cross_val_score
cv_scores = cross_val_score(clf, X, y)
print("Mean score {:0.3f} +/- {:0.2f}".format(cv_scores.mean(), cv_scores.std()*2))
Explanation: Create a RandomForest Classifier
This is a fairly powerful classifier that operates with few data-preparation requirements.
End of explanation
X = df[['Pclass']]
y = df['Survived']
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=0)
print("Training and test set sizes:", X_train.shape, X_test.shape)
clf.fit(X_train, y_train)
print(clf.score(X_train, y_train))
print(clf.score(X_test, y_test))
# not very predictive!
cm = confusion_matrix(y_test, clf.predict(X_test))
plot_confusion_matrix_with_annots(cm, labels=confusion_matrix_labels)
from sklearn.cross_validation import cross_val_score
cv_scores = cross_val_score(clf, X, y)
print("Mean score {:0.3f} +/- {:0.2f}".format(cv_scores.mean(), cv_scores.std()*2))
Explanation: Try Pclass on its own
End of explanation
X = df[['is_female', 'Pclass']]
y = df['Survived']
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    train_size=0.7, 
                                                    random_state=0)
print("Training and test set sizes:", X_train.shape, X_test.shape)
clf.fit(X_train, y_train)
print("Scoring on training data:", clf.score(X_train, y_train))
print("Scoring on testing data:", clf.score(X_test, y_test))
cm = confusion_matrix(y_test, clf.predict(X_test))
plot_confusion_matrix_with_annots(cm, labels=confusion_matrix_labels)
cv_scores = cross_val_score(clf, X, y)
print("Mean score {:0.3f} +/- {:0.2f}".format(cv_scores.mean(), cv_scores.std()*2))
Explanation: Add Pclass
Actually this hardly helps things at all
End of explanation
df['Age'].min()
df['Age'].head(10)
df['Age'].isnull().sum()
df['Age_sentinel'] = df['Age'].fillna(-100)
df['Age_sentinel'].head(10)
# note Age or Pclass by itself isn't so useful, combined they're useful
feature_names = ['is_female', 'Age_sentinel', 'Pclass']
X = df[feature_names]
y = df['Survived']
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=0)
clf.fit(X_train, y_train)
print(clf.score(X_train, y_train))
print(clf.score(X_test, y_test))
# no real improvement!
cm = confusion_matrix(y_test, clf.predict(X_test))
plot_confusion_matrix_with_annots(cm, labels=confusion_matrix_labels)
cv_scores = cross_val_score(clf, X, y)
print("Mean score {:0.3f} +/- {:0.2f}".format(cv_scores.mean(), cv_scores.std()*2))
Explanation: Replace NaN for Age with a sentinel value
If we replace NaN values with a sentinel value that's chosen to be 'a long way away' from the next nearest real value, we'll be able to spot the sentinel when we're debugging our data and the RandomForest classifier will be able to use this column of data as a feature.
End of explanation
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=100)
df['is_mr'] = df['Name'].str.count(", Mr.")
df['family_size'] = df['SibSp'] + df['Parch']
# note Age or Pclass by itself isn't so useful, combined they're useful
feature_names = ['is_female', 'Age_sentinel', 'Pclass', 
                 'is_mr', 'family_size', 'Fare']
X = df[feature_names]
y = df['Survived']
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    train_size=0.7, 
                                                    random_state=0)
clf.fit(X_train, y_train)
print("Scoring on training data:", clf.score(X_train, y_train))
print("Scoring on testing data:", clf.score(X_test, y_test))
cm = confusion_matrix(y_test, clf.predict(X_test))
plot_confusion_matrix_with_annots(cm, labels=confusion_matrix_labels)
cv_scores = cross_val_score(clf, X, y)
print("Mean score {:0.3f} +/- {:0.2f}".format(cv_scores.mean(), cv_scores.std()*2))
import numpy as np
sns.barplot(feature_names, clf.feature_importances_, order=np.array(feature_names)[np.argsort(clf.feature_importances_)[::-1]])
_=plt.title("Relative feature importances")
Explanation: Add new features
End of explanation
X = df[['Pclass', 'Fare', 'is_mr', 'Fare']]
y = df['Survived']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=0)
clf.fit(X_train, y_train)
y_proba = clf.predict_proba(X_test)
max_error = abs(y_proba[:,1] - y_test)
max_error = max_error.sort_values(ascending=False)
max_error.head()
plt.plot(range(max_error.shape[0]), max_error)
plt.xlabel("Test set example")
plt.ylabel("Abs(error) per examples")
_=plt.title("Quite a lot of examples have >50% error")
# eyeball some of the worsts offenders - what ideas do these give you?
df.ix[max_error.index[:3]]
Explanation: Steps you might take to improve things
Add more features (do this first) such as:
is_miss, 
maybe the port of embarkation (encoded as a boolean)
Test if increasing n_estimators helps in the RandomForest
Could you use a different value for the NaN Ages such as a mean?
Find worst incorrect scores
Take a train/test split, for the test split find the maximally-wrong guesses based on the underlying probabilities.
End of explanation
<END_TASK>