<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Pandas and Scikit-learn
Pandas is a Python library that contains high-level data structures and manipulation tools designed for data analysis. Think of Pandas as a Python version of Excel. Scikit-learn, on the other hand, is an open-source machine learning library for Python.
While Scikit-learn does a lot of the heavy lifting, what's equally important is ensuring that raw data is processed in such a way that we are able to 'feed' it to Scikit-learn. Hence, the ability to manipulate raw data with Pandas makes it an indispensible part of our toolkit.
Kaggle
Kaggle is the leading platform for data science competitions. Participants compete for cash prizes by submitting the best predictive model to problems posted on the competition website.
https
Step1:  Pandas - Cleaning data
We then review a selection of the data.
Step2:  We notice that the columns describe features of the Titanic passengers, such as age, sex, and class. Of particular interest is the column Survived, which describes whether or not the passenger survived. When training our model, what we are essentially doing is assessing how each feature impacts whether or not the passenger survived (or if the feature makes an impact at all).
Exercise
Step3:  Next, we review the type of data in the columns, and their respective counts.
Step4:  We notice that the columns Age and Embarked have NaNs or missing values. As previously discussed, we take the approach of simply removing the rows with missing values.
Step5:  Question
If you were to fill in the missing values, with what values would you fill them with? Why?
Scikit-learn only takes numerical arrays as inputs. As such, we would need to convert the categorical columns Sex and Embarked into numerical ones. We first review the range of values for the column Sex, and create a new column that represents the data as numbers.
Step6:  Similarly for Embarked, we review the range of values and create a new column called Port that represents, as a numerical value, where each passenger embarks from.
Step7:  Question
- What problems might we encounter by mapping C, S, and Q in the column Embarked to the values 1, 2, and 3? In other words, what does the ordering imply? Does the same problem exist for the column Sex?
Now that we have numerical columns that encapsulate the information provided by the columns Sex and Embarked, we can proceed to drop them from our data set.
Step8:  We review the columns our final, processed data set.
Step9:  For convenience, we move the column Survived to the left-most column. We note that the left-most column is indexed as 0.
Step10:  In our final review of our training data, we check that (1) the column Survived is the left-most column (2) there are no NaN values, and (3) all the values are in numerical form.
Step11:  Finally, we convert the processed training data from a Pandas dataframe into a numerical (Numpy) array.
Step12:  Scikit-learn - Training the model
In this section, we'll simply use the model as a black box. We'll review more sophisticated techniques in later sections.
Here we'll be using the Random Forest model. The intuition is as follows
Step13:  We use the processed training data to 'train' (or 'fit') our model. The column Survived will be our second input, and the set of other features (with the column PassengerId omitted) as the first.
Step14:  Scikit-learn - Making predictions
We first load the test data.
Step15:  We then review a selection of the data.
Step16:  We notice that test data has columns similar to our training data, but not the column Survived. We'll use our trained model to predict values for the column Survived.
As before, we process the test data in a similar fashion to what we did to the training data.
Step17:  We now apply the trained model to the test data (omitting the column PassengerId) to produce an output of predictions.
Step18:  Pandas - Preparing for submission
We simply create a Pandas dataframe by combining the index from the test data with the output of predictions.
Step19:  We briefly review our predictions.
Step20:  Finally, we output our results to a .csv file.
Step21:  However, it appears that we have a problem. The Kaggle submission website expects "the solution file to have 418 predictions."
https
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
df = pd.read_csv('../data/train.csv')
Explanation: Pandas and Scikit-learn
Pandas is a Python library that contains high-level data structures and manipulation tools designed for data analysis. Think of Pandas as a Python version of Excel. Scikit-learn, on the other hand, is an open-source machine learning library for Python.
While Scikit-learn does a lot of the heavy lifting, what's equally important is ensuring that raw data is processed in such a way that we are able to 'feed' it to Scikit-learn. Hence, the ability to manipulate raw data with Pandas makes it an indispensible part of our toolkit.
Kaggle
Kaggle is the leading platform for data science competitions. Participants compete for cash prizes by submitting the best predictive model to problems posted on the competition website.
https://www.kaggle.com/competitions
Learning machine learning via Kaggle problems allows us to take a highly-directed approach because:
1. The problems are well-defined and the data is provided, allowing us to immediately focus on manipulating the data, and
2. The leaderboard allows us to keep track of how well we're doing.
In the following set of exercises, we will be reviewing the data from the Kaggle Titanic competition. Our aim is to make predictions on whether or not specific passengers on the Titanic survived, based on characteristics such as age, sex and class.
Section 1-0 - First Cut
We will start by processing the training data, after which we will be able to use to 'train' (or 'fit') our model. With the trained model, we apply it to the test data to make the predictions. Finally, we output our predictions into a .csv file to make a submission to Kaggle and see how well they perform.
It is very common to encounter missing values in a data set. In this section, we will take the simplest (or perhaps, simplistic) approach of ignoring the whole row if any part of it contains an NaN value. We will build on this approach in later sections.
Pandas - Extracting data
First, we load the training data from a .csv file. This is the similar to the data found on the Kaggle website: 
https://www.kaggle.com/c/titanic-gettingStarted/data
End of explanation
df.head(10)
Explanation: Pandas - Cleaning data
We then review a selection of the data.
End of explanation
df = df.drop(['Name', 'Ticket', 'Cabin'], axis=1)
Explanation: We notice that the columns describe features of the Titanic passengers, such as age, sex, and class. Of particular interest is the column Survived, which describes whether or not the passenger survived. When training our model, what we are essentially doing is assessing how each feature impacts whether or not the passenger survived (or if the feature makes an impact at all).
Exercise:
- Write the code to review the tail-end section of the data. 
We observe that the columns Name, Ticket and Cabin are, for our current purposes, irrelevant. We proceed to remove them from our data set.
End of explanation
df.info()
Explanation: Next, we review the type of data in the columns, and their respective counts.
End of explanation
df = df.dropna()
Explanation: We notice that the columns Age and Embarked have NaNs or missing values. As previously discussed, we take the approach of simply removing the rows with missing values.
End of explanation
df['Sex'].unique()
df['Gender'] = df['Sex'].map({'female': 0, 'male':1}).astype(int)
Explanation: Question
If you were to fill in the missing values, with what values would you fill them with? Why?
Scikit-learn only takes numerical arrays as inputs. As such, we would need to convert the categorical columns Sex and Embarked into numerical ones. We first review the range of values for the column Sex, and create a new column that represents the data as numbers.
End of explanation
df['Embarked'].unique()
df['Port'] = df['Embarked'].map({'C':1, 'S':2, 'Q':3}).astype(int)
Explanation: Similarly for Embarked, we review the range of values and create a new column called Port that represents, as a numerical value, where each passenger embarks from.
End of explanation
df = df.drop(['Sex', 'Embarked'], axis=1)
Explanation: Question
- What problems might we encounter by mapping C, S, and Q in the column Embarked to the values 1, 2, and 3? In other words, what does the ordering imply? Does the same problem exist for the column Sex?
Now that we have numerical columns that encapsulate the information provided by the columns Sex and Embarked, we can proceed to drop them from our data set.
End of explanation
cols = df.columns.tolist()
print(cols)
Explanation: We review the columns our final, processed data set.
End of explanation
cols = [cols[1]] + cols[0:1] + cols[2:]
df = df[cols]
Explanation: For convenience, we move the column Survived to the left-most column. We note that the left-most column is indexed as 0.
End of explanation
df.head(10)
df.info()
Explanation: In our final review of our training data, we check that (1) the column Survived is the left-most column (2) there are no NaN values, and (3) all the values are in numerical form.
End of explanation
train_data = df.values
Explanation: Finally, we convert the processed training data from a Pandas dataframe into a numerical (Numpy) array.
End of explanation
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators = 100)
Explanation: Scikit-learn - Training the model
In this section, we'll simply use the model as a black box. We'll review more sophisticated techniques in later sections.
Here we'll be using the Random Forest model. The intuition is as follows: each feature is reviewed to see how much impact it makes to the outcome. The most prominent feature is segmented into a 'branch'. A collection of branches is a 'tree'. The Random Forest model, broadly speaking, creates a 'forest' of trees and aggregates the results.
http://en.wikipedia.org/wiki/Random_forest
End of explanation
model = model.fit(train_data[0:,2:], train_data[0:,0])
Explanation: We use the processed training data to 'train' (or 'fit') our model. The column Survived will be our second input, and the set of other features (with the column PassengerId omitted) as the first.
End of explanation
df_test = pd.read_csv('../data/test.csv')
Explanation: Scikit-learn - Making predictions
We first load the test data.
End of explanation
df_test.head(10)
Explanation: We then review a selection of the data.
End of explanation
df_test = df_test.drop(['Name', 'Ticket', 'Cabin'], axis=1)
df_test = df_test.dropna()
df_test['Gender'] = df_test['Sex'].map({'female': 0, 'male':1})
df_test['Port'] = df_test['Embarked'].map({'C':1, 'S':2, 'Q':3})
df_test = df_test.drop(['Sex', 'Embarked'], axis=1)
test_data = df_test.values
Explanation: We notice that test data has columns similar to our training data, but not the column Survived. We'll use our trained model to predict values for the column Survived.
As before, we process the test data in a similar fashion to what we did to the training data.
End of explanation
output = model.predict(test_data[:,1:])
Explanation: We now apply the trained model to the test data (omitting the column PassengerId) to produce an output of predictions.
End of explanation
result = np.c_[test_data[:,0].astype(int), output.astype(int)]
df_result = pd.DataFrame(result[:,0:2], columns=['PassengerId', 'Survived'])
Explanation: Pandas - Preparing for submission
We simply create a Pandas dataframe by combining the index from the test data with the output of predictions.
End of explanation
df_result.head(10)
Explanation: We briefly review our predictions.
End of explanation
df_result.to_csv('../results/titanic_1-0.csv', index=False)
Explanation: Finally, we output our results to a .csv file.
End of explanation
df_result.shape
Explanation: However, it appears that we have a problem. The Kaggle submission website expects "the solution file to have 418 predictions."
https://www.kaggle.com/c/titanic-gettingStarted/submissions/attach
We compare this to our result.
End of explanation
<END_TASK>