<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Read the data
Step1:  Filter the data
Choose only some sports
Choose only some columns as they are not all relevant, e.g. latitude, longtiude, timezone
Deal with NaN values in the cadence columns by training two models
Step2:  Scale and encode the data
Step3:  Split the data
Step4:  Train the model
Step5:  Test the model
<ASSISTANT_TASK:>
Python Code:
database_path = os.path.join('..', 'data', 'Ana', 'database', 'Garmin-Ana-180226-1.csv')
print(os.path.abspath(database_path))
data = pd.read_csv(database_path)
data.columns = [s.replace('_', ' ') for s in data.columns]
data = data.set_index('file name')
data.head()
data.shape
data.groupby('sport').agg({'activity':'count'})
Explanation: Read the data
End of explanation
data = data[data['sport'].isin(['cycling', 'walking', 'running', 'hiking', 'driving', 'train', 'squash', 'yoga'])].copy()
data.groupby('sport').agg({'activity':'count'})
# these are the columns we want to use for the machine learning model
columns = ['avg speed', 'total distance', 'total elapsed time', 'avg heart rate', 'max heart rate', 
           'avg cadence', 'max cadence', 'total calories']
# for column in ['avg cadence', 'max cadence']:
#     data[column] = data[column].fillna(data[column].mean()) # fill cadence NaNs with the average of that column
#     data[column] = data[column].fillna(0) # fill NaN values with 0 (this is not ideal as cadence values are typically 50-60)
data = data.dropna(axis=0, subset=['avg cadence', 'max cadence']) # drop rows in which the cadence is NaN
# data = data[data['avg cadence'].isna()] # keep rows in which the cadence is NaN
# data = data.dropna(axis=1) # remove columns with NaN values (should be just the cadence columns)
data.groupby('sport').agg({'activity':'count'})
Explanation: Filter the data
Choose only some sports
Choose only some columns as they are not all relevant, e.g. latitude, longtiude, timezone
Deal with NaN values in the cadence columns by training two models:
One model will deal with activities with contain cadence data.
The other model will deal with activities which do NOT contain cadence data.
There are many more cycling activities than any other sport. This might bias the model.
End of explanation
features = data[list(set(columns) & set(data.columns))].copy()
features.shape
features_scaler = preprocessing.MinMaxScaler()
# features_scaler = preprocessing.StandardScaler()
features_scaler.fit(features)
sports = data['sport'].copy()
sports_encoder = preprocessing.LabelEncoder()
sports_encoder.fit(sports)
Explanation: Scale and encode the data
End of explanation
features_train, features_test, sports_train, sports_test = model_selection.train_test_split(features, sports)
Explanation: Split the data
End of explanation
# sport_classifier = tree.DecisionTreeClassifier()
sport_classifier = ensemble.RandomForestClassifier(n_estimators=10)
sport_classifier.fit(features_scaler.transform(features_train), sports_encoder.transform(sports_train))
Explanation: Train the model
End of explanation
predictions = features_test.copy()
predictions['real sport'] = sports_test
predictions['predicted sport'] = sports_encoder.inverse_transform(
    sport_classifier.predict(features_scaler.transform(features_test)))
# add columns indicating the probabilities for each sport
probablities = sport_classifier.predict_proba(features_scaler.transform(features_test))
for i, column in enumerate(['%s probability' % s for s in sports_encoder.classes_]):
    predictions[column] = probablities[:, i]
# only keep rows in which the prediction was wrong
predictions = predictions[predictions['real sport'] != predictions['predicted sport']] 
# predictions = predictions.sort_values('cycling probability', ascending=False)
# predictions = predictions[predictions['cycling probability'] < 1]
predictions.head(20)
x = 'avg speed'
y = 'avg heart rate'
sport_colours = {'walking': 'g', 'cycling': 'b', 'driving': 'k', 'yoga': 'm', 'mixed':'y', 
                 'hiking': 'orange', 'running': 'r', 'train': 'c', 'other': 'purple', 'squash': 'grey',
                 'water': 'pink'}
fig, ax = plt.subplots(1, 1, figsize=[15,10])
for sport in sorted(predictions['real sport'].unique()):
    ax.scatter(predictions[predictions['real sport'] == sport][x], 
               predictions[predictions['real sport'] == sport][y],
               marker='o', s=100, alpha=0.5,
               color=sport_colours[sport],
               label="REAL - %s" % sport)
    
for sport in sorted(predictions['predicted sport'].unique()):
    ax.scatter(predictions[predictions['predicted sport'] == sport][x], 
               predictions[predictions['predicted sport'] == sport][y],
               marker='o', s=500, alpha=0.5, linewidth=5,
               color=sport_colours[sport], facecolors='none',
               label="PREDICTED - %s" % sport)
ax.set_xlabel(x)
ax.set_ylabel(y)
# ax.set_xlim([0,5])
# ax.set_ylim([0,5000])
ax.legend()
ax.grid()
sport_classifier.score(features_scaler.transform(features_test), sports_encoder.transform(sports_test))
Explanation: Test the model
End of explanation
<END_TASK>