<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Bicycle-stations Availability Prediction
Use a supervised machine learning algorithm to predict the availability for each bicyle-sharing stations in Lyon (France) based on the history data.
I use the tree method XGBoost to predict a "probability" of bikes availability for each station. A number close to 1. means that you have several available bikes. A number close to 0. means you don't have many bikes.
Some Imports
Step1:  The module prediction.py contains some functions dedicated to the bicyle-sharing stations predictions.
Step2:  Read and prepare the data
Step3:  Min and max dates of the timeseries
Step4:  Clean up some columns 
Step5:  Pipe some processing data functions 
Step6:  This is the final dataset. For further prediction, I could add some weather forecasts data to these features.
Prediction
Let's select a time window (start, stop) to a single prediction.
Two weeks of data for training
Predict the 'probability' for the next 30 minutes and 1 hour for every station.
Split the dataset into train and test dataset
Compute the RMSE (Root Mean Squared Error) between the observations (i.e. probability) and the predicted values
Step7:  The fit function create some data structure for the XGBoost from the train and test DataFrames (i.e. xgb.DMatrix), configure the model and launch it with the objective
Step8:  Store the results
Step9:  Some maps
CSV file with station coordinates
Step10:  Some stations were removed when the data were cleaned up. Remove them from the location data.
Step11:  Some station names contains the ' character. Replace it by the HTML code for folium.
Step12:  Select the prediction data for a specific timestamp
Step13:  See 
https
Step14:  Red stations if there are 0 or a few available bikes
Blue stations if there are several available bikes
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import numpy as np
import pandas as pd
import graphviz
from xgboost import plot_tree, plot_importance, to_graphviz
import matplotlib as mpl
from matplotlib import pyplot as plt
import seaborn as sns
import folium
%load_ext watermark
%watermark -d -v -p numpy,pandas,xgboost,matplotlib,folium -g -m -w
Explanation: Bicycle-stations Availability Prediction
Use a supervised machine learning algorithm to predict the availability for each bicyle-sharing stations in Lyon (France) based on the history data.
I use the tree method XGBoost to predict a "probability" of bikes availability for each station. A number close to 1. means that you have several available bikes. A number close to 0. means you don't have many bikes.
Some Imports
End of explanation
from sources.prediction import (datareader, complete_data, cleanup, bikes_probability,
                        time_resampling, prepare_data_for_training, fit, prediction)
Explanation: The module prediction.py contains some functions dedicated to the bicyle-sharing stations predictions.
End of explanation
DATAFILE = '../data/lyon.csv'
raw = datareader(DATAFILE)
raw.head()
Explanation: Read and prepare the data
End of explanation
print(raw.last_update.min())
print(raw.last_update.max())
Explanation: Min and max dates of the timeseries
End of explanation
df_clean = cleanup(raw)
df_clean.head()
Explanation: Clean up some columns : drop some lines with the 'CLOSED' status, drop duplicates, remove missing values, etc.
End of explanation
df = (df_clean.pipe(time_resampling)
      .pipe(complete_data)
      .pipe(bikes_probability))
df.head()
Explanation: Pipe some processing data functions :
time resampling to get data every 10 minutes (i.e. 10T)
add some columns : day of the week, hour of the day, etc.
compute the bikes (number) "probability". It's just a normalization for each station with num_avail_bikes / total
End of explanation
start = pd.Timestamp("2017-07-11T00:00:00") # Tuesday
predict_date = pd.Timestamp("2017-07-26T10:00:00") # wednesday
# predict the next 30 minutes
freq = '30T'
# number of predictions at 'predict_date'.
# Here, the next 30 minutes and the next hour (30 minutes + 30 minutes).
# If you want to predict the next 3 hours, every 30 minutes, thus set periods=6
periods = 2
train_X, train_Y, test_X, test_Y = prepare_data_for_training(df,
                                                             predict_date,
                                                             freq=freq,
                                                             start=start,
                                                             periods=periods,
                                                             observation='probability')
train_X.head()
# Observation with a shift of T+30 minutes in comparison to train_X.
# This is the 'future' availability used for traning.
train_Y.head()
train_X.iloc[3200:3210]
print(train_X.tail())
print(test_X.head())
Explanation: This is the final dataset. For further prediction, I could add some weather forecasts data to these features.
Prediction
Let's select a time window (start, stop) to a single prediction.
Two weeks of data for training
Predict the 'probability' for the next 30 minutes and 1 hour for every station.
Split the dataset into train and test dataset
Compute the RMSE (Root Mean Squared Error) between the observations (i.e. probability) and the predicted values
End of explanation
# the 'booster'
bst, train_prg = fit(train_X, train_Y, test_X, test_Y)
train_prg = pd.DataFrame({"train": train_prg["train"]['rmse'], "test": train_prg["test"]['rmse']})
colors = sns.color_palette('Set1', 2)
sns.palplot(colors)
with sns.axes_style("darkgrid", {'xtick.major.size': 8.0}):
    fig, ax = plt.subplots(figsize=(10,6))
for k, label, color in zip(train_prg.values.T, range(2), colors):
    print(k)
    plt.plot(100*k, color=color, label=label)
    
plt.legend(train_prg.columns)
plt.xlabel('XGBoost iteration')
plt.ylabel("Error (%)")
plt.xticks(np.linspace(0, 25, 6))
plt.yticks(np.linspace(0, 30, 6))
sns.despine()
plt.tight_layout()
plt.savefig("../images/lyon_prediction_training_curves.png")
# compute the prediction from test_*
pred = prediction(bst, test_X, test_Y)
pred[:5]
print("Number of predictions: {}".format(len(pred)))
# Compute the RMSE
rmse = np.sqrt(np.mean((pred - test_Y)**2))
rmse
# must install graphviz
# plot_tree(bst)
Explanation: The fit function create some data structure for the XGBoost from the train and test DataFrames (i.e. xgb.DMatrix), configure the model and launch it with the objective: 'reg:logistic'. It's a regression, not a classification.
End of explanation
result = test_X.copy()
result['ts_future'] = test_Y.index.shift(1, freq=freq)
result['observation'] = test_Y.copy()
result['ts_future'] = test_Y.index.shift(1, freq=freq)
result['prediction'] = pred
result['error'] = pred - test_Y
result['relative_error'] = 100. * np.abs(pred - test_Y) / test_Y
result['quad_error'] = (pred - test_Y)**2
result.to_csv("prediction-freq-{}-{}.csv".format(freq, predict_date))
result.head(10)
Explanation: Store the results
End of explanation
locations = pd.read_csv("../data/lyon-stations.csv")
locations.shape
Explanation: Some maps
CSV file with station coordinates
End of explanation
mask = locations['idstation'].isin(result.station.unique())
mask.sum()
locations = locations[mask]
locations = locations.rename_axis({'idstation': 'station'}, axis=1)
locations.head()
Explanation: Some stations were removed when the data were cleaned up. Remove them from the location data.
End of explanation
locations["nom"] = locations['nom'].str.replace("'", "&apos;")
Explanation: Some station names contains the ' character. Replace it by the HTML code for folium.
End of explanation
data_to_plot = result.loc[predict_date]
data_to_plot.shape
data_to_plot.head()
yhat = data_to_plot[['station', 'prediction']].merge(locations, on='station')
yhat.head()
y = data_to_plot[['station', 'observation']].merge(locations, on='station')
error = data_to_plot[['station', 'error']].merge(locations, on='station')
colormap = 'RdYlBu'
cmap = plt.get_cmap(colormap)
Explanation: Select the prediction data for a specific timestamp
End of explanation
# show the colormap use to plot the stations, values [0, 1]
gradient = np.linspace(0, 1, 256)
gradient = np.vstack((gradient, gradient))
fig, ax = plt.subplots(1)
fig.subplots_adjust(top=0.95, bottom=0.80, left=0.2, right=0.99)
ax.set_xticks([0., 64, 128, 192, 256])
ax.set_xticklabels([0., 0.25, 0.5, 0.75, 1.])
ax.set_xlabel('Bikes Availability')
ax.imshow(gradient, aspect='auto', cmap=cmap, vmin=0, vmax=1)
plt.title('Colormap used to plot stations')
Explanation: See 
https://matplotlib.org/examples/color/colormaps_reference.html
https://matplotlib.org/api/colors_api.html
End of explanation
color = lambda x: mpl.colors.to_hex(cmap(x))
# Lyon (France) Position
position = [45.750000, 4.850000]
mp_pred = folium.Map(location=position, zoom_start=13, tiles='cartodbpositron')
# Map of the predicted values
for _,row in yhat.iterrows():
    folium.CircleMarker(
        location=[row['lat'], row['lon']],
        radius=2,
        popup=row['nom'],
        color=color(row['prediction']),
        fill=True,
        fill_opacity=0.3,
        fill_color=color(row['prediction'])
    ).add_to(mp_pred)
mp_pred
# Map for the observation
mp_obs = folium.Map(location=position, zoom_start=13, tiles='cartodbpositron')
# Map of the observations
for _,row in y.iterrows():
    folium.CircleMarker(
        location=[row['lat'], row['lon']],
        radius=2,
        popup=row['nom'],
        color=color(row['observation']),
        fill=True,
        fill_opacity=0.3,
        fill_color=color(row['observation'])
    ).add_to(mp_obs)
mp_obs
# Colormap for error (by default, the color map fits for [0, 1] values)
norm = mpl.colors.Normalize(vmin=-1, vmax=1)
color_error = lambda x: mpl.colors.to_hex(cmap(norm(x)))
# Map for the errors
mp_error = folium.Map(location=position, zoom_start=13, tiles='cartodbpositron')
# Map of the errors
for _,row in error.iterrows():
    folium.CircleMarker(
        location=[row['lat'], row['lon']],
        radius=2,
        popup=row['nom'],
        color=color_error(row['error']),
        fill=True,
        fill_opacity=0.3,
        fill_color=color_error(row['error'])
    ).add_to(mp_error)
mp_error
Explanation: Red stations if there are 0 or a few available bikes
Blue stations if there are several available bikes
End of explanation
<END_TASK>