<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Machine Learning - Hands on Lab - Session #1
Lecturer
Step1:  2. Dataset Loading & Visualition
2.1 How does the expected "output" look like ?
Sample Submission File - sample_submission_NDF.csv
We load the csv file and visualise it straight away
Step2:  As we can see, it is a fairly simple file with only two columns
Step3:  Here are the information available on our test users
Step4:  We have a file containing a list of countries with the following features
Step5:  Displayed as followed, it is fairly hard to really see what is in this file.
Let's try to sort the file in regards with the first column ("age_bucket") and then by the third one ("gender")
Step6:  Now it appears that we have a file containing demographic information by country, by age and by gender.
For each "age-bucket", "country" and gender, we have the following information
Step7:  => We can conclude that we only have data about the year 2015 !
1.2.4 train_users_2.csv
We load the csv file and visualise it straight away
Step8:  Here are the information available on our test users
Step9:  1.3 Summary - Important Information
1.3.1 Train - Test Split Analysis
Step10:  1.3.2 - Summary
All the users in this dataset are from the USA (written on Kaggle)
12 Possibilities for the country of destination
Data are split by data 
Step11:  1.4.1 Feature Analysis
Step12:  1.4.2 Output Analysis - Is the dataset balanced ?
1.4.2.1 Dummy Analysis
Let's plot the the bookings distribution by destination in order to see whether the dataset is balanced or not.
Step13:  1.4.2.2 Same visualisation, with data grouped by Booking-year
Step14:  1.4.2.3 Conclusion
Over the years the repartition has changed. As we are doing predictions with user data from July 2014, this changes may require us to focus on the most recent data during training. It is most likely to have similar pattern than test data.
1.4.3 Account Creation Analysis
We want to evaluate the quantity of data we would have left if we only consider the N-last months of the data.
We have seen that the "early users" are very different of the actual ones. Let's see how much of data we can delete of the dataset.
Step15:  If we ony consider of 2013 and 2014, we still have more than 70% of the data for training.
This is important, because session data start in January 2014.
"In the sessions dataset, the data only dates back to 1/1/2014, while the users dataset dates back to 2010."
1.4.4 Reported Ages of User
<ASSISTANT_TASK:>
Python Code:
import os
from datetime import datetime
import numpy as np
import pandas as pd
import bokeh.charts as bk
import bokeh.plotting as bk_plt
import bokeh.models as bk_md
bk.output_notebook()
import urllib.request
 
# Download the file from `url` and save it locally under `file_name`:
zip_url = "https://github.com/dataventureutc/Kaggle-HandsOnLab/blob/master/data/data.zip?raw=true"
urllib.request.urlretrieve(zip_url, "data.zip")
 
import zipfile
 
zip_ref = zipfile.ZipFile("data.zip", 'r')
zip_ref.extractall("data/")
zip_ref.close()
Explanation: Machine Learning - Hands on Lab - Session #1
Lecturer: Jonathan DEKHTIAR
Date: 2017-03-13
<br/><br/>
Contact: contact@jonathandekhtiar.eu
Twitter: @born2data
LinkedIn: JonathanDEKHTIAR
Personal Website: JonathanDEKHTIAR
RSS Feed: FeedCrunch.io
Tech. Blog: born2data.com
Github: DEKHTIARJonathan
<br/><br/>
```
 2017 March 13
 In place of a legal notice, here is a blessing:
    May you do good and not evil.
    May you find forgiveness for yourself and forgive others.
    May you share freely, never taking more than you give.
**
```
1. Project Initialisation
1.1 Importing Librairies
End of explanation
df_sample_submission = pd.read_csv("data/sample_submission_NDF.csv")
df_sample_submission.head(n=5) # Only display a few lines and not the whole dataframe
Explanation: 2. Dataset Loading & Visualition
2.1 How does the expected "output" look like ?
Sample Submission File - sample_submission_NDF.csv
We load the csv file and visualise it straight away
End of explanation
df_test_users = pd.read_csv("data/test_users.csv")
df_test_users.head(n=5) # Only display a few lines and not the whole dataframe
Explanation: As we can see, it is a fairly simple file with only two columns:
* user identifier 
* country code of destination
=> This file represent how we must format our data for the delivery. 
There is no data in this file, it only is a template.
2.2 Loading Data Files
2.2.1 Test Users - test_users.csv
We load the csv file and visualise it straight away
End of explanation
df_countries = pd.read_csv("data/countries.csv")
df_countries.head(n=5) # Only display a few lines and not the whole dataframe
Explanation: Here are the information available on our test users:
id: user ID
date_account_created: date of account creation
timestamp_first_active: timestamp of the first activity, note that it can be earlier than date_account_created or date_first_booking because a user can search before signing up
date_first_booking: date of first booking
gender
age
signup_method: What method was used to signup (Facebook, basic, etc.)
signup_flow: the page a user came to signup up from
language: international language preference
affiliate_channel: what kind of paid marketing
affiliate_provider: where the marketing is e.g. google, craigslist, other
first_affiliate_tracked: whats the first marketing the user interacted with before the signing up
signup_app
first_device_type
first_browser
1.2.2 countries.csv
We load the csv file and visualise it straight away
End of explanation
df_age_gender_bkts = pd.read_csv("data/age_gender_bkts.csv")
df_age_gender_bkts.head(n=5) # Only display a few lines and not the whole dataframe
Explanation: We have a file containing a list of countries with the following features:
country code
latitude
longitude
distance in km
surface in km2
language
levenshtein distance with English Language
=> General information about countries of destination
1.2.3 age_gender_bkts.csv
We load the csv file and visualise it straight away
End of explanation
tmp_sorted = df_age_gender_bkts.sort_values(by=['age_bucket', 'gender'])
tmp_sorted.head(n=12) # Only display a few lines and not the whole dataframe
Explanation: Displayed as followed, it is fairly hard to really see what is in this file.
Let's try to sort the file in regards with the first column ("age_bucket") and then by the third one ("gender")
End of explanation
tmp_sorted.year.unique() # We query the distinct values over the last column "year"
Explanation: Now it appears that we have a file containing demographic information by country, by age and by gender.
For each "age-bucket", "country" and gender, we have the following information:
* Population in thousands
* year of the statistic
End of explanation
df_train_users = pd.read_csv("data/train_users_2.csv")
df_train_users.head(n=5) # Only display a few lines and not the whole dataframe
Explanation: => We can conclude that we only have data about the year 2015 !
1.2.4 train_users_2.csv
We load the csv file and visualise it straight away
End of explanation
df_sessions = pd.read_csv("data/sessions.csv")
print("There are " + str(df_sessions.shape[0])+ " rows in the dataset")
df_sessions.head(n=5) # Only display a few lines and not the whole dataframe
Explanation: Here are the information available on our test users:
id: user ID
date_account_created: date of account creation
timestamp_first_active: timestamp of the first activity, note that it can be earlier than date_account_created or date_first_booking because a user can search before signing up
date_first_booking: date of first booking
gender
age
signup_method: What method was used to signup (Facebook, basic, etc.)
signup_flow: the page a user came to signup up from
language: international language preference
affiliate_channel: what kind of paid marketing
affiliate_provider: where the marketing is e.g. google, craigslist, other
first_affiliate_tracked: whats the first marketing the user interacted with before the signing up
signup_app
first_device_type
first_browser
<span style="color:red">country_destination: this is the target variable that we want to predict </span>
Remark on country_destination =>  12 Possibilities: US, FR, CA, GB, ES, IT, PT, NL,DE, AU, NDF (no destination found), and other
1.2.5 session.csv
We load the csv file and visualise it straight away
End of explanation
training_min_date = df_train_users.date_account_created.min()
training_max_date = df_train_users.date_account_created.max()
print ("Training Date Range : [" + training_min_date +", " + training_max_date +"]")
# ===============
testing_min_date = df_test_users.date_account_created.min()
testing_max_date = df_test_users.date_account_created.max()
print ("Testing Date Range : [" + testing_min_date +", " + testing_max_date +"]")
Explanation: 1.3 Summary - Important Information
1.3.1 Train - Test Split Analysis
End of explanation
df_train_users.transpose().ix[:,:5] 
Explanation: 1.3.2 - Summary
All the users in this dataset are from the USA (written on Kaggle)
12 Possibilities for the country of destination
Data are split by data : 
Train data: [2010, June 2014] 
Test data: [July 2014, September 2014]
1.4 Train Data Analysis
Let us focus on training data. 
We transpose the df for a better reading and stop horinzontally scrolling.
We take the 6 first examples in order to have a sense of what's happening.
End of explanation
# How many rows in the DataFrame 
row_count = len(df_train_users.index)
print ("Row Count = " + str(row_count))
print("\n")
# ===============================
# Print what's the size of the ID
field_length = df_train_users.id.astype(str).map(len)
id_maxlength = len(df_train_users.loc[field_length.argmax(), 'id'])
id_minlength = len(df_train_users.loc[field_length.argmin(), 'id'])
if (id_maxlength != id_minlength):
    print ("ID Length = [" + str(id_minlength) + ", " + str(id_maxlength) + "]")
else:
    print ("ID Length = " + str(id_maxlength))
print("\n")
# ===============================
# Count NaN Values for date_first_booking
NaN_Count_date_first_booking = df_train_users.date_first_booking.isnull().sum()
print ("date_first_booking NaN Count = " + str(NaN_Count_date_first_booking) \
+ " && " +  str("%.2f" % (float(NaN_Count_date_first_booking)/row_count*100)) +"%")
print("\n")
# ===============================
# Possible Values for gender
gender_repartition = df_train_users.gender.value_counts()
for gender, count in gender_repartition.iteritems():
    print ("Gender: " + gender + " && Count: " + str(count) + " && " \
           + str("%.2f" % (float(count)/row_count*100)) +"%")
print("\n")
# ===============================
# Count NaN Values for age
NaN_Count_age = df_train_users.age.isnull().sum()
print ("age NaN Count = " + str(NaN_Count_age)  + " && "  \
       +  str("%.2f" % (float(NaN_Count_age)/row_count*100)) +"%")
print("\n")
# ===============================
# Possible Values for signup_method
signup_method_repartition = df_train_users.signup_method.value_counts()
for method, count in signup_method_repartition.iteritems():
    print ("Method: " + method + " && Count: " + str(count) + " && " \
           + str("%.2f" % (float(count)/row_count*100)) +"%")
    
print("\n")
# ===============================
# Possible Values for language
language_repartition = df_train_users.language.value_counts()
for language, count in language_repartition.iteritems():
    print ("language: " + language + " && Count: " + str(count)  \
           + " && " + str("%.2f" % (float(count)/row_count*100)) +"%")
    break
    
print("\n")
# ===============================
# Possible Values for affiliate_channel
affiliate_channel_repartition = df_train_users.affiliate_channel.value_counts()
i = 0
for channel, count in affiliate_channel_repartition.iteritems():
    print ("affiliate_channel: " + channel + " && Count: " + str(count) + " && " \
           + str("%.2f" % (float(count)/row_count*100)) +"%")
    i += 1
    if (i == 3):
        break
print("\n")
# ===============================
# Possible Values for first_affiliate_tracked
first_affiliate_tracked_repartition = df_train_users.first_affiliate_tracked.value_counts()
i = 0
for affiliate, count in first_affiliate_tracked_repartition.iteritems():
    print ("first_affiliate_tracked: " + affiliate + " && Count: " + str(count) + \
    " && " + str("%.2f" % (float(count)/row_count*100)) +"%")
    
    i += 1
    if (i == 3):
        break
print("\n")
# ===============================
# Possible Values for signup_app
signup_app_repartition = df_train_users.signup_app.value_counts()
for signup_app, count in signup_app_repartition.iteritems():
    print ("signup_app: " + signup_app + " && Count: " + str(count) + " && " \
           + str("%.2f" % (float(count)/row_count*100)) +"%")
print("\n")
# ===============================
# Possible Values for first_browser
first_browser_repartition = df_train_users.first_browser.value_counts()
i = 0
for first_browser, count in first_browser_repartition.iteritems():
    print ("first_browser: " + first_browser + " && Count: " + str(count) + \
    " && " + str("%.2f" % (float(count)/row_count*100)) +"%")
    
    i += 1
    if (i == 6):
        break
print("\n")
Explanation: 1.4.1 Feature Analysis
End of explanation
dict_output = dict()
country_destination_repartition = df_train_users.country_destination.value_counts()
for country_destination, count in country_destination_repartition.iteritems():
    dict_output[country_destination] = float(count)/row_count*100
df_output = pd.DataFrame(list(dict_output.items()),
                      columns=['Country', 'Repartition'])
df_output.sort_values(by=['Repartition'], ascending=False).head(n=5)
p = bk.Bar(df_output, label=['Country'], values='Repartition', ylabel='Booking Proportion in %')
bk.show(p)
Explanation: 1.4.2 Output Analysis - Is the dataset balanced ?
1.4.2.1 Dummy Analysis
Let's plot the the bookings distribution by destination in order to see whether the dataset is balanced or not.
End of explanation
df_output_by_year_and_month = df_train_users.loc[:,['country_destination','date_account_created']]
df_output_by_year_and_month.loc[:,'year_account_created'] = df_output_by_year_and_month['date_account_created'].apply(
    lambda x: datetime.strptime(x, "%Y-%m-%d").strftime("%Y")
)
df_output_by_year_and_month.loc[:,'month_account_created'] = df_output_by_year_and_month['date_account_created'].apply(
    lambda x: datetime.strptime(x, "%Y-%m-%d").strftime("%m")
)
df_output_by_year_and_month.loc[:,'year_month_account_created'] = df_output_by_year_and_month['date_account_created'].apply(
    lambda x: datetime.strptime(x, "%Y-%m-%d").strftime("%Y - %m")
)
df_output_by_year_and_month.head(n=3)
tmp_df = df_output_by_year_and_month.loc[:,['country_destination','year_account_created']]
rslt = tmp_df.groupby("year_account_created")['country_destination'].value_counts()
list_year = [x[0] for x in rslt.index.values]
list_dest = [x[1] for x in rslt.index.values]
list_count = rslt.values
df_tmp = pd.DataFrame(
    {'Year': list_year,
     'Destination': list_dest,
     'Count': list_count
    }
)
df_tmp.sort_values(by=["Destination", "Year"]).head(n=5)
df_pivot = pd.pivot_table(df_tmp, values='Count', index=['Destination'], columns=['Year'], 
                          aggfunc=np.sum, margins=True).astype(int)
total_by_years = df_pivot.loc[['All']]
for year, col in total_by_years.iteritems():
    subtotal_by_year = col[0]
    
    df_pivot[year] = df_pivot[year]/subtotal_by_year*100
    
df_pivot = df_pivot.sort_values(by="All", ascending=False).drop("All") # We sort by the column "All" and remove the line "All"
df_pivot
df_pivot2 = df_pivot.copy()
df_pivot2["Destination"] = df_pivot.index
df_pivot2 = pd.melt(df_pivot2, id_vars="Destination")
df_pivot2 = df_pivot2.drop(df_pivot2[df_pivot2.Year == "All"].index) # We remove the column 'All' from the plotting
df_pivot2.head(n=3)
p = bk.Bar(df_pivot2, label=['Destination'], values='value', group="Year", legend='top_right', ylabel='Booking Proportion in %')
bk.show(p)
Explanation: 1.4.2.2 Same visualisation, with data grouped by Booking-year
End of explanation
tmp_df = df_output_by_year_and_month[['country_destination','year_month_account_created']].sort_values(
    by="year_month_account_created"
)
tmp_df = tmp_df.groupby(["year_month_account_created"]).agg("count")
tmp_df.head(n=4)
# Plot Data
x_data = [str(x) for x in tmp_df["country_destination"].index.values]
y_data = tmp_df["country_destination"].values
p = bk_plt.figure(plot_width=980, plot_height=400, y_range=[0, y_data.max()+1000], x_range=x_data)
p.line(x_data, y_data, line_width=2)
p.circle(x_data, y_data, fill_color="white", size=8)
# Draw a line when X% of the users of the dataset have sign-up
split_1 = 20 # 80% left
split_2 = 30 # 70% left
split_3 = 50 # 50% left
split_4 = 70 # 30% left
thrs_value_1 = 213450 * split_1 / 100
thrs_value_2 = 213450 * split_2 / 100
thrs_value_3 = 213450 * split_3 / 100
thrs_value_4 = 213450 * split_4 / 100
index_1 = 0
index_2 = 0
index_3 = 0
index_4 = 0
tmp_sum = 0
for val in y_data:
    tmp_sum += val
    
    if tmp_sum <= thrs_value_1:
        index_1 += 1     
        index_2 += 1
        index_3 += 1
        index_4 += 1
    elif tmp_sum <= thrs_value_2:    
        index_2 += 1
        index_3 += 1
        index_4 += 1
    elif tmp_sum <= thrs_value_3:
        index_3 += 1
        index_4 += 1
    elif tmp_sum <= thrs_value_4:
        index_4 += 1
    else:
        break
           
        
vline_1 = bk_md.Span(location=index_1, dimension='height', line_color='blue', line_width=3) # 80% Data Left
vline_2 = bk_md.Span(location=index_2, dimension='height', line_color='green', line_width=3) # 70% Data Left
vline_3 = bk_md.Span(location=index_3, dimension='height', line_color='orange', line_width=3) # 50% Data Left
vline_4 = bk_md.Span(location=index_4, dimension='height', line_color='red', line_width=3) # 30% Data Left
p.renderers.extend([vline_1, vline_2, vline_3, vline_4])
# We set label orientation
from math import pi
p.xaxis.major_label_orientation = pi/2
# We display the graph
bk_plt.show(p)
Explanation: 1.4.2.3 Conclusion
Over the years the repartition has changed. As we are doing predictions with user data from July 2014, this changes may require us to focus on the most recent data during training. It is most likely to have similar pattern than test data.
1.4.3 Account Creation Analysis
We want to evaluate the quantity of data we would have left if we only consider the N-last months of the data.
We have seen that the "early users" are very different of the actual ones. Let's see how much of data we can delete of the dataset.
End of explanation
df_age = df_train_users[["age"]]
na_count = df_age.isnull().sum()[0]
row_count = df_age.shape[0]
na_percent = "%.2f" % (float(na_count) / row_count * 100)
print ("Count of NaN Values = " + str(na_count) + " => " + str(na_percent) + "%")
print ("Total Row Count = " + str(row_count))
df_age = df_age.dropna().reset_index(drop=True)
df_age.head(n=3)
df_age_bucket = df_age.groupby(pd.cut(df_age['age'], np.arange(0,999999,5))).count()
idx_tmp = df_age_bucket.index.values
bckt_count_tmp = df_age_bucket.age.values
data = dict()
for bck, cnt in zip(idx_tmp, bckt_count_tmp):
    start_point = int(bck[1:].partition(",")[0])
    end_point = int(bck[:-1].partition(",")[2])
    
    range_name = "("+ "%03d" % (start_point) + "," + "%03d" % (end_point) + "]"
    if start_point < 100:
        data[range_name] = cnt
    elif start_point >= 100 and start_point < 1000:
        if not "(100, 999]" in data:
            data["(100, 999]"] = cnt
        else:
            data["(100, 999]"] += cnt
    else:
        if not "+1000" in data:
            data["+1000"] = cnt
        else:
            data["+1000"] += cnt
            
df_age_stats = pd.DataFrame({
    'Bucket': list(data.keys()),
    'Count': list(data.values())
}).sort_values(by="Bucket").reset_index(drop=True)
df_age_stats.head(n=5)
x_range = df_age_stats["Bucket"].values
bar2 = bk.Bar(
    df_age_stats, 
    values = 'Count', 
    label  = ['Bucket'],
    title  = "Reported Ages of Users", 
    width  = 980,
    ylabel = "Number of Users by Bucket"
)
bk.show(bar2)
Explanation: If we ony consider of 2013 and 2014, we still have more than 70% of the data for training.
This is important, because session data start in January 2014.
"In the sessions dataset, the data only dates back to 1/1/2014, while the users dataset dates back to 2010."
1.4.4 Reported Ages of User
End of explanation
<END_TASK>