<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Comparing Switzerland to the world on Amazon
We found these statistics about the dataset on amazon
Step1:  1 Explore products dataset
We start by looking at the product data without the reviews.
Load the product data in a dataframe
Step2:  Our data contains false positives. We found no way of filtering them out besides a quick manual check.
Step3:  We decided to remove the obvious ones
Step4:  Let's take a took at the data. We start with detecting NA values
Step5:  We fill na values as follows
Step6:  The average price of the swiss products is $79.73.
Let's look at the most expansive product on amazon
Step7:  Let's look at some statistics about the products
Step8:  Which brand occurs the most in our dataset?
Step9:  Which category occurs the most?
Step10:  2. Explore review dataset
transform data
Step11:  Transform categories into list
Step12:  Give number to each review according to n-th review of a user
Step13:  Give number to each review according to n-th review of product
Step14:  Transform date into readable date
Step15:  3. Analysis
How many times each rating is given?
Step16:  Do ratings evolve over time?
Step17:  4. Comparison to the world
We will now start with making plots to compare the swiss products to the analysis of Max Woolf, as mentioned earlier.
Step18:  This graph shows the accumalative distribution of reviews a user gives on average on swiss products.
Step19:  This graph shows the accumalative distribution of reviews a swiss product receives on average.
Step20:  This graph shows the average rating by product category.
Step21:  Distribution of ratings
Step22:  This graph illustrates the average score a user gives (provided that he at least reviewd 5 products in our dataset).
Step23:  This graph illustrates the average score a product has (provided that it has at least 5 reviews).
Step24:  This graph shows how the ratings a person gives evolve over time on average.
Step25:  This graph shows how the ratings a product receives evolve over time on average.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import datetime
from ggplot import *
plt.style.use('seaborn-whitegrid')
plt.style.use('seaborn-notebook')
#['grayscale', 'fivethirtyeight', 'seaborn-deep', 'bmh', 'seaborn-poster', 'seaborn-ticks', 'seaborn-dark', 'seaborn-darkgrid', 'seaborn-whitegrid', 'seaborn-white', 'seaborn-bright', 'seaborn-paper', 'seaborn-pastel', 'seaborn-colorblind', 'seaborn-notebook', 'seaborn-dark-palette', 'dark_background', 'ggplot', 'seaborn-muted', 'seaborn-talk', 'classic']
Explanation: Comparing Switzerland to the world on Amazon
We found these statistics about the dataset on amazon: http://minimaxir.com/2017/01/amazon-spark/, credits: Max Woolf. Based on his analysis we made a comparison computing the same analysis but with only swiss reviews.
End of explanation
def parse(path):
  f = open(path, 'r')
  for l in f:
    yield eval(l)
  f.close()
def getDF(path):
  i = 0
  df = {}
  for d in parse(path):
    df[i] = d
    i += 1
  return pd.DataFrame.from_dict(df, orient='index')
df_products = getDF('data/swiss_products.json')
df_products = df_products.sort_values(['brand', 'price'])
df_products[df_products['brand'] == 'victorinox']
Explanation: 1 Explore products dataset
We start by looking at the product data without the reviews.
Load the product data in a dataframe
End of explanation
df_products.brand.value_counts()
Explanation: Our data contains false positives. We found no way of filtering them out besides a quick manual check.
End of explanation
false_positives = ['samsonite', 'mazda', 'lacoste', 'pelikan', 'bell']
df_products = df_products[~df_products.brand.isin(false_positives)]
Explanation: We decided to remove the obvious ones
End of explanation
for column_name in df_products.columns:
    print("NA values for " + column_name + ": " + str(df_products[column_name].isnull().sum()))
Explanation: Let's take a took at the data. We start with detecting NA values
End of explanation
df_products['salesrank'].fillna(value='{}', inplace=True)
df_products['related'].fillna(value='{}', inplace=True)
df_products['description'].fillna(value='', inplace=True)
av_price = df_products['price'].mean()
df_products['price'].fillna(value=av_price, inplace=True)
av_price
Explanation: We fill na values as follows: empty for salesrank, related and description, and average price for price
End of explanation
df_products[df_products.price == max(df_products['price'])].iloc[0].asin
from IPython.display import Image
Image(filename='data/expensivewatch.jpg') 
Explanation: The average price of the swiss products is $79.73.
Let's look at the most expansive product on amazon
End of explanation
df_products.price.describe()
Explanation: Let's look at some statistics about the products
End of explanation
df_products.brand.value_counts()[:20].plot(kind='bar', figsize=(10,5), title="Number of occurences of brands")
Explanation: Which brand occurs the most in our dataset?
End of explanation
Which category occurs the most
flatten = lambda l: [item for sublist in l for item in sublist]
categories_list = []
for categories in df_products.categories:
    for categorie in flatten(categories):
        categories_list.append(categorie)
pd.Series(categories_list).value_counts()[:20].plot(kind='bar', figsize=(10,5), title="Number of occurences of categories")
Explanation: Which category occurs the most?
End of explanation
df = pd.read_csv("data/amazon_ratings.csv")
df.head(10)
Explanation: 2. Explore review dataset
transform data
End of explanation
def get_category(x): 
    while 'u\'' in x:
        x = x.replace('u\'','\'')
    for y in eval(x):
        for first in y:
            return first
df['category'] = df['category'].apply(get_category)
df.head(10)
Explanation: Transform categories into list
End of explanation
df = df.sort_values(['user_id', 'timestamp'])
df[10:20]
df['nth_user'] = 1
user_id = ''
counter = 1
for i in range(0,df.shape[0]):
    if df.iloc[i].user_id != user_id:
        counter = 1
        user_id = df.iloc[i].user_id
    else:
        counter += 1
        df.set_value(i, 'nth_user', counter)
df[10:20]        
Explanation: Give number to each review according to n-th review of a user
End of explanation
df = df.sort_values(['item_id', 'timestamp']).reset_index(drop=True)
df.head()
df['nth_product'] = 1
item_id = ''
counter = 1
for i in range(0,df.shape[0]):
    if df.iloc[i].item_id != item_id:
        counter = 1
        item_id = df.iloc[i].item_id
    else:
        counter += 1
        df.set_value(i, 'nth_product', counter)
df.head()   
Explanation: Give number to each review according to n-th review of product
End of explanation
def get_time(x): 
    return datetime.datetime.fromtimestamp(
        int(x)
    ).strftime('%Y-%m-%d %H:%M:%S')
df['time'] = df['timestamp'].apply(get_time)
df.head(10)
Explanation: Transform date into readable date
End of explanation
df_ratings = df.groupby(df.rating).count().ix[:,0]
df_ratings
df_ratings.plot.bar()
Explanation: 3. Analysis
How many times each rating is given?
End of explanation
def percentage_rating(year): 
    df_year = df[df.time.str.contains(year)]
    counts_year = df_year.groupby(['rating']).count().ix[:,0]
    print(df_year.shape)
    counts_year = counts_year/sum(counts_year)
    ret = pd.DataFrame(counts_year)
    ret.columns = [year]
    return ret.transpose()
percentage_rating('1999')
ax = pd.concat([percentage_rating('1999'), percentage_rating('2014')]).transpose().plot.bar()
ax.set_title("Percentage of ratings given per year", fontsize=18)
ax.set_xlabel("Rating given", fontsize=16)
ax.set_ylabel("Percentage of total reviews", fontsize=16)
ax.legend(loc=2,prop={'size':14})
Explanation: Do ratings evolve over time?
End of explanation
df_user_review_counts = df.groupby(df.user_id)
counts = df_user_review_counts.count().ix[:,0].value_counts().sort_index()
counts
num_reviews = []
prop = []
s = 0
for i in range(0,counts.shape[0]):
    num_reviews.append(counts.index[i])
    s += counts.iloc[i]/sum(counts)
    prop.append(s)
df_counts = pd.DataFrame(
    {'num_reviews': num_reviews,
     'prop': prop
    })
df_counts.head()
Explanation: 4. Comparison to the world
We will now start with making plots to compare the swiss products to the analysis of Max Woolf, as mentioned earlier.
End of explanation
ax = df_counts.plot(x='num_reviews', y='prop', figsize=(10,7),
                    color="#2980b9", legend=None)
ax.set_title("Cumulative Proportion of # Amazon Reviews Given by User", fontsize=18)
ax.set_xlabel("# Reviews Given By User", fontsize=14)
ax.set_ylabel("Cumulative Proportion of All Amazon Reviewers", fontsize=14)
ax.set_ylim((0,1.1))
ax.yaxis.set_ticks(np.arange(0, 1.1, 0.25))
ax.set_xlim((0,50))
df_user_review_counts = df.groupby(df.item_id)
counts = df_user_review_counts.count().ix[:,0].value_counts().sort_index()
counts
num_reviews = []
prop = []
s = 0
for i in range(0,counts.shape[0]):
    num_reviews.append(counts.index[i])
    s += counts.iloc[i]/sum(counts)
    prop.append(s)
df_counts = pd.DataFrame(
    {'num_reviews': num_reviews,
     'prop': prop
    })
df_counts.head()
Explanation: This graph shows the accumalative distribution of reviews a user gives on average on swiss products.
End of explanation
ax = df_counts.plot(x='num_reviews', y='prop', figsize=(10,7),
                    color="#27ae60", legend=None)
ax.set_title("Cumulative Proportion of # Reviews Given For Product", fontsize=18)
ax.set_xlabel("# Reviews Given For Product", fontsize=14)
ax.set_ylabel("Cumulative Proportion of All Amazon Products", fontsize=14)
ax.set_ylim((0,1.1))
ax.yaxis.set_ticks(np.arange(0, 1.1, 0.25))
ax.set_xlim((0,50))
df_user_review_counts = df.groupby(df.category)
counts = df_user_review_counts.count().ix[:,0]
categories = pd.DataFrame({'avg_rating': df_user_review_counts.agg({'rating': 'mean'})['rating'], 
                            'count': counts})
categories
Explanation: This graph shows the accumalative distribution of reviews a swiss product receives on average.
End of explanation
# [((categories.index == "baby products") | (categories.index == "office & school supplies"))]
cats = categories['avg_rating'].sort_values(ascending=True)
ax = cats.plot.barh(width=1.0, color = "#e67e22", alpha=0.9, figsize=(6, 16))
ax.set_title("Average Rating Score Given For Amazon Reviews, by Product Category", fontsize = 16)
ax.set_xlabel("Avg. Rating For Reviews Given in Category", fontsize = 14)
ax.set_ylabel("Category", fontsize = 14)
for i in range(0,cats.shape[0]):
    height = cats[i]
    ax.text(height-1/3, i-1/4,
            '%.2f' % height,
            ha='center', va='bottom', color = 'white', fontweight='bold', size=16)
    
Explanation: This graph shows the average rating by product category.
End of explanation
counts = df.groupby(df.user_id).count().ix[:,0]
avg_rating = df.groupby(df.user_id).agg({'rating': 'mean'})['rating']
users = pd.DataFrame({'avg_rating': avg_rating, 
                            'count_reviews': counts})
users = users[users.count_reviews > 4]
users['avg_rating'] = round(users['avg_rating'],1)
users.head()
counts = users['avg_rating'].value_counts()
counts = pd.DataFrame(counts).sort_index()
counts2 = counts.reset_index()
counts2.head()
av = sum(counts2['index']*counts2['avg_rating'])/ sum(counts2['avg_rating'])
av
Explanation: Distribution of ratings
End of explanation
ax = counts.plot.bar(figsize=(8,5), color ='#2980b9', legend=None)
ax.axvline(counts2[counts2['index']==round(av,1)].index, color='k', linestyle='--')
ax.set_title("Distribution of Average Ratings by User, for Amazon Products")
ax.set_xlabel("Average Rating for Amazon Products Given By User (5 Ratings Minimum)")
ax.set_ylabel("Count of Users")
ax.set_xticks([0,7,16,26,36])
ax.set_xticklabels(['1','2','3','4','5'])
counts = df.groupby(df.item_id).count().ix[:,0]
avg_rating = df.groupby(df.item_id).agg({'rating': 'mean'})['rating']
products = pd.DataFrame({'avg_rating': avg_rating, 
                            'count_reviews': counts})
products = products[products.count_reviews > 4]
products['avg_rating'] = round(products['avg_rating'],1)
products.head()
counts = products['avg_rating'].value_counts()
counts = pd.DataFrame(counts).sort_index()
counts2 = counts.reset_index()
counts2.head()
av = sum(counts2['index']*counts2['avg_rating'])/ sum(counts2['avg_rating'])
av
Explanation: This graph illustrates the average score a user gives (provided that he at least reviewd 5 products in our dataset).
End of explanation
ax = counts.plot.bar(figsize=(10,6), color ='#27ae60', legend=None, width = 1)
ax.axvline(counts2[counts2['index']==round(av,1)].index, color='k', linestyle='--')
ax.set_title("Distribution of Overall Ratings on Amazon Products", fontsize = 18)
ax.set_xlabel("Overall Amazon Product Ratings (5 Ratings Minimum)", fontsize = 14)
ax.set_ylabel("Count of Products", fontsize = 14)
ax.set_xticks([0,9,19,29,39])
ax.set_xticklabels(['1.1','2','3','4','5'])
df_breakdown_users =df[df.nth_user <= 50].groupby(['nth_user', 'rating']).count()
df_breakdown_users = df_breakdown_users.reset_index(level=1)
df_breakdown_users.head()
s = 0
df2 = pd.DataFrame({'rating_5':[], 'rating_4':[], 'rating_3':[], 'rating_2':[], 'rating_1':[]})
for i in range(1,51):
    s = sum(df_breakdown_users[df_breakdown_users.index == i]['user_id'])
    df2 = df2.append(pd.DataFrame({'rating_5':[0],
                                       'rating_4':[0],
                                       'rating_3':[0],
                                       'rating_2':[0],
                                       'rating_1':[0]},
                    index = [i]))
    for j in range(1,6):
        result = df_breakdown_users[(df_breakdown_users.index == i) & (df_breakdown_users.rating == j*1.0)]
        if not result.empty:
            df2.set_value(i,'rating_'+str(j), result.user_id/s)
df2 = df2[["rating_5","rating_4","rating_3","rating_2","rating_1"]]
df2.head()
Explanation: This graph illustrates the average score a product has (provided that it has at least 5 reviews).
End of explanation
ax = df2.plot.bar(stacked=True, color = ['#003a09', '#049104', '#d36743', '#dd310f', '#870202'], width = 0.9)
ax.set_xticks([10,20,30,40,50])
ax.set_xticklabels(['10','20','30','40','50'])
ax.set_title("Breakdown of Amazon Ratings Given by Users, by n-th Rating Given\n\n")
ax.set_xlabel("n-th Rating Given by User")
ax.set_ylabel("Proportion")
ax.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
           ncol=5, mode="expand", borderaxespad=0.)
ax.set_ylim((0,1))
df_breakdown_users =df[df.nth_product <= 50].groupby(['nth_product', 'rating']).count()
df_breakdown_users = df_breakdown_users.reset_index(level=1)
df_breakdown_users.head()
s = 0
df2 = pd.DataFrame({'rating_5':[], 'rating_4':[], 'rating_3':[], 'rating_2':[], 'rating_1':[]})
for i in range(1,51):
    s = sum(df_breakdown_users[df_breakdown_users.index == i]['item_id'])
    df2 = df2.append(pd.DataFrame({'rating_5':[0],
                                       'rating_4':[0],
                                       'rating_3':[0],
                                       'rating_2':[0],
                                       'rating_1':[0]},
                    index = [i]))
    for j in range(1,6):
        result = df_breakdown_users[(df_breakdown_users.index == i) & (df_breakdown_users.rating == j*1.0)]
        if not result.empty:
            df2.set_value(i,'rating_'+str(j), result.user_id/s)
df2 = df2[["rating_5","rating_4","rating_3","rating_2","rating_1"]]
df2.head()
Explanation: This graph shows how the ratings a person gives evolve over time on average.
End of explanation
ax = df2.plot.bar(stacked=True, color = ['#003a09', '#049104', '#d36743', '#dd310f', '#870202'], width = 0.9)
ax.set_xticks([10,20,30,40,50])
ax.set_xticklabels(['10','20','30','40','50'])
ax.set_title("Breakdown of Ratings for Amazon Products by n-th Item Rating Given\n\n")
ax.set_xlabel("n-th Review for Item Given")
ax.set_ylabel("Proportion")
ax.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
           ncol=5, mode="expand", borderaxespad=0.)
ax.set_ylim((0,1))
Explanation: This graph shows how the ratings a product receives evolve over time on average.
End of explanation
<END_TASK>