<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data Incubator Fellowship Semifinalist Challenge
<a href="mailto
Step1:  Import libraries
Step7:  User-defined functions
Step8:  Load dataset
https
Step9:  Pre-processing data
Drop less useful columns
Step10:  Fix missing value
Step11:  Standardize country code
Step12:  Extract serving_size into gram value
Step13:  Parse additives
Step14:  Organic or Not
[TODO]
pick up word 'Organic' from product_name column
pick up word 'Organic','org' from ingredients_text column
Add creation_date
Step15:  Visualize food features
Food labels yearly trend
Step16:  Top countries
Step17:  Nutrition grade
Step18:  Nutrition score
Step19:  Serving size
Step20:  Energy, fat, ...
Energy
Fat
Saturated-Fat
Trans-Fat
Step21:  Carbohydrates, protein, fiber
Carbohydrates
Cholesterol
Proteins
Fiber
Step22:  Sugar, Vitamins
Sugars
Salt
Vitamin-A
Vitamin-C
Step23:  Minerals
Calcium
Iron
Sodium
Step24:  Explore food nutrition
Are Amercan and French food different?
Step25:  Which food has more sugar?
<ASSISTANT_TASK:>
Python Code:
from jyquickhelper import add_notebook_menu
add_notebook_menu()
Explanation: Data Incubator Fellowship Semifinalist Challenge
<a href="mailto:wen.gong@gmail.com"><font size=+2>Wen Gong</font></a>
Motivation
<br>
<font color=red size=+2>Know what we eat, </font>
<font color=green size=+2> Gain insight into food, </font>
<font color=blue size=+2> Choose healthy diet.</font>
<a href=https://world.openfoodfacts.org/>
<img src=https://static.openfoodfacts.org/images/misc/openfoodfacts-logo-en-178x150.png width=300 height=200>
</a>
<font color=black size=+1>Task: Explore feasibility of building model to predict nutrition grade 
<font color=blue size=+1>Goal: help people make healthy decision in their daily diet</font>
</font> 
<font color=black size=+1>Focus on 2 questions:</font>
Are Amercan and French food different?
Which food has more sugar?
End of explanation
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb
%matplotlib inline
Explanation: Import libraries
End of explanation
def remove_na_rows(df, cols=None):
    
    remove row with NaN in any column
    
    if cols is None:
        cols = df.columns
    return df[np.logical_not(np.any(df[cols].isnull().values, axis=1))]
def trans_country_name(x):
    
    translate country name to code (2-char)
    
    try:
        country_name = x.split(',')[0]
        if country_name in dictCountryName2Code:
            return dictCountryName2Code[country_name]
    except:
        return None
def parse_additives(x):
    
    parse additives column values into a list
    
    try:
        dict = {}
        for item in x.split(']'): 
            token = item.split('->')[0].replace("[", "").strip() 
            if token: dict[token] = 1
        return [len(dict.keys()),  sorted(dict.keys())]
    except:
        return None
def trans_serving_size(x):
    
    pick up gram value from serving_size column
    
    try:
        serving_g = float((x.split('(')[0]).replace("g", "").strip())
        return serving_g
    except:
        return 0.0
def distplot2x2(cols):
    
    make dist. plot on 2x2 grid for up to 4 features   
    
    sb.set(style="white", palette="muted")
    f, axes = plt.subplots(2, 2, figsize=(7, 7), sharex=False)
    b, g, r, p = sb.color_palette("muted", 4)
    colors = [b, g, r, p]
    axis = [axes[0,0],axes[0,1],axes[1,0],axes[1,1]]
    for n,col in enumerate(cols):
        sb.distplot(food[col].dropna(), hist=True, rug=False, color=colors[n], ax=axis[n])
Explanation: User-defined functions
End of explanation
food = pd.read_excel("../../0.0-Datasets/openfoodfacts_100k.xlsx")
food.shape
# (99999, 162)
food.columns
food.head()
Explanation: Load dataset
https://www.kaggle.com/openfoodfacts/world-food-facts
This dataset contains Food Nutrition Fact for 100 000+ food products from 150 countries.
<img src=https://static.openfoodfacts.org/images/products/00419796/front_en.3.full.jpg>
End of explanation
# columns_to_keep = ['code','product_name','created_datetime','brands','categories','origins','manufacturing_places','energy_100g','fat_100g','saturated-fat_100g','trans-fat_100g','cholesterol_100g','carbohydrates_100g','sugars_100g','omega-3-fat_100g','omega-6-fat_100g','fiber_100g','proteins_100g','salt_100g','sodium_100g','alcohol_100g','vitamin-a_100g','vitamin-c_100g','potassium_100g','chloride_100g','calcium_100g','phosphorus_100g','iron_100g','magnesium_100g','zinc_100g','copper_100g','manganese_100g','fluoride_100g','ingredients_text','countries','countries_en','serving_size','additives','nutrition_grade_fr','nutrition_grade_uk','nutrition-score-fr_100g','nutrition-score-uk_100g','url','image_url','image_small_url']
columns_to_keep = ['code','product_name','created_datetime','brands','energy_100g','fat_100g','saturated-fat_100g','trans-fat_100g','cholesterol_100g','carbohydrates_100g','sugars_100g','fiber_100g','proteins_100g','salt_100g','sodium_100g','vitamin-a_100g','vitamin-c_100g','calcium_100g','iron_100g','ingredients_text','countries','countries_en','serving_size','additives','nutrition_grade_fr','nutrition-score-fr_100g','url']
food = food[columns_to_keep]
Explanation: Pre-processing data
Drop less useful columns
End of explanation
columns_numeric_all = ['energy_100g','fat_100g','saturated-fat_100g','trans-fat_100g','cholesterol_100g','carbohydrates_100g','sugars_100g','omega-3-fat_100g','omega-6-fat_100g','fiber_100g','proteins_100g','salt_100g','sodium_100g','alcohol_100g','vitamin-a_100g','vitamin-c_100g','potassium_100g','chloride_100g','calcium_100g','phosphorus_100g','iron_100g','magnesium_100g','zinc_100g','copper_100g','manganese_100g','fluoride_100g','nutrition-score-fr_100g','nutrition-score-uk_100g']
columns_numeric = set(columns_numeric_all) & set(columns_to_keep)
columns_categoric = set(columns_to_keep) - set(columns_numeric)
# turn off
if False:
    for col in columns_numeric:
        if not col in ['nutrition-score-fr_100g', 'nutrition-score-uk_100g']:
            food[col] = food[col].fillna(0)
    for col in columns_categoric:
        if col in ['nutrition_grade_fr', 'nutrition_grade_uk']:
            food[col] = food[col].fillna('-')
        else:        
            food[col] = food[col].fillna('')
# list column names: categoric vs numeric
columns_categoric, columns_numeric
food.head(3)
Explanation: Fix missing value
End of explanation
# standardize country
country_lov = pd.read_excel("../../0.0-Datasets/country_cd.xlsx")
# country_lov.shape
# country_lov.head()
# country_lov[country_lov['GEOGRAPHY_NAME'].str.startswith('United')].head()
# country_lov['GEOGRAPHY_CODE'].tolist()
# country_lov.ix[0,'GEOGRAPHY_CODE'], country_lov.ix[0,'GEOGRAPHY_NAME']
# create 2 dictionaries
dictCountryCode2Name = {}
dictCountryName2Code = {}
for i in country_lov.index:
    dictCountryCode2Name[country_lov.ix[i,'GEOGRAPHY_CODE']] = country_lov.ix[i,'GEOGRAPHY_NAME']
    dictCountryName2Code[country_lov.ix[i,'GEOGRAPHY_NAME']] = country_lov.ix[i,'GEOGRAPHY_CODE']    
# add Country_Code column - pick 1st country from list
food['countries_en'] = food['countries_en'].fillna('')
food['country_code'] = food['countries_en'].apply(str).apply(lambda x: trans_country_name(x))
# add country_code to columns_categoric set
columns_categoric.add('country_code')
# verify bad country
food[food['country_code'] != food['countries']][['country_code', 'countries']].head(20)
food['ingredients_text'].head()  # leave as is
Explanation: Standardize country code
End of explanation
# add serving_size in gram column
food['serving_size'].head(10)
food['serving_size'] = food['serving_size'].fillna('')
food['serving_size_gram'] = food['serving_size'].apply(lambda x: trans_serving_size(x))
# add serving_size_gram 
columns_numeric.add('serving_size_gram')
food[['serving_size_gram', 'serving_size']].head()
Explanation: Extract serving_size into gram value
End of explanation
food['additives'].head(10)
food['additives'] = food['additives'].fillna('')
food['additive_list'] = food['additives'].apply(lambda x: parse_additives(x))
# add additive_list 
columns_categoric.add('additive_list')
food[['additive_list', 'additives']].head()
Explanation: Parse additives
End of explanation
food["creation_date"] = food["created_datetime"].apply(str).apply(lambda x: x[:x.find("T")])
def extract_year(x):
    try:
        return int(x[:x.find("-")])
    except:
        return None
    
food["year_added"] = food["created_datetime"].dropna().apply(str).apply(extract_year)
# add creation_date 
columns_categoric.add('creation_date')
columns_numeric.add('year_added')
food[['created_datetime', 'creation_date', 'year_added']].head()
# food['product_name']
food.head(3)
columns_numeric
Explanation: Organic or Not
[TODO]
pick up word 'Organic' from product_name column
pick up word 'Organic','org' from ingredients_text column
Add creation_date
End of explanation
year_added = food['year_added'].value_counts().sort_index()
#year_added
year_i = [int(x) for x in year_added.index]
x_pos = np.arange(len(year_i))
year_added.plot.bar()
plt.xticks(x_pos, year_i)
plt.title("Food labels added per year")
Explanation: Visualize food features
Food labels yearly trend
End of explanation
TOP_N = 10
dist_country = food['country_code'].value_counts()
top_country = dist_country[:TOP_N][::-1]
country_s = [dictCountryCode2Name[x] for x in top_country.index]
y_pos = np.arange(len(country_s))
top_country.plot.barh()
plt.yticks(y_pos, country_s)
plt.title("Top {} Country Distribution".format(TOP_N))
Explanation: Top countries
End of explanation
# dist_nutri_grade = food['nutrition_grade_uk'].value_counts()
# no value
dist_nutri_grade = food[food["nutrition_grade_fr"].isin(['a','b','c','d','e'])]
dist_nutri_grade = dist_nutri_grade['nutrition_grade_fr'].value_counts()
dist_nutri_grade.sort_index(ascending=False).plot.barh()
plt.title("Nutrition Grade Dist")
Explanation: Nutrition grade
End of explanation
food['nutrition-score-fr_100g'].dropna().plot.hist()
plt.title("{} Dist.".format("Nutri-Score"))
Explanation: Nutrition score
End of explanation
food['serving_size_gram'].dropna().plot.hist()
plt.title("{} Dist.".format("Serving Size (g)"))
Explanation: Serving size
End of explanation
distplot2x2([ 'energy_100g','fat_100g','saturated-fat_100g','trans-fat_100g'])
Explanation: Energy, fat, ...
Energy
Fat
Saturated-Fat
Trans-Fat
End of explanation
distplot2x2(['carbohydrates_100g', 'cholesterol_100g', 'proteins_100g', 'fiber_100g'])
Explanation: Carbohydrates, protein, fiber
Carbohydrates
Cholesterol
Proteins
Fiber
End of explanation
distplot2x2([ 'sugars_100g', 'salt_100g', 'vitamin-a_100g', 'vitamin-c_100g'])
Explanation: Sugar, Vitamins
Sugars
Salt
Vitamin-A
Vitamin-C
End of explanation
distplot2x2(['calcium_100g', 'iron_100g',  'sodium_100g'])
Explanation: Minerals
Calcium
Iron
Sodium
End of explanation
df = food[food["country_code"].isin(['US','FR'])][['energy_100g', 'sugars_100g','country_code']]
df = remove_na_rows(df)
df.head(3)
sb.pairplot(df, hue="country_code", size=2.5)
Explanation: Explore food nutrition
Are Amercan and French food different?
End of explanation
# prepare a small dataframe for ['US', 'FR']
df2 = food[food["country_code"].isin(['US','FR'])][['energy_100g', 'sugars_100g','country_code','nutrition_grade_fr']]
df2 = df2[df2["nutrition_grade_fr"].isin(['a','d'])]
df2 = df2.sort_values(by="nutrition_grade_fr")
# df2.head()
# create a grid of scatter plot
g = sb.FacetGrid(df2, row="nutrition_grade_fr", col="country_code", margin_titles=True)
g.map(plt.scatter, "sugars_100g", "energy_100g", color="steelblue")
g.set(xlim=(0, 100), ylim=(0, 8000))
Explanation: Which food has more sugar?
End of explanation
<END_TASK>