<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Working with data 2017. Class 2
Contact
Javier Garcia-Bernardo
garcia@uva.nl
0. Structure
Data types, structures and code II
Tidy data
Step1:  Save CSV to disk (df.to_csv())
path_or_buf = filename
sep = separator
na_rep (default "") = how do you want to represent missing values
header = do you want to save the column names?
index (default True) = is there an index to save? usually False
Step2:  2. Tidy data
2.1 Observations and variables
A variable is any characteristics, number, or quantity that can be measured or counted
Step3:  2.2 What is tidy data?
Based on
Step4:  2.3 Tidying messy datasets
Real datasets can, and often do, violate the three precepts of tidy data in almost every way
imaginable. While occasionally you do get a dataset that you can start analysing immediately,
this is the exception,  not the rule
Step5:  Data to study
Step6:  2.5 Multiple variables are stored in one column
Step7:  2.6 Variables are stored in both rows and columns
Step8:  3. Merging and concatenating datasets
Concatenate
Step9:  Note that it's not a problem if the variables are not all the same
But in that case you'll have missing values
Step10:  3.2 Merge datasets
Step11:  4. Change columns, replace and apply functions
Some times we want to replace some categories with numbers
This is especially important with text (and we'll talk more in the class about text)
Step12:  4.1 Multiply columns
Step13:  Is there a relationship between green area per capita and population density? Why?
Imagine we want to calculate the fraction of green area
$$\frac{10^{6} m^2_{green}}{people} \cdot \frac{people}{km^2_{city}} = \frac{10^{6} m^2_{green}}{km^2_{city}}$$
Step14:  4.2 Categories to numbers
Step15:  4.2 Keeping part of a the values in a column (text)
We'll talk more about this when we talk about text analysis
Step16:  4.3 Doing math with the column
Step17:  5. Dealing with missing data
Let's imagine we have a data of a survey, with age, income, education, political ideas, location, and if you will vote for Trump or Hillary.
We may have some missing values. This missing values can be
Step18:  4.1 Ignoring data
Step19:  4.3 Imputing with mean/median/mode
Step20:  4.4 Imputing using neighbous
We may go over this another day, it's not an easy topic.
But the basic idea is that you are probably similar to your neighbors (variables are correlated)
Step21:  But first of all we need to make sure we have missing values instead of "-9" or other strings
The way to do it is to explore the file and then read it
Step22:  6.0 SCATTER using sns.lmplot
Use to plot the relationship between two quantitative variables (and one qualitative variable using color)
- x = x axis
- y = y axis
- h = separate using color qualitative variable
- data = dataframe
- fit_reg = True to fit regression lines
- col = separate by another qualitative variable
- row = separate by anoyther qualitative variable
- size = size
- aspect = aspect (1 = square)
- sharex = share x axis
- sharey = share y axis
Step23:  6.1a HISTOGRAM
Step24:  A more useful example using OECD data
Step25:  6.1b TWO HISTOGRAMS
Step26:  6.1c sns.violinplot
Plot many histograms vis-a-vis
- x = x scale
- y = variable to measure
- hue (optional) = divide violin in two parts
- data = data
- width = size of violin
- inner = show quartiles or boxplot or all data
http
Step27:  6.2 parallel coordinates
To plot how many quantitative variables relate to each other.
- frame
Step28:  We are going to normalize values using standard scores
https
Step29:  6.3 Factorplot
http
<ASSISTANT_TASK:>
Python Code:
#Normal inputs
import pandas as pd
import numpy as np
import seaborn as sns
import pylab as plt
%matplotlib inline
from IPython.display import Image, display
#Make the notebook wider
from IPython.core.display import display, HTML 
display(HTML("<style>.container { width:90% !important; }</style>"))
#Create a toy dataframe
df = pd.DataFrame([[1,2,3],[1,2,3]],columns=["a","b","c"])
df
Explanation: Working with data 2017. Class 2
Contact
Javier Garcia-Bernardo
garcia@uva.nl
0. Structure
Data types, structures and code II
Tidy data: Pivoting and melting dataframes
Merging and concatenating dataframes
Working with missing data
My second plots
End of explanation
#Saving the toy dataframe above
df.to_csv("data/toy_df.csv",sep="\t",index=False)
Explanation: Save CSV to disk (df.to_csv())
path_or_buf = filename
sep = separator
na_rep (default "") = how do you want to represent missing values
header = do you want to save the column names?
index (default True) = is there an index to save? usually False
End of explanation
Image(url="http://www.abs.gov.au/websitedbs/a3121120.nsf/4a256353001af3ed4b2562bb00121564/5869990518bbb7feca257b55002261e0/Body/3.109A!OpenElement&FieldElemFormat=gif")
Explanation: 2. Tidy data
2.1 Observations and variables
A variable is any characteristics, number, or quantity that can be measured or counted: e.g. income, height, happiness
Numeric: Numbers
Continuous: Area, height, etc
Discrete: The rolls of a die, many answers in surveys, etc
Categorical: Categories
Ordinal: Can be ordered. E.g. "high", "middle" and "low" income countries
Nominal: Cannot be ordered. E.g. "blue", "red" and "yellow" 
An observation is each of the values of a variable: e.g. 800, 1.93, 8, "middle"
End of explanation
%%html
<h3>There are three variables in this data set. What are they? </h3> 
<table>
    <tr> <th></th> <th>2010</th> <th>2015</th> </tr>
    <tr> <td>Male</td> <td>0</td> <td>5</td> </tr>
    <tr> <td>Female</td> <td>1</td> <td>4</td> </tr>
</table>
%%html
<h3>Tidy data</h3> 
<table>
    <tr> <th>year</th> <th>sex</th> <th>count</th> </tr>
    <tr> <td>2010</td> <td>female</td> <td>1</td> </tr>
    <tr> <td>2010</td> <td>male</td> <td>0</td> </tr>
    <tr> <td>2015</td> <td>female</td> <td>4</td> </tr>
    <tr> <td>2015</td> <td>male</td> <td>5</td> </tr>
</table>
Explanation: 2.2 What is tidy data?
Based on: http://vita.had.co.nz/papers/tidy-data.pdf
Tidy data is a standard way of mapping the meaning of a dataset to its structure.  [...] In
tidy data:
1.  Each variable forms a column.
2.  Each observation forms a row.
3.  Each type of observational unit forms a table.
Advantages: Facilitate manipulation, visualisation and modelling
Combining tables (merging two tables by the year)
Plotting (df["x"] vs df["y"])
Statistical models ("y" ~ "x" + C("year"))
Examples adapted from http://stat405.had.co.nz/lectures/18-tidy-data.pdf
End of explanation
#Example of melting (columns to rows)
print("Melting")
Image("figures/melt.png",width=400)
#Example of pivoting (rows to columns)
print("Pivoting")
Image("figures/pivot.png",width=800)
Explanation: 2.3 Tidying messy datasets
Real datasets can, and often do, violate the three precepts of tidy data in almost every way
imaginable. While occasionally you do get a dataset that you can start analysing immediately,
this is the exception,  not the rule:
- Column headers are values, not variable names --> Melt
- Multiple variables are stored in one column --> Pivot
- Variables are stored in both rows and columns --> Melt then pivot
End of explanation
#Data on population density by year
df = pd.read_csv("data/columns_headers.csv")
df.head()
### Fix
#Variables present in columns already
variables_already_presents = ['METRO_ID', 'Metropolitan areas']
#Variables to combine in one
columns_combine = ['2000', '2001', '2002','2003', '2004', '2005', '2006', '2007', '2008', '2009', '2010', '2011','2012', '2013', '2014']
#Fix
fixed = pd.melt(df,id_vars=variables_already_presents,
           value_vars=columns_combine,
           var_name="Year",
           value_name="POP_DENS")
fixed.head(10)
Explanation: Data to study: variations of https://stats.oecd.org/Index.aspx?DataSetCode=CITIES (which has problem 2)
2.4 Column headers are values, not variable names: MELT
End of explanation
#Actual data from OECD
df = pd.read_csv("data/multiple_variables.csv")
df.head(2)
### Fix
#Columns already present
variables_already_present = ["METRO_ID","Metropolitan areas","Year"]
#Column with the variables
column_to_split = ["VAR"]
#Column with the values
column_with_values = "Value"
#Fix
df_fixed = df.pivot_table(column_with_values,
               variables_already_present,
               column_to_split)
df_fixed.head()
#use .reset_index() to convert index to columns
df_fixed.reset_index()
Explanation: 2.5 Multiple variables are stored in one column: PIVOT_TABLE
End of explanation
df = pd.read_csv("data/multiple_variables_and_headers.csv")
df.head()
print(list(df.columns)[4:])
cols = []
for column in df.columns:
    try: 
        int(column)
        cols.append(column)
    except:
        pass
print(cols)
### Fix setp 1
variables_already_presents = ['METRO_ID', 'Metropolitan areas','VAR']
columns_combine = cols
df = pd.melt(df,
           id_vars=variables_already_presents,
           value_vars=columns_combine,
           var_name="Year",
           value_name="Value")
df.head()
### Fix step 2
column_with_values = "Value"
column_to_split = ["VAR"]
variables_already_present = ["METRO_ID","Metropolitan areas","Year"]
df.pivot_table(column_with_values,
               variables_already_present,
               column_to_split).reset_index().head()
Explanation: 2.6 Variables are stored in both rows and columns: MELT then PIVOT
End of explanation
#Add more observations
Image(url="http://pandas.pydata.org/pandas-docs/stable/_images/merging_concat_basic.png")
madrid = pd.read_csv("data/mad_twovariables.csv").head(2)
amsterdam = pd.read_csv("data/ams_twovariables.csv").head(2)
display(madrid)
display(amsterdam)
combined = pd.concat([madrid,amsterdam])
combined
Explanation: 3. Merging and concatenating datasets
Concatenate:
- Some times you obtain more data (with the same variables or different)
Merging:
- Many times you have several datasets that you have to combine according to a shared variable
3.1 Concatenate files (add more observations)
End of explanation
a = pd.DataFrame([[1,2],[3,4]],columns=["a","b"])
b = pd.DataFrame([[1,2],[3,4]],columns=["c","b"])
display(a)
display(b)
pd.concat([a,b])
Explanation: Note that it's not a problem if the variables are not all the same
But in that case you'll have missing values
End of explanation
Image(url="http://pandas.pydata.org/pandas-docs/stable/_images/merging_merge_on_key.png",width=700)
ams_green = pd.read_csv("data/ams_green.csv")
ams_pop = pd.read_csv("data/ams_popdens.csv")
display(ams_green.head(2))
display(ams_pop.head(2))
combined = pd.merge(ams_green,ams_pop,
         left_on=["METRO_ID","Metropolitan areas","Year"],
         right_on=["METRO_ID","Metropolitan areas","Year"],
         how="inner")
combined.head()
print("Inner join pd.merge(left_df,right_df,how=\"inner\")")
display(Image(url="http://pandas.pydata.org/pandas-docs/stable/_images/merging_merge_on_key_inner.png"))
print("Right join pd.merge(left_df,right_df,how=\"right\")", "The same with left")
display(Image(url="http://pandas.pydata.org/pandas-docs/stable/_images/merging_merge_on_key_right.png"))
print("Outer join pd.merge(left_df,right_df,how=\"outer\")")
display(Image(url="http://pandas.pydata.org/pandas-docs/stable/_images/merging_merge_on_key_outer.png"))
Explanation: 3.2 Merge datasets
End of explanation
def mean_(some_numbers):
    return np.mean(some_numbers)
a = mean_([1,2,3])
print(a)
np.mean([1,2,3])
def read_our_csv():
    #reading the raw data from oecd
    df = pd.read_csv("data/CITIES_19122016195113034.csv",sep="\t")
    #fixing the columns (the first one is ""METRO_ID"" instead of "METRO_ID")
    cols = list(df.columns)
    cols[0] = "METRO_ID"
    df.columns = cols
    
    #pivot the table
    column_with_values = "Value"
    column_to_split = ["VAR"]
    variables_already_present = ["METRO_ID","Metropolitan areas","Year"]
    df_fixed = df.pivot_table(column_with_values,
                 variables_already_present,
                 column_to_split).reset_index()
    
    return df_fixed
    
df_fixed = read_our_csv()
df_fixed.head()
Explanation: 4. Change columns, replace and apply functions
Some times we want to replace some categories with numbers
This is especially important with text (and we'll talk more in the class about text)
End of explanation
#Read and fix the data
df_fixed = read_our_csv()
df_fixed = df_fixed.loc[df_fixed["Year"]==2000,["GREEN_AREA_PC","POP_DENS"]]    
df_fixed.head()
plt.figure(figsize=(6,4))
plt.scatter(df_fixed["GREEN_AREA_PC"],df_fixed["POP_DENS"],edgecolor="none",color=(70/255,140/255,210/255))
plt.yscale("log")
plt.xscale("log")
plt.ylim(10,10000)
plt.xlim(10,10000)
plt.xlabel("Green area (m^2/million people)")
plt.ylabel("Density population (people/km^s)")
Explanation: 4.1 Multiply columns
End of explanation
#fraction of green area
df_fixed["fraction_green"] = df_fixed["GREEN_AREA_PC"]*df_fixed["POP_DENS"]/1E6
#make the figure
plt.figure(figsize=(6,4))
plt.scatter(df_fixed["fraction_green"],df_fixed["POP_DENS"],edgecolor="none",color=(70/255,140/255,210/255))
plt.yscale("log")
plt.xscale("log")
plt.ylim(10,10000)
plt.xlim(0.0001,1)
plt.xlabel("Percentage green (m^2 green / km^2 land)")
plt.ylabel("Density population (population/km^2)")
Explanation: Is there a relationship between green area per capita and population density? Why?
Imagine we want to calculate the fraction of green area
$$\frac{10^{6} m^2_{green}}{people} \cdot \frac{people}{km^2_{city}} = \frac{10^{6} m^2_{green}}{km^2_{city}}$$
End of explanation
#Imagine we want to convert the income to a category
df = pd.DataFrame({"kids": [2,3,3,1,3,1,4,4,2], "income": [">100",">100","<40","<40","40-100","<40","40-100",">100","40-100"]})
df
d_cat2income = {">100": 100, 
                "40-100": 70, 
                "<40": 40}
d_cat2income
d_cat2income.get(">90")
df["income_number"] = df["income"].apply(lambda x: 
                                         x[0])
df.head()
#First we need a dictionary
d_cat2income = {">100": 100, "40-100": 70, "<40": 40}
def category2income(value):
    return d_cat2income.get(value)
df = pd.DataFrame({"kids": [2,3,3,1,3,1,4,4,2], "income": [">100",">100","40-100","<40","40-100","<40","40-100",">100","40-100"]})
df["income"] = df["income"].apply(category2income)
df.head()
#Or without creating a new function
df = pd.DataFrame({"kids": [2,3,3,1,3,1,4,4,2], "income": [">100",">100","40-100","<40","40-100","<40","40-100",">100","40-100"]})
df["income"] = df["income"].apply(lambda x: d_cat2income.get(x))
df.head()
Explanation: 4.2 Categories to numbers
End of explanation
df_fixed = read_our_csv().loc[:,["METRO_ID","Year"]]
df_fixed.head()
def keep_2_letters(value):
    return value[:2]
df_fixed["Country"] = df_fixed["METRO_ID"].apply(keep_2_letters)
df_fixed.head()
#or in one line
df_fixed["Country"] = df_fixed["METRO_ID"].apply(lambda x: x[:2])
df_fixed.head()
Explanation: 4.2 Keeping part of a the values in a column (text)
We'll talk more about this when we talk about text analysis
End of explanation
#for example doing the log of a column. This is useful when the distribution is lognormal (such as income, gdp).
#But somehow confusing, the distribution of the income of many people IS LOGNORMAL.
#The distribution of the mean income of many cities IS NOT LOGNORMAL.
df_fixed = read_our_csv().loc[:,["METRO_ID","Year","GDP_PC"]]
df_fixed["GDP_PC_log"] = np.log10(df_fixed["GDP_PC"])
df_fixed.head()
Explanation: 4.3 Doing math with the column
End of explanation
#Read and fix the data
df_fixed = read_our_csv()
#Remove rows with missing values
cols = ["LABOUR_PRODUCTIVITY","UNEMP_R","GDP_PC"]
df_fixed = df_fixed.dropna(subset=cols)
#Creating a column for country
df_fixed["C"] = df_fixed["METRO_ID"].apply(lambda x: x[:2])
#Keeping italy
df_fixed = df_fixed.loc[df_fixed["C"]=="IT",["C","METRO_ID","Metropolitan areas"] +cols]
#We are going to normalize values dividing by the mean (so new values have a mean of 1)
df_fixed.loc[:,cols] = df_fixed[cols]/np.nanmean(df_fixed[cols],0)
#Make a copy of the data
df_original = df_fixed.copy()
#Take a random sample of 20 values of productivity
sample = set(df_fixed.loc[:,"LABOUR_PRODUCTIVITY"].sample(20))
#Deleting those values (saying that they are np.NaN (missing))
df_fixed.loc[df_fixed["LABOUR_PRODUCTIVITY"].isin(sample),"LABOUR_PRODUCTIVITY"] = np.NaN
df_fixed.head()
Explanation: 5. Dealing with missing data
Let's imagine we have a data of a survey, with age, income, education, political ideas, location, and if you will vote for Trump or Hillary.
We may have some missing values. This missing values can be:
- MCAR (missing completely at random), which means that we have a representative sample. 
    - This for example could happen if during the survey collection there were some IT problems.
    - It is usually a strong assumption but it is usually made.
    - However imagine if young voters of Trump are less likely to answer --> Then your data is MAR.
    - Usually in surveys you make sure you ask to a percentage of people of age and location that correspond with the real population. But you may be missing an important variable (for example US pollsters didn't ask to a representative sample in terms of education). 
    - We can either ignore the rows or impute the values based on people that did answer. 
MAR (missing at random), which means that we don't have a representative sample, but we can use another column to impute missing values. 
We can correct the data by using other people that did answer. For instance, two people living in the same area, with the same age, income, education and political ideas are likely to vote similar, so if you only know how one of them intends to vote you can say that the other one will vote the same (there are methods for this, don't do it by hand!)
MNAR (missing not at random), which means that we don't have a representative sample, and imputation is very very hard. This can happen for example if Trump voters are less likely to open the door, then they are not even in your sample.
We are in trouble and the methods to correct for this are way beyond the scope of the class.
What are the strategies to correct for missing values?
4.1 Ignore those values (only if your data is MCAR)
Impute those values (always better but more complicated)
4.2 Use the mean/median/mode as the value (only works well in MCAR)
4.3 Use similar values -> fancyimpute (another time)
End of explanation
#How to fix by ignoring the rows
ignoring = df_fixed.dropna(subset=["LABOUR_PRODUCTIVITY"])
Explanation: 4.1 Ignoring data
End of explanation
#How to fix by imputing with mean/median/mode
mean_inputed = df_fixed.fillna(df_fixed.mean())
Explanation: 4.3 Imputing with mean/median/mode
End of explanation
#Based on this. Similar points for unemployment have similar points for productivity
sns.lmplot(x="LABOUR_PRODUCTIVITY",y="UNEMP_R",data=df_fixed,fit_reg=False)
Image("figures/labels.png")
print("Using a random sample => MCAR DATA")
Image("figures/kmeans_vs_mean.png")
print("Using a biasad sample => MAR DATA")
Image("figures/kmeans_vs_mean_worst_case.png")
Explanation: 4.4 Imputing using neighbous
We may go over this another day, it's not an easy topic.
But the basic idea is that you are probably similar to your neighbors (variables are correlated)
End of explanation
#Read and fix the data. also drop missing values
df = read_our_csv().loc[:,["METRO_ID","Year","Metropolitan areas","LABOUR_PRODUCTIVITY","UNEMP_R"]]
#Drop rows with missing values
df = df.dropna()
#Keep years 2000 and 2012
df = df.loc[df["Year"].isin([2000,2012])]
#Add a column for country
df["C"] = df["METRO_ID"].apply(lambda x: x[:2] if isinstance(x,str) else "")
df.head()
Explanation: But first of all we need to make sure we have missing values instead of "-9" or other strings
The way to do it is to explore the file and then read it:
pd.read_csv(filename,...,na_values=[here the list of missing values]
5. Dealing with outliers
In short:
- If the outlier is obviously wrong you should treat it as missing data. For instance if you record a value of 19 in a 0-10 scale (it was problably 1 or 9, but we don't know).
- If the outlier is *not obviously wrong", you should keep it. For instance if most people say 3 in a 0-10 scale but a couple say 9, then you need to keep it. You can use robust statistics, that are not so affected by outliers. 
More info at another time, but read this:
http://www.theanalysisfactor.com/outliers-to-drop-or-not-to-drop/
6. Plotting
Another scatter plot
Histogram, violinplot and jointplot
Parallel coordinates
Factorplot
End of explanation
sns.lmplot?
keep_countries = ['IT', 'NL',  'DE',  'ES', 'GR']
sns.lmplot(x="LABOUR_PRODUCTIVITY",y="UNEMP_R",hue="C",col="Year"
           ,data=df.loc[df["C"].isin(keep_countries)],fit_reg=False,
          size=3,aspect=1.4,sharex=True,sharey=True)
plt.show()
Explanation: 6.0 SCATTER using sns.lmplot
Use to plot the relationship between two quantitative variables (and one qualitative variable using color)
- x = x axis
- y = y axis
- h = separate using color qualitative variable
- data = dataframe
- fit_reg = True to fit regression lines
- col = separate by another qualitative variable
- row = separate by anoyther qualitative variable
- size = size
- aspect = aspect (1 = square)
- sharex = share x axis
- sharey = share y axis
End of explanation
#Roll two dices 100 times
dice_rolls = np.random.randint(1,7,1000) + np.random.randint(1,7,1000)
dice_rolls
#Count the number of each element to create the distribution
from collections import Counter
Counter(dice_rolls)
#And we can visualize it with a histogram
sns.distplot(dice_rolls, kde=False, rug=False)
Explanation: 6.1a HISTOGRAM: sns.distplot
Creates a histogram. Important to see the distribution of your data, to find outliers, etc.
x: required, the data
rug (default True): plot ticks indicating where the value exists 
kde (default Ture): use kernel distribution estimator to plot the shape
hist_kws={"edgecolor":"none"}
kde_kws={"cut":0}
fit_kws={"cut":0,"color":"gray"})
What's a histogram?
A representation of the distribution of a variable
For example. If you draw two dice, then you will get a lot of 7s, many 6s and 8s, some 5s and 9s, a few 4s and 10st, very few32s and 11st and almost no 2s ans 12st.
End of explanation
#Read and fix the data. also drop missing values
df = read_our_csv().loc[:,["METRO_ID","Year","Metropolitan areas","LABOUR_PRODUCTIVITY","UNEMP_R"]]
#Drop rows with missing values
df = df.dropna()
#Keep years 2000 and 2012
df = df.loc[df["Year"].isin([2000,2012])]
#Add a column for country
df["C"] = df["METRO_ID"].apply(lambda x: x[:2] if isinstance(x,str) else "")
df.head()
#Keep x and y values
x = df["LABOUR_PRODUCTIVITY"]
#Make plot
plt.figure(figsize=(6,4))
sns.distplot(x, kde=True, rug=False,hist_kws={"edgecolor":"none"},kde_kws={"cut":0})
plt.ylabel("Frequency")
plt.xlabel("Labour productivity")
#fitting a distribution
from scipy.stats import norm,lognorm,expon
#Keep x and y values
x = df["LABOUR_PRODUCTIVITY"]
#Make plot
plt.figure(figsize=(6,4))
sns.distplot(x, kde=False, fit=norm, rug=False, 
             hist_kws={"edgecolor":"none","normed":True},fit_kws={"cut":0,"color":"gray"})
plt.ylabel("Frequency")
plt.xlabel("Labour productivity")
Explanation: A more useful example using OECD data
End of explanation
sns.jointplot?
#Make plot
sns.jointplot(x="LABOUR_PRODUCTIVITY", y="UNEMP_R", data=df,size=6)
plt.ylabel("Frequency")
plt.xlabel("Labour productivity")
plt.show()
sns.jointplot(x="LABOUR_PRODUCTIVITY", y="UNEMP_R", data=df,kind="hex",
              marginal_kws=dict(bins=20, rug=False, kde=True, kde_kws={"cut":0}), gridsize = 15,size=6)
Explanation: 6.1b TWO HISTOGRAMS: sns.jointplot
Creates two histogrmams one vs the other. Important to see the distribution of your data, to find outliers, etc.
Also important when plotting events in a map
- x: required, the data
- y: required, the other data
- kind (default "scatter"): kind of plot to draw, scatter and hex are nice
- marginal_kws=dict(bins=20, rug=True, kde=True) -> To modify the outside number of bins  
- gridsize = 10 -> To modify the inside number of bins
http://seaborn.pydata.org/generated/seaborn.jointplot.html
End of explanation
cols_get = ["METRO_ID","Metropolitan areas","Year","UNEMP_R",'GDP_PC',"LABOUR_PRODUCTIVITY"]
df_fixed = read_our_csv().loc[:,cols_get]
df_fixed["C"] = df_fixed["METRO_ID"].apply(lambda x: x[:2] if isinstance(x,str) else "")
south = df_fixed.loc[df_fixed["C"].isin(["IT","ES","GR","DE"])]
south.head()
plt.figure(figsize=(6,4))
sns.violinplot(x="C", y="LABOUR_PRODUCTIVITY", data=south,width=1,inner="quartiles")
Explanation: 6.1c sns.violinplot
Plot many histograms vis-a-vis
- x = x scale
- y = variable to measure
- hue (optional) = divide violin in two parts
- data = data
- width = size of violin
- inner = show quartiles or boxplot or all data
http://seaborn.pydata.org/generated/seaborn.violinplot.html
End of explanation
from pandas.tools.plotting import parallel_coordinates
cols_get = ["METRO_ID","Metropolitan areas","Year","UNEMP_R",'GDP_PC',"LABOUR_PRODUCTIVITY"]
df_fixed = read_our_csv().loc[:,cols_get]
df_fixed["C"] = df_fixed["METRO_ID"].apply(lambda x: x[:2] if isinstance(x,str) else "")
italy = df_fixed.loc[df_fixed["C"].isin(["IT"])]
italy = italy.loc[italy["Year"].isin([2001])]
italy.head()
Explanation: 6.2 parallel coordinates
To plot how many quantitative variables relate to each other.
- frame: the data
- class_column: column containing the labels (for different colors)
- cols: The list of columns to use
- colormap: colormap to use
End of explanation
#Here we normalize the variables so th
cols = ["UNEMP_R",'GDP_PC',"LABOUR_PRODUCTIVITY"]
for c in cols: 
    italy[c] = (italy[c]- np.mean(italy[c]))/np.std(italy[c]) 
    
parallel_coordinates(italy.loc[(italy["Metropolitan areas"] !="Italy")],"Year",cols=cols,colormap="Paired")
d_city2regio = {"Bari": "S",
"Bologna": "C",
"Catania": "S",
"Florence": "C",
"Genova": "N",
"Milan": "N",
"Naples": "S",
"Palermo": "S",
"Rome": "C",
"Turin": "N",
"Venice": "N"}
italy["Area"] = italy["Metropolitan areas"].apply(lambda x: d_city2regio.get(x))
parallel_coordinates(italy.loc[(italy["Metropolitan areas"] !="Italy")],"Area",cols=cols,colormap="Paired")
Explanation: We are going to normalize values using standard scores
https://en.wikipedia.org/wiki/Standard_score
A series of values have a mean m and a standard deviation d
If you substract all values by m and divide them by d ($\frac{x - mean(x)}{std(x)}$) your new values have a mean of 0 and a standard deviation of 1
End of explanation
cols_get = ["METRO_ID","Metropolitan areas","Year","UNEMP_R",'GDP_PC',"LABOUR_PRODUCTIVITY"]
df_fixed = read_our_csv().loc[:,cols_get]
df_fixed["C"] = df_fixed["METRO_ID"].apply(lambda x: x[:2] if isinstance(x,str) else "")
south = df_fixed.loc[df_fixed["C"].isin(["IT","ES","GR","DE"])]
south.head()
sns.factorplot(x="Year", y="UNEMP_R", hue="C", size=4,aspect=1.8,data=south,ci=95)
sns.factorplot(x="Year", y="UNEMP_R", hue="C", size=4,aspect=3,
               data=south.loc[south["Year"].isin([2000,2010])],kind="violin",width=0.9,inner="quartiles")
Explanation: 6.3 Factorplot
http://seaborn.pydata.org/generated/seaborn.factorplot.html
To visualize interactions between variables, as well as many distributions.
- x = x axis
- y = y axis
- data = dataframe
- hue = divide into several lines
- size = size of figure
- aspect = aspect of figure (1 = square)
- ci = size of confidence intervals (0 for no confidence intervals)
See examples for documentation
End of explanation
<END_TASK>