<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data Analysis Tools
Assignment
Step1:  General information on the Gapminder data
Step2:  Data analysis
Step3:  Correlation test
Step4:  The Pearson test proves a significant positive relationship between income per person and residential electricity consumption as the p-value is below 0.05.
Moreover, the square of the correlation coefficient, i.e. the coefficient of determination, is 0.425. This means that we can predict 42.5% of the variability of residential electricity consumption knowing the income per person.
Potential moderator
Now comes the analysis of Pearson correlation between different urban rate group to see if the urban rate is a moderator on the relationship between income per person and residential electricity consumption.
Step5:  For all urban rate categories, the p-value is below the threshold of 0.05. Therefore the urban rate does not moderate the relationship between income per person and residential electricity consumption. In other words the residential electricity consumption has a significant positive relationship in regard to the income per person whatever the urban rate in the country.
By plotting the scatter plots of the four groups, we can see that the correlation is indeed present for all of them. One corollary finding from the graphics below is the tendency of countries with higher income per person to have higher urban rate.
<ASSISTANT_TASK:>
Python Code:
# Magic command to insert the graph directly in the notebook
%matplotlib inline
# Load a useful Python libraries for handling data
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
import scipy.stats as stats
import seaborn as sns
import matplotlib.pyplot as plt
from IPython.display import Markdown, display
# Read the data
data_filename = r'gapminder.csv'
data = pd.read_csv(data_filename, low_memory=False)
data = data.set_index('country')
Explanation: Data Analysis Tools
Assignment: Testing a Potential Moderator
Following is the Python program I wrote to fulfill the last assignment of the Data Analysis Tools online course.
I used Jupyter Notebook as it is a pretty way to write code and present results.
Research question for this assignment
Using the Gapminder database, I found a significant correlation between the income per person (the explanatory variable) and the residential electricity consumption (the response variable). For this exercice, I would like to see if the urban rate is a potential moderator.
Data management
For the question I'm interested in, the countries for which data are missing will be discarded. As missing data in Gapminder database are replace directly by NaN no special data treatment is needed.
End of explanation
display(Markdown("Number of countries: {}".format(len(data))))
display(Markdown("Number of variables: {}".format(len(data.columns))))
subdata2 = (data[['incomeperperson', 'urbanrate', 'relectricperperson']]
                .assign(income=lambda x: pd.to_numeric(x['incomeperperson'], errors='coerce'),
                        urbanrate=lambda x: pd.to_numeric(x['urbanrate'], errors='coerce'),
                        electricity=lambda x: pd.to_numeric(x['relectricperperson'], errors='coerce'))
                .dropna())
Explanation: General information on the Gapminder data
End of explanation
sns.distplot(subdata2.income)
plt.xlabel("Income per person (constant 2000 US$)")
_ = plt.title("Distribution of the income per person")
sns.distplot(subdata2.electricity)
plt.xlabel("Residential electricity consumption (kWh)")
_ = plt.title("Distribution of the residential electricity consumption")
sns.distplot(subdata2.urbanrate)
plt.xlabel("Urban rate (%)")
_ = plt.title("Urban rate distribution")
Explanation: Data analysis
End of explanation
sns.regplot(x='income', y='electricity', data=subdata2)
plt.xlabel('Income per person (2000 US$)')
plt.ylabel('Residential electricity consumption (kWh)')
_ = plt.title('Scatterplot for the association between the income and the residential electricity consumption')
correlation, pvalue = stats.pearsonr(subdata2['income'], subdata2['electricity'])
display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))
display(Markdown("And the coefficient of determination is {:.3g}.".format(correlation**2)))
Explanation: Correlation test
End of explanation
def urban_group(row):
    if row['urbanrate'] < 25.0:
        return '0%<=..<25%'
    elif row['urbanrate'] < 50.0:
        return '25%<=..<50%'
    elif row['urbanrate'] < 75.0:
        return '50%<=..<75%'
    else:
        return '75%=<'
    
subdata3 = subdata2.copy()
subdata3['urban_group'] = pd.Categorical(subdata3.apply(lambda x: urban_group(x), axis=1))
summary = dict()
for group in subdata3.urban_group.cat.categories:
    moderator_group = subdata3[subdata3['urban_group'] == group]
    summary[group] = stats.pearsonr(moderator_group['income'], moderator_group['electricity'])
    
df = (pd.DataFrame(summary)
        .rename(index={0:'Pearson r', 1:'p-value'}))
df2 = (df.stack()
         .unstack(level=0))
df2.index.name = 'Urban rate'
df2
Explanation: The Pearson test proves a significant positive relationship between income per person and residential electricity consumption as the p-value is below 0.05.
Moreover, the square of the correlation coefficient, i.e. the coefficient of determination, is 0.425. This means that we can predict 42.5% of the variability of residential electricity consumption knowing the income per person.
Potential moderator
Now comes the analysis of Pearson correlation between different urban rate group to see if the urban rate is a moderator on the relationship between income per person and residential electricity consumption.
End of explanation
g = sns.FacetGrid(subdata3.reset_index(),
                  col='urban_group', hue='urban_group', col_wrap=2, size=4)
_ =g.map(sns.regplot, 'income', 'electricity')
Explanation: For all urban rate categories, the p-value is below the threshold of 0.05. Therefore the urban rate does not moderate the relationship between income per person and residential electricity consumption. In other words the residential electricity consumption has a significant positive relationship in regard to the income per person whatever the urban rate in the country.
By plotting the scatter plots of the four groups, we can see that the correlation is indeed present for all of them. One corollary finding from the graphics below is the tendency of countries with higher income per person to have higher urban rate.
End of explanation
<END_TASK>