<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Examples and Exercises from Think Stats, 2nd Edition
http
Step1:  Scatter plots
I'll start with the data from the BRFSS again.
Step2:  The following function selects a random subset of a DataFrame.
Step3:  I'll extract the height in cm and the weight in kg of the respondents in the sample.
Step4:  Here's a simple scatter plot with alpha=1, so each data point is fully saturated.
Step5:  The data fall in obvious columns because they were rounded off.  We can reduce this visual artifact by adding some random noice to the data.
NOTE
Step6:  Heights were probably rounded off to the nearest inch, which is 2.8 cm, so I'll add random values from -1.4 to 1.4.
Step7:  And here's what the jittered data look like.
Step8:  The columns are gone, but now we have a different problem
Step9:  That's better.  This version of the figure shows the location and shape of the distribution most accurately.  There are still some apparent columns and rows where, most likely, people reported their height and weight using rounded values.  If that effect is important, this figure makes it apparent; if it is not important, we could use more aggressive jittering to minimize it. 
An alternative to a scatter plot is something like a HexBin plot, which breaks the plane into bins, counts the number of respondents in each bin, and colors each bin in proportion to its count.
Step10:  In this case the binned plot does a pretty good job of showing the location and shape of the distribution.  It obscures the row and column effects, which may or may not be a good thing.
Exercise
Step11:  Plotting percentiles
Sometimes a better way to get a sense of the relationship between variables is to divide the dataset into groups using one variable, and then plot percentiles of the other variable.
First I'll drop any rows that are missing height or weight.
Step12:  Then I'll divide the dataset into groups by height.
Step13:  Here are the number of respondents in each group
Step14:  Now we can compute the CDF of weight within each group.
Step15:  And then extract the 25th, 50th, and 75th percentile from each group.
Step16:  Exercise
Step17:  Correlation
The following function computes the covariance of two variables using NumPy's dot function.
Step18:  And here's an example
Step19:  Covariance is useful for some calculations, but it doesn't mean much by itself.  The coefficient of correlation is a standardized version of covariance that is easier to interpret.
Step20:  The correlation of height and weight is about 0.51, which is a moderately strong correlation.
Step21:  NumPy provides a function that computes correlations, too
Step22:  The result is a matrix with self-correlations on the diagonal (which are always 1), and cross-correlations on the off-diagonals (which are always symmetric).
Pearson's correlation is not robust in the presence of outliers, and it tends to underestimate the strength of non-linear relationships.
Spearman's correlation is more robust, and it can handle non-linear relationships as long as they are monotonic.  Here's a function that computes Spearman's correlation
Step23:  For heights and weights, Spearman's correlation is a little higher
Step24:  A Pandas Series provides a method that computes correlations, and it offers spearman as one of the options.
Step25:  The result is the same as for the one we wrote.
Step26:  An alternative to Spearman's correlation is to transform one or both of the variables in a way that makes the relationship closer to linear, and the compute Pearson's correlation.
Step27:  Exercises
Using data from the NSFG, make a scatter plot of birth weight versus mother’s age. Plot percentiles of birth weight versus mother’s age. Compute Pearson’s and Spearman’s correlations. How would you characterize the relationship between these variables?
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function, division
%matplotlib inline
import numpy as np
import brfss
import thinkstats2
import thinkplot
Explanation: Examples and Exercises from Think Stats, 2nd Edition
http://thinkstats2.com
Copyright 2016 Allen B. Downey
MIT License: https://opensource.org/licenses/MIT
End of explanation
df = brfss.ReadBrfss(nrows=None)
Explanation: Scatter plots
I'll start with the data from the BRFSS again.
End of explanation
def SampleRows(df, nrows, replace=False):
    indices = np.random.choice(df.index, nrows, replace=replace)
    sample = df.loc[indices]
    return sample
Explanation: The following function selects a random subset of a DataFrame.
End of explanation
sample = SampleRows(df, 5000)
heights, weights = sample.htm3, sample.wtkg2
Explanation: I'll extract the height in cm and the weight in kg of the respondents in the sample.
End of explanation
thinkplot.Scatter(heights, weights, alpha=1)
thinkplot.Config(xlabel='Height (cm)',
                 ylabel='Weight (kg)',
                 axis=[140, 210, 20, 200],
                 legend=False)
Explanation: Here's a simple scatter plot with alpha=1, so each data point is fully saturated.
End of explanation
def Jitter(values, jitter=0.5):
    n = len(values)
    return np.random.normal(0, jitter, n) + values
Explanation: The data fall in obvious columns because they were rounded off.  We can reduce this visual artifact by adding some random noice to the data.
NOTE: The version of Jitter in the book uses noise with a uniform distribution.  Here I am using a normal distribution.  The normal distribution does a better job of blurring artifacts, but the uniform distribution might be more true to the data.
End of explanation
heights = Jitter(heights, 1.4)
weights = Jitter(weights, 0.5)
Explanation: Heights were probably rounded off to the nearest inch, which is 2.8 cm, so I'll add random values from -1.4 to 1.4.
End of explanation
thinkplot.Scatter(heights, weights, alpha=1.0)
thinkplot.Config(xlabel='Height (cm)',
                 ylabel='Weight (kg)',
                 axis=[140, 210, 20, 200],
                 legend=False)
Explanation: And here's what the jittered data look like.
End of explanation
thinkplot.Scatter(heights, weights, alpha=0.1, s=10)
thinkplot.Config(xlabel='Height (cm)',
                 ylabel='Weight (kg)',
                 axis=[140, 210, 20, 200],
                 legend=False)
Explanation: The columns are gone, but now we have a different problem: saturation.  Where there are many overlapping points, the plot is not as dark as it should be, which means that the outliers are darker than they should be, which gives the impression that the data are more scattered than they actually are.
This is a surprisingly common problem, even in papers published in peer-reviewed journals.
We can usually solve the saturation problem by adjusting alpha and the size of the markers, s.
End of explanation
thinkplot.HexBin(heights, weights)
thinkplot.Config(xlabel='Height (cm)',
                 ylabel='Weight (kg)',
                 axis=[140, 210, 20, 200],
                 legend=False)
Explanation: That's better.  This version of the figure shows the location and shape of the distribution most accurately.  There are still some apparent columns and rows where, most likely, people reported their height and weight using rounded values.  If that effect is important, this figure makes it apparent; if it is not important, we could use more aggressive jittering to minimize it. 
An alternative to a scatter plot is something like a HexBin plot, which breaks the plane into bins, counts the number of respondents in each bin, and colors each bin in proportion to its count.
End of explanation
# Solution goes here
heights = Jitter(df.htm3, 1.4)
weights = Jitter(df.wtkg2, 1)
thinkplot.Scatter(heights, weights, alpha=0.05, s=2)
thinkplot.Config(axis=[135, 205, 25, 210])
Explanation: In this case the binned plot does a pretty good job of showing the location and shape of the distribution.  It obscures the row and column effects, which may or may not be a good thing.
Exercise:  So far we have been working with a subset of only 5000 respondents.  When we include the entire dataset, making an effective scatterplot can be tricky.  As an exercise, experiment with Scatter and HexBin to make a plot that represents the entire dataset well.
End of explanation
cleaned = df.dropna(subset=['htm3', 'wtkg2'])
Explanation: Plotting percentiles
Sometimes a better way to get a sense of the relationship between variables is to divide the dataset into groups using one variable, and then plot percentiles of the other variable.
First I'll drop any rows that are missing height or weight.
End of explanation
bins = np.arange(135, 210, 5)
indices = np.digitize(cleaned.htm3, bins)
groups = cleaned.groupby(indices)
Explanation: Then I'll divide the dataset into groups by height.
End of explanation
for i, group in groups:
    print(i, len(group))
Explanation: Here are the number of respondents in each group:
End of explanation
mean_heights = [group.htm3.mean() for i, group in groups]
cdfs = [thinkstats2.Cdf(group.wtkg2) for i, group in groups]
Explanation: Now we can compute the CDF of weight within each group.
End of explanation
for percent in [75, 50, 25]:
    weight_percentiles = [cdf.Percentile(percent) for cdf in cdfs]
    label = '%dth' % percent
    thinkplot.Plot(mean_heights, weight_percentiles, label=label)
    
thinkplot.Config(xlabel='Height (cm)',
                 ylabel='Weight (kg)',
                 axis=[140, 210, 20, 200],
                 legend=False)
Explanation: And then extract the 25th, 50th, and 75th percentile from each group.
End of explanation
# Solution goes here
cleaned.head()
bins = np.arange(140, 210, 10)
indices = np.digitize(cleaned.htm3, bins)
groups = cleaned.groupby(indices)
cdfs = [thinkstats2.Cdf(group.wtkg2) for i, group in groups]
thinkplot.Cdfs(cdfs)
thinkplot.Config(xlabel='weight', ylabel='cdf')
Explanation: Exercise: Yet another option is to divide the dataset into groups and then plot the CDF for each group.  As an exercise, divide the dataset into a smaller number of groups and plot the CDF for each group.
End of explanation
def Cov(xs, ys, meanx=None, meany=None):
    xs = np.asarray(xs)
    ys = np.asarray(ys)
    if meanx is None:
        meanx = np.mean(xs)
    if meany is None:
        meany = np.mean(ys)
    cov = np.dot(xs-meanx, ys-meany) / len(xs)
    return cov
Explanation: Correlation
The following function computes the covariance of two variables using NumPy's dot function.
End of explanation
heights, weights = cleaned.htm3, cleaned.wtkg2
Cov(heights, weights)
Explanation: And here's an example:
End of explanation
def Corr(xs, ys):
    xs = np.asarray(xs)
    ys = np.asarray(ys)
    meanx, varx = thinkstats2.MeanVar(xs)
    meany, vary = thinkstats2.MeanVar(ys)
    corr = Cov(xs, ys, meanx, meany) / np.sqrt(varx * vary)
    return corr
Explanation: Covariance is useful for some calculations, but it doesn't mean much by itself.  The coefficient of correlation is a standardized version of covariance that is easier to interpret.
End of explanation
Corr(heights, weights)
Explanation: The correlation of height and weight is about 0.51, which is a moderately strong correlation.
End of explanation
np.corrcoef(heights, weights)
Explanation: NumPy provides a function that computes correlations, too:
End of explanation
import pandas as pd
def SpearmanCorr(xs, ys):
    xranks = pd.Series(xs).rank()
    yranks = pd.Series(ys).rank()
    return Corr(xranks, yranks)
Explanation: The result is a matrix with self-correlations on the diagonal (which are always 1), and cross-correlations on the off-diagonals (which are always symmetric).
Pearson's correlation is not robust in the presence of outliers, and it tends to underestimate the strength of non-linear relationships.
Spearman's correlation is more robust, and it can handle non-linear relationships as long as they are monotonic.  Here's a function that computes Spearman's correlation:
End of explanation
SpearmanCorr(heights, weights)
Explanation: For heights and weights, Spearman's correlation is a little higher:
End of explanation
def SpearmanCorr(xs, ys):
    xs = pd.Series(xs)
    ys = pd.Series(ys)
    return xs.corr(ys, method='spearman')
Explanation: A Pandas Series provides a method that computes correlations, and it offers spearman as one of the options.
End of explanation
SpearmanCorr(heights, weights)
Explanation: The result is the same as for the one we wrote.
End of explanation
Corr(cleaned.htm3, np.log(cleaned.wtkg2))
Explanation: An alternative to Spearman's correlation is to transform one or both of the variables in a way that makes the relationship closer to linear, and the compute Pearson's correlation.
End of explanation
import first
live, firsts, others = first.MakeFrames()
live = live.dropna(subset=['agepreg', 'totalwgt_lb'])
# Solution goes here
mom_age = live.agepreg
weight = live.totalwgt_lb
thinkplot.Scatter(mom_age, weight)
thinkplot.Config(xlabel='mom\'s age', ylabel='birthweight')
# Solution goes here
print('correlation', Corr(mom_age, weight))
print('spearman', SpearmanCorr(mom_age, weight))
# Solution goes here
# Solution goes here
Explanation: Exercises
Using data from the NSFG, make a scatter plot of birth weight versus mother’s age. Plot percentiles of birth weight versus mother’s age. Compute Pearson’s and Spearman’s correlations. How would you characterize the relationship between these variables?
End of explanation
<END_TASK>