<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Examples and Exercises from Think Stats, 2nd Edition
http
Step1:  Again, I'll load the NSFG pregnancy file and select live births
Step2:  Here's the histogram of birth weights
Step3:  To normalize the disrtibution, we could divide through by the total count
Step4:  The result is a Probability Mass Function (PMF).
Step5:  More directly, we can create a Pmf object.
Step6:  Pmf provides Prob, which looks up a value and returns its probability
Step7:  The bracket operator does the same thing.
Step8:  The Incr method adds to the probability associated with a given values.
Step9:  The Mult method multiplies the probability associated with a value.
Step10:  Total returns the total probability (which is no longer 1, because we changed one of the probabilities).
Step11:  Normalize divides through by the total probability, making it 1 again.
Step12:  Here's the PMF of pregnancy length for live births.
Step13:  Here's what it looks like plotted with Hist, which makes a bar graph.
Step14:  Here's what it looks like plotted with Pmf, which makes a step function.
Step15:  We can use MakeFrames to return DataFrames for all live births, first babies, and others.
Step16:  Here are the distributions of pregnancy length.
Step17:  And here's the code that replicates one of the figures in the chapter.
Step18:  Here's the code that generates a plot of the difference in probability (in percentage points) between first babies and others, for each week of pregnancy (showing only pregnancies considered "full term").
Step19:  Biasing and unbiasing PMFs
Here's the example in the book showing operations we can perform with Pmf objects.
Suppose we have the following distribution of class sizes.
Step20:  This function computes the biased PMF we would get if we surveyed students and asked about the size of the classes they are in.
Step21:  The following graph shows the difference between the actual and observed distributions.
Step22:  The observed mean is substantially higher than the actual.
Step23:  If we were only able to collect the biased sample, we could "unbias" it by applying the inverse operation.
Step24:  We can unbias the biased PMF
Step25:  And plot the two distributions to confirm they are the same.
Step26:  Pandas indexing
Here's an example of a small DataFrame.
Step27:  We can specify column names when we create the DataFrame
Step28:  We can also specify an index that contains labels for the rows.
Step29:  Normal indexing selects columns.
Step30:  We can use the loc attribute to select rows.
Step31:  If you don't want to use the row labels and prefer to access the rows using integer indices, you can use the iloc attribute
Step32:  loc can also take a list of labels.
Step33:  If you provide a slice of labels, DataFrame uses it to select rows.
Step34:  If you provide a slice of integers, DataFrame selects rows by integer index.
Step35:  But notice that one method includes the last elements of the slice and one does not.
In general, I recommend giving labels to the rows and names to the columns, and using them consistently.
Exercises
Exercise
Step36:  Exercise
Step37:  Exercise
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function, division
%matplotlib inline
import numpy as np
import nsfg
import first
import thinkstats2
import thinkplot
Explanation: Examples and Exercises from Think Stats, 2nd Edition
http://thinkstats2.com
Copyright 2016 Allen B. Downey
MIT License: https://opensource.org/licenses/MIT
End of explanation
preg = nsfg.ReadFemPreg()
live = preg[preg.outcome == 1]
Explanation: Again, I'll load the NSFG pregnancy file and select live births:
End of explanation
hist = thinkstats2.Hist(live.birthwgt_lb, label='birthwgt_lb')
thinkplot.Hist(hist)
thinkplot.Config(xlabel='Birth weight (pounds)', ylabel='Count')
Explanation: Here's the histogram of birth weights:
End of explanation
n = hist.Total()
pmf = hist.Copy()
for x, freq in hist.Items():
    hist[x] = freq / n
Explanation: To normalize the disrtibution, we could divide through by the total count:
End of explanation
thinkplot.Hist(pmf)
thinkplot.Config(xlabel='Birth weight (pounds)', ylabel='PMF')
Explanation: The result is a Probability Mass Function (PMF).
End of explanation
pmf = thinkstats2.Pmf([1, 2, 2, 3, 5])
pmf
Explanation: More directly, we can create a Pmf object.
End of explanation
pmf.Prob(2)
Explanation: Pmf provides Prob, which looks up a value and returns its probability:
End of explanation
pmf[2]
Explanation: The bracket operator does the same thing.
End of explanation
pmf.Incr(2, 0.2)
pmf[2]
Explanation: The Incr method adds to the probability associated with a given values.
End of explanation
pmf.Mult(2, 0.5)
pmf[2]
Explanation: The Mult method multiplies the probability associated with a value.
End of explanation
pmf.Total()
Explanation: Total returns the total probability (which is no longer 1, because we changed one of the probabilities).
End of explanation
pmf.Normalize()
pmf.Total()
Explanation: Normalize divides through by the total probability, making it 1 again.
End of explanation
pmf = thinkstats2.Pmf(live.prglngth, label='prglngth')
Explanation: Here's the PMF of pregnancy length for live births.
End of explanation
thinkplot.Hist(pmf)
thinkplot.Config(xlabel='Pregnancy length (weeks)', ylabel='Pmf')
Explanation: Here's what it looks like plotted with Hist, which makes a bar graph.
End of explanation
thinkplot.Pmf(pmf)
thinkplot.Config(xlabel='Pregnancy length (weeks)', ylabel='Pmf')
Explanation: Here's what it looks like plotted with Pmf, which makes a step function.
End of explanation
live, firsts, others = first.MakeFrames()
Explanation: We can use MakeFrames to return DataFrames for all live births, first babies, and others.
End of explanation
first_pmf = thinkstats2.Pmf(firsts.prglngth, label='firsts')
other_pmf = thinkstats2.Pmf(others.prglngth, label='others')
Explanation: Here are the distributions of pregnancy length.
End of explanation
width=0.45
axis = [27, 46, 0, 0.6]
thinkplot.PrePlot(2, cols=2)
thinkplot.Hist(first_pmf, align='right', width=width)
thinkplot.Hist(other_pmf, align='left', width=width)
thinkplot.Config(xlabel='Pregnancy length(weeks)', ylabel='PMF', axis=axis)
thinkplot.PrePlot(2)
thinkplot.SubPlot(2)
thinkplot.Pmfs([first_pmf, other_pmf])
thinkplot.Config(xlabel='Pregnancy length(weeks)', axis=axis)
Explanation: And here's the code that replicates one of the figures in the chapter.
End of explanation
weeks = range(35, 46)
diffs = []
for week in weeks:
    p1 = first_pmf.Prob(week)
    p2 = other_pmf.Prob(week)
    diff = 100 * (p1 - p2)
    diffs.append(diff)
thinkplot.Bar(weeks, diffs)
thinkplot.Config(xlabel='Pregnancy length(weeks)', ylabel='Difference (percentage points)')
Explanation: Here's the code that generates a plot of the difference in probability (in percentage points) between first babies and others, for each week of pregnancy (showing only pregnancies considered "full term").
End of explanation
d = { 7: 8, 12: 8, 17: 14, 22: 4, 
     27: 6, 32: 12, 37: 8, 42: 3, 47: 2 }
pmf = thinkstats2.Pmf(d, label='actual')
Explanation: Biasing and unbiasing PMFs
Here's the example in the book showing operations we can perform with Pmf objects.
Suppose we have the following distribution of class sizes.
End of explanation
def BiasPmf(pmf, label):
    new_pmf = pmf.Copy(label=label)
    for x, p in pmf.Items():
        new_pmf.Mult(x, x)
        
    new_pmf.Normalize()
    return new_pmf
Explanation: This function computes the biased PMF we would get if we surveyed students and asked about the size of the classes they are in.
End of explanation
biased_pmf = BiasPmf(pmf, label='observed')
thinkplot.PrePlot(2)
thinkplot.Pmfs([pmf, biased_pmf])
thinkplot.Config(xlabel='Class size', ylabel='PMF')
Explanation: The following graph shows the difference between the actual and observed distributions.
End of explanation
print('Actual mean', pmf.Mean())
print('Observed mean', biased_pmf.Mean())
Explanation: The observed mean is substantially higher than the actual.
End of explanation
def UnbiasPmf(pmf, label=None):
    new_pmf = pmf.Copy(label=label)
    for x, p in pmf.Items():
        new_pmf[x] *= 1/x
        
    new_pmf.Normalize()
    return new_pmf
Explanation: If we were only able to collect the biased sample, we could "unbias" it by applying the inverse operation.
End of explanation
unbiased = UnbiasPmf(biased_pmf, label='unbiased')
print('Unbiased mean', unbiased.Mean())
Explanation: We can unbias the biased PMF:
End of explanation
thinkplot.PrePlot(2)
thinkplot.Pmfs([pmf, unbiased])
thinkplot.Config(xlabel='Class size', ylabel='PMF')
Explanation: And plot the two distributions to confirm they are the same.
End of explanation
import numpy as np
import pandas
array = np.random.randn(4, 2)
df = pandas.DataFrame(array)
df
Explanation: Pandas indexing
Here's an example of a small DataFrame.
End of explanation
columns = ['A', 'B']
df = pandas.DataFrame(array, columns=columns)
df
Explanation: We can specify column names when we create the DataFrame:
End of explanation
index = ['a', 'b', 'c', 'd']
df = pandas.DataFrame(array, columns=columns, index=index)
df
Explanation: We can also specify an index that contains labels for the rows.
End of explanation
df['A']
Explanation: Normal indexing selects columns.
End of explanation
df.loc['a']
Explanation: We can use the loc attribute to select rows.
End of explanation
df.iloc[0]
Explanation: If you don't want to use the row labels and prefer to access the rows using integer indices, you can use the iloc attribute:
End of explanation
indices = ['a', 'c']
df.loc[indices]
Explanation: loc can also take a list of labels.
End of explanation
df['a':'c']
Explanation: If you provide a slice of labels, DataFrame uses it to select rows.
End of explanation
df[0:2]
Explanation: If you provide a slice of integers, DataFrame selects rows by integer index.
End of explanation
resp = nsfg.ReadFemResp()
# Solution goes here
pmf = thinkstats2.Pmf(resp.numkdhh, label='actual')
# Solution goes here
print('actual')
thinkplot.pmf(pmf)
thinkplot.Config(xlabel='# of children', ylabel='PMF')
# Solution goes here
# biased numbers
biased = BiasPmf(pmf, label='Biased')
# Solution goes here
thinkplot.PrePlot(2)
thinkplot.Pmfs([pmf, biased])
thinkplot.Config(xlabel='number of children', ylabel='pmf')
# Solution goes here
# mean of pmf
pmf.Mean()
# Solution goes here
# biased mean
biased.Mean()
Explanation: But notice that one method includes the last elements of the slice and one does not.
In general, I recommend giving labels to the rows and names to the columns, and using them consistently.
Exercises
Exercise: Something like the class size paradox appears if you survey children and ask how many children are in their family. Families with many children are more likely to appear in your sample, and families with no children have no chance to be in the sample.
Use the NSFG respondent variable numkdhh to construct the actual distribution for the number of children under 18 in the respondents' households.
Now compute the biased distribution we would see if we surveyed the children and asked them how many children under 18 (including themselves) are in their household.
Plot the actual and biased distributions, and compute their means.
End of explanation
live, firsts, others = first.MakeFrames()
preg_map = nsfg.MakePregMap(live)
# Solution goes here
hist = thinkstats2.Hist()
for i,j in preg_map.items():
#     print('i=',i,'j=',j)
    if len(j) >= 2:
        pair = (preg.loc[j[0:2]].prglngth)
        diff = np.diff(pair)[0]
        hist[diff] += 1
# Solution goes here
thinkplot.Hist(hist)
# Solution goes here
Explanation: Exercise: I started this book with the question, "Are first babies more likely to be late?" To address it, I computed the difference in means between groups of babies, but I ignored the possibility that there might be a difference between first babies and others for the same woman.
To address this version of the question, select respondents who have at least live births and compute pairwise differences. Does this formulation of the question yield a different result?
Hint: use nsfg.MakePregMap:
End of explanation
import relay
results = relay.ReadResults()
speeds = relay.GetSpeeds(results)
speeds = relay.BinData(speeds, 3, 12, 100)
pmf = thinkstats2.Pmf(speeds, 'actual speeds')
thinkplot.Pmf(pmf)
thinkplot.Config(xlabel='Speed (mph)', ylabel='PMF')
# Solution goes here
def Observed_Pmf(pmf, speed, label=None):
    mod = pmf.Copy(label=label)
    for i in mod.Values():
#         print('values=',i)
        diff = abs(i -speed)
        mod[i] *= diff
    mod.Normalize()
    return mod
        
newpmf = Observed_Pmf(pmf, 7, label='observed')
thinkplot.Pmf(newpmf)
thinkplot.Config(xlabel='speed',ylabel='pmf')
Explanation: Exercise: In most foot races, everyone starts at the same time. If you are a fast runner, you usually pass a lot of people at the beginning of the race, but after a few miles everyone around you is going at the same speed.
When I ran a long-distance (209 miles) relay race for the first time, I noticed an odd phenomenon: when I overtook another runner, I was usually much faster, and when another runner overtook me, he was usually much faster.
At first I thought that the distribution of speeds might be bimodal; that is, there were many slow runners and many fast runners, but few at my speed.
Then I realized that I was the victim of a bias similar to the effect of class size. The race was unusual in two ways: it used a staggered start, so teams started at different times; also, many teams included runners at different levels of ability.
As a result, runners were spread out along the course with little relationship between speed and location. When I joined the race, the runners near me were (pretty much) a random sample of the runners in the race.
So where does the bias come from? During my time on the course, the chance of overtaking a runner, or being overtaken, is proportional to the difference in our speeds. I am more likely to catch a slow runner, and more likely to be caught by a fast runner. But runners at the same speed are unlikely to see each other.
Write a function called ObservedPmf that takes a Pmf representing the actual distribution of runners’ speeds, and the speed of a running observer, and returns a new Pmf representing the distribution of runners’ speeds as seen by the observer.
To test your function, you can use relay.py, which reads the results from the James Joyce Ramble 10K in Dedham MA and converts the pace of each runner to mph.
Compute the distribution of speeds you would observe if you ran a relay race at 7 mph with this group of runners.
End of explanation
<END_TASK>