<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Import and Validation
Copyright 2015 Allen Downey
License
Step2:  The NSFG data is in a fixed-width format, documented in a Stata dictionary file.  ReadFemPreg reads the dictionary and then reads the data into a Pandas DataFrame.
Step3:  After loading a DataFrame, I check the shape.
Step4:  And take a look at the first few rows.
Step5:  Then I validate the variables I am likely to need.  The encoding of agepreg is non-obvious.
Step6:  Which is why you have to read the codebook
Step7:  For live births, birthweight is coded as two integers, birthwgt_lb and birthwgt_oz.  We can use describe to summarize variables.
Step8:  Most of that looks reasonable, but the maximum is 99 lbs!  Let's look at the distribution of values
Step9:  Consulting the code book, we see that 97, 98, and 99 are sentinel values indicating "not ascertained", "refused", and "don't know" (that is, the respondent did not know).
Also, the 51 pound baby is undoubtably an error.  We can replace unrealistic values with NaN.
Step10:  And if we don't care about the different sentinel values, we can replace them all with NaN.
Step11:  Next, it will be convenient to recode birthwgt_lb and birthwgt_oz with a single floating-point value.
Step13:  After testing these cleanings and recodings, we can encapsuate them in a function.  As we work with additional variables, we might add more lines to this function.
Step14:  The NSFG codebook includes summaries for many variables, which we can use to make sure the data is uncorrupted, and we are interpreting it correctly.
Step15:  The distribution of pregordr is consistent with the summary in the codebook.
After running a few checks like this, I document them using assert statements.
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function, division
import numpy as np
import thinkstats2
Explanation: Import and Validation
Copyright 2015 Allen Downey
License: Creative Commons Attribution 4.0 International
End of explanation
def ReadFemPreg(dct_file='2002FemPreg.dct',
                dat_file='2002FemPreg.dat.gz'):
    Reads the NSFG pregnancy data.
    dct_file: string file name
    dat_file: string file name
    returns: DataFrame
    
    dct = thinkstats2.ReadStataDct(dct_file)
    preg = dct.ReadFixedWidth(dat_file, compression='gzip')
    return preg
Explanation: The NSFG data is in a fixed-width format, documented in a Stata dictionary file.  ReadFemPreg reads the dictionary and then reads the data into a Pandas DataFrame.
End of explanation
preg = ReadFemPreg()
print(preg.shape)
Explanation: After loading a DataFrame, I check the shape.
End of explanation
preg.head()
Explanation: And take a look at the first few rows.
End of explanation
preg.agepreg
Explanation: Then I validate the variables I am likely to need.  The encoding of agepreg is non-obvious.
End of explanation
preg.agepreg /= 100
preg.agepreg.mean()
Explanation: Which is why you have to read the codebook:
http://www.icpsr.umich.edu/nsfg6/Controller?displayPage=labelDetails&fileCode=PREG&section=A&subSec=8016&srtLabel=611935
We can convert to a more obvious representation easily enough:
End of explanation
preg.birthwgt_lb.describe()
Explanation: For live births, birthweight is coded as two integers, birthwgt_lb and birthwgt_oz.  We can use describe to summarize variables.
End of explanation
preg.birthwgt_lb.value_counts().sort_index()
Explanation: Most of that looks reasonable, but the maximum is 99 lbs!  Let's look at the distribution of values:
End of explanation
preg.loc[preg.birthwgt_lb > 20, 'birthwgt_lb'] = np.nan
Explanation: Consulting the code book, we see that 97, 98, and 99 are sentinel values indicating "not ascertained", "refused", and "don't know" (that is, the respondent did not know).
Also, the 51 pound baby is undoubtably an error.  We can replace unrealistic values with NaN.
End of explanation
na_vals = [97, 98, 99]
preg.birthwgt_lb.replace(na_vals, np.nan, inplace=True)
preg.birthwgt_oz.replace(na_vals, np.nan, inplace=True)
Explanation: And if we don't care about the different sentinel values, we can replace them all with NaN.
End of explanation
preg['totalwgt_lb'] = preg.birthwgt_lb + preg.birthwgt_oz / 16.0
Explanation: Next, it will be convenient to recode birthwgt_lb and birthwgt_oz with a single floating-point value.
End of explanation
def CleanFemPreg(preg):
    Recodes variables from the pregnancy frame.
    preg: DataFrame
    
    # mother's age is encoded in centiyears; convert to years
    preg.agepreg /= 100.0
    # birthwgt_lb contains at least one bogus value (51 lbs)
    # replace with NaN
    preg.loc[preg.birthwgt_lb > 20, 'birthwgt_lb'] = np.nan
    
    # replace 'not ascertained', 'refused', 'don't know' with NaN
    na_vals = [97, 98, 99]
    preg.birthwgt_lb.replace(na_vals, np.nan, inplace=True)
    preg.birthwgt_oz.replace(na_vals, np.nan, inplace=True)
    preg.hpagelb.replace(na_vals, np.nan, inplace=True)
    preg.babysex.replace([7, 9], np.nan, inplace=True)
    preg.nbrnaliv.replace([9], np.nan, inplace=True)
    # birthweight is stored in two columns, lbs and oz.
    # convert to a single column in lb
    # NOTE: creating a new column requires dictionary syntax,
    # not attribute assignment (like preg.totalwgt_lb)
    preg['totalwgt_lb'] = preg.birthwgt_lb + preg.birthwgt_oz / 16.0    
    # due to a bug in ReadStataDct, the last variable gets clipped;
    # so for now set it to NaN
    preg.cmintvw = np.nan
Explanation: After testing these cleanings and recodings, we can encapsuate them in a function.  As we work with additional variables, we might add more lines to this function.
End of explanation
preg.pregordr.value_counts().sort_index()
Explanation: The NSFG codebook includes summaries for many variables, which we can use to make sure the data is uncorrupted, and we are interpreting it correctly.
End of explanation
assert len(preg) == 13593
assert preg.caseid[13592] == 12571
assert preg.pregordr.value_counts()[1] == 5033
assert preg.nbrnaliv.value_counts()[1] == 8981
assert preg.babysex.value_counts()[1] == 4641
assert preg.birthwgt_lb.value_counts()[7] == 3049
assert preg.birthwgt_oz.value_counts()[0] == 1037
assert preg.prglngth.value_counts()[39] == 4744
assert preg.outcome.value_counts()[1] == 9148
assert preg.birthord.value_counts()[1] == 4413
assert preg.agepreg.value_counts()[22.75] == 100
assert preg.totalwgt_lb.value_counts()[7.5] == 302
weights = preg.finalwgt.value_counts()
key = max(weights.keys())
assert preg.finalwgt.value_counts()[key] == 6
Explanation: The distribution of pregordr is consistent with the summary in the codebook.
After running a few checks like this, I document them using assert statements.
End of explanation
<END_TASK>