<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Chapter Three analysis
This notebook doesn't reproduce everything from scratch. To do that, you would need to go into the code folder for this chapter and recreate models using one of the "reproduce" scripts. My goal here is just to do data analysis on existing models and metadata files, to document some of the figures I quote in the chapter,
Step1:  Lists of publications used
in fiction
Step2:  and in poetry
Step3:  Get total numbers of poetry and fiction
In this section I'm simply counting the numbers of volumes I have available in my HathiTrust samples of poetry and fiction, in the relevant date ranges.
Step4:  Just adding up fiction from different sources
Step5:  Can a quarter-century of fiction predict prestige in the rest of the century?
The code below assesses accuracy on a century of fiction, using four models, each only trained on a quarter-century of the evidence.
Note that instead of using a threshold at 0.5, I use the mean prediction for each model as the threshold for assessing accuracy. I think this is a principled solution because models perceive their ancestors as less likely to be reviewed, and their descendants as more likely to be reviewed.
As it happens, this doesn't actually improve accuracy; you get a marginally higher number if you just use 0.5 as the midpoint for all models. But I'm leaving it in, because I think it's in principle the right way to address the diachronic dilemma here. (We could of course also make publication date a variable in the model, but that would tend to obscure an issue I want to foreground.)
Step6:  Can a quarter-century of poetry predict prestige in the rest of the century?
Same drill, just on poetry. We lose more accuracy, but note that most of that is in the first quarter-century, which we know is much less accurate than the others.
Here, it's worth noting, the "midpoint" variable serves us well. Change is more rapid in poetry.
Step7:  Calculating effect size for diachronic drift relative to synchronic standards
In a cell above, I wrote that "change is more rapid in poetry." How much more?
To put this more precisely
Step8:  Interpretation
Step9:  Interpretation
Step10:  Interactions with gender and nationality
First, we might want to know simply how many men or women there are in these datasets. Technically we'll be counting volumes rather than authors.
Step11:  Gender and reviewed status
Step12:  36% of reviewed volumes are by women but 42% of random volumes.
Step13:  26% of reviewed volumes are by women, but only 24% of random volumes. Although women are deeply underrepresented in the poetry dataset as a whole, their distribution across the "reviewed" boundary is favorable.
Nationality and reviewed status
Step14:  The US is 65% of random volumes, but 43% of reviewed volumes in poetry.
Step15:  Variance shared between balanced models and the original models
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
poetry = pd.read_csv('metadata/prestigepoemeta.csv')
fiction = pd.read_csv('metadata/prestigeficmeta.csv')
print(poetry.shape, fiction.shape)
Explanation: Chapter Three analysis
This notebook doesn't reproduce everything from scratch. To do that, you would need to go into the code folder for this chapter and recreate models using one of the "reproduce" scripts. My goal here is just to do data analysis on existing models and metadata files, to document some of the figures I quote in the chapter,
End of explanation
def forcefloat(astring):
    try:
        fval = float(astring)
    except:
        fval = 0
    return fval
fiction['yrrev'] = fiction['yrrev'].apply(forcefloat)
grouped = fiction.loc[ :, ['earliestdate', 'yrrev', 'pubname']].groupby('pubname')
venues = grouped.aggregate(['min', 'max', 'count'])
venues
Explanation: Lists of publications used
in fiction
End of explanation
poetry['yrrev'] = poetry['yrrev'].apply(forcefloat)
grouped = poetry.loc[ :, ['firstpub', 'yrrev', 'pubname']].groupby('pubname')
poevenues = grouped.aggregate(['min', 'max', 'count'])
poevenues
Explanation: and in poetry
End of explanation
def forceint(astring):
    try:
        intval = int(astring)
    except:
        intval = float('nan')
    return intval
prefic = pd.read_csv('/Users/tunder/work/genre/metadata/ficmeta.csv', encoding = 'latin-1', low_memory = False)
prefic['startdate'] = prefic.startdate.apply(forceint)
prefic.fillna(0, inplace = True)
prepoetry = pd.read_csv('/Users/tunder/work/genre/metadata/poemeta.csv')
prepoetry['startdate'] = prepoetry.startdate.apply(forceint)
prepoetry.fillna(0, inplace = True)
numficafter1850 = sum(prefic.startdate > 1849)
numpoeafter1820 = sum(prepoetry.startdate > 1819)
print(str(numficafter1850) + " fiction")
print(str(numpoeafter1820) + " poetry")
del(prefic)
del(prepoetry)
def forceintup(astring):
    try:
        intval = int(astring)
    except:
        intval = 3000
    return intval
postfic = pd.read_csv('/Users/tunder/Dropbox/python/train20/subfiction/filteredfiction.csv', dtype = {'metadatasuspicious': object})
postfic['inferreddate'] = postfic.inferreddate.apply(forceintup)
postfic.fillna(3000, inplace = True)
numficbefore1950 = sum(postfic.inferreddate < 1949)
print(str(numficbefore1950) + " fiction")
Explanation: Get total numbers of poetry and fiction
In this section I'm simply counting the numbers of volumes I have available in my HathiTrust samples of poetry and fiction, in the relevant date ranges.
End of explanation
89077 + 23549
Explanation: Just adding up fiction from different sources:
End of explanation
accuracies = []
for i in range (1850, 1950, 25):
    inpath = 'results/segment' + str(i) + '.applied.csv'
    res = pd.read_csv(inpath)
    colname = 'segment' + str(i)
    midpoint = np.mean(res[colname])
    
    right = 0
    wrong = 0
    
    for idx in res.index:
        sample = res.loc[idx, 'tags']
        logistic = res.loc[idx, colname]
        if sample == 'vulgar':
            if logistic <= midpoint:
                right += 1
            else:
                wrong += 1
        elif sample == 'elite':
            if logistic >= midpoint:
                right += 1
            else:
                wrong += 1
    
    accuracy = right/(right + wrong)
    print(accuracy)
    accuracies.append(accuracy)
    
print()
print(sum(accuracies) / len(accuracies))
        
Explanation: Can a quarter-century of fiction predict prestige in the rest of the century?
The code below assesses accuracy on a century of fiction, using four models, each only trained on a quarter-century of the evidence.
Note that instead of using a threshold at 0.5, I use the mean prediction for each model as the threshold for assessing accuracy. I think this is a principled solution because models perceive their ancestors as less likely to be reviewed, and their descendants as more likely to be reviewed.
As it happens, this doesn't actually improve accuracy; you get a marginally higher number if you just use 0.5 as the midpoint for all models. But I'm leaving it in, because I think it's in principle the right way to address the diachronic dilemma here. (We could of course also make publication date a variable in the model, but that would tend to obscure an issue I want to foreground.)
End of explanation
accuracies = []
for i in range (1820, 1920, 25):
    inpath = 'results/poe_quarter_' + str(i) + '.applied.csv'
    res = pd.read_csv(inpath)
    colname = 'poe_quarter_' + str(i)
    midpoint = np.mean(res[colname])
    
    right = 0
    wrong = 0
    
    for idx in res.index:
        sample = res.loc[idx, 'tags']
        logistic = res.loc[idx, colname]
        if sample == 'random':
            if logistic <= midpoint:
                right += 1
            else:
                wrong += 1
        elif sample == 'reviewed':
            if logistic >= midpoint:
                right += 1
            else:
                wrong += 1
    
    accuracy = right/(right + wrong)
    print(accuracy)
    accuracies.append(accuracy)
    
print()
print(sum(accuracies) / len(accuracies))
        
Explanation: Can a quarter-century of poetry predict prestige in the rest of the century?
Same drill, just on poetry. We lose more accuracy, but note that most of that is in the first quarter-century, which we know is much less accurate than the others.
Here, it's worth noting, the "midpoint" variable serves us well. Change is more rapid in poetry.
End of explanation
# One way to think about this is r-squared.
# In that sense, it's a small factor, but
# definitely significant.
from scipy.stats import pearsonr
fpoe = pd.read_csv('results/fullpoetry.results.csv')
r, p = pearsonr(fpoe.logistic, fpoe.dateused)
print(r, r**2, p)
# A more intelligible measure of effect size is to
# ask how much the midpoint of the probabilistic
# prediction moves in a year.
import statsmodels.formula.api as smf
# create a fitted model in one line
lm = smf.ols(formula='logistic ~ dateused', data=fpoe).fit()
# print the coefficients
lm.params
Explanation: Calculating effect size for diachronic drift relative to synchronic standards
In a cell above, I wrote that "change is more rapid in poetry." How much more?
To put this more precisely: when you establish a synchronic axis of distinction, how rapidly does the midpoint drift as you move forward in time?
End of explanation
ffic = pd.read_csv('results/fullfiction.results.csv')
lm = smf.ols(formula='logistic ~ dateused', data=ffic).fit()
# print the coefficients
lm.params
Explanation: Interpretation: multiplied by 10, I see that as a change of 2% per decade in poetry. The underlying unit is a probabilistic scale of 0% chance of fiction to 100% chance.
End of explanation
r, p = pearsonr(ffic.logistic, ffic.dateused)
print(r, r**2, p)
Explanation: Interpretation: multiplied by 10, I see that as a change of 1% per decade in fiction.
End of explanation
fiction = pd.read_csv('metadata/prestigeficmeta.csv')
fiction['gender'] = fiction.gender.str.strip()
grouped = fiction.loc[ :, ['tags', 'earliestdate', 'gender']].groupby(['gender'])
ficbygender = grouped.aggregate(['count'])
ficbygender
poetry = pd.read_csv('metadata/prestigepoemeta.csv')
poetry['gender'] = poetry.gender.str.strip()
grouped = poetry.loc[ :, ['tags', 'earliestdate', 'gender']].groupby(['gender'])
poebygender = grouped.aggregate(['count'])
poebygender
Explanation: Interactions with gender and nationality
First, we might want to know simply how many men or women there are in these datasets. Technically we'll be counting volumes rather than authors.
End of explanation
fiction = pd.read_csv('metadata/prestigeficmeta.csv')
fiction['gender'] = fiction.gender.str.strip()
grouped = fiction.loc[ :, ['tags', 'earliestdate', 'gender']].groupby(['tags', 'gender'])
ficbygender = grouped.aggregate(['count'])
ficbygender
Explanation: Gender and reviewed status
End of explanation
poetry = pd.read_csv('metadata/prestigepoemeta.csv')
poetry['gender'] = poetry.gender.str.strip()
grouped = poetry.loc[ :, ['tags', 'earliestdate', 'gender']].groupby(['tags', 'gender'])
poebygender = grouped.aggregate(['count'])
poebygender
Explanation: 36% of reviewed volumes are by women but 42% of random volumes.
End of explanation
poetry['nationality'] = poetry.nationality.str.strip()
grouped = poetry.loc[ :, ['tags', 'earliestdate', 'nationality']].groupby(['tags', 'nationality'])
poebynation = grouped.aggregate(['count'])
poebynation
Explanation: 26% of reviewed volumes are by women, but only 24% of random volumes. Although women are deeply underrepresented in the poetry dataset as a whole, their distribution across the "reviewed" boundary is favorable.
Nationality and reviewed status
End of explanation
fiction = pd.read_csv('metadata/prestigeficmeta.csv')
fiction['nationality'] = fiction.nationality.str.strip()
grouped = fiction.loc[ :, ['tags', 'earliestdate', 'nationality']].groupby(['nationality', 'tags'])
ficbynation = grouped.aggregate(['count'])
ficbynation
Explanation: The US is 65% of random volumes, but 43% of reviewed volumes in poetry.
End of explanation
gbf = pd.read_csv('results/gender_balanced_fiction.results.csv', index_col = 'volid')
print(gbf.shape)
gbf['genderbalancedprob'] = gbf.logistic
ff = pd.read_csv('results/fullfiction.results.csv', index_col = 'volid')
ff['originalprob'] = ff.logistic
print(ff.shape)
inboth = gbf.loc[: ,['genderbalancedprob']].join(ff.loc[ :, 'originalprob'], how = 'inner')
print(inboth.shape)
r, p = pearsonr(inboth.genderbalancedprob, inboth.originalprob)
print(r, r**2, p)
nbf = pd.read_csv('results/nation_balanced_fiction.csv', index_col = 'volid')
print(nbf.shape)
nbf['nationbalancedprob'] = nbf.logistic
ff = pd.read_csv('results/fullfiction.results.csv', index_col = 'volid')
ff['originalprob'] = ff.logistic
print(ff.shape)
inboth = nbf.loc[: ,['nationbalancedprob']].join(ff.loc[ :, 'originalprob'], how = 'inner')
print(inboth.shape)
r, p = pearsonr(inboth.nationbalancedprob, inboth.originalprob)
print(r, r**2, p)
gbp = pd.read_csv('results/gender_balanced_poetry.results.csv', index_col = 'volid')
print(gbp.shape)
gbp['genderbalancedprob'] = gbp.logistic
fp = pd.read_csv('results/fullpoetry.results.csv', index_col = 'volid')
fp['originalprob'] = fp.logistic
print(fp.shape)
inboth = gbp.loc[: ,['genderbalancedprob']].join(fp.loc[ :, 'originalprob'], how = 'inner')
print(inboth.shape)
r, p = pearsonr(inboth.genderbalancedprob, inboth.originalprob)
print(r, r**2, p)
nbp = pd.read_csv('results/nation_balanced_poetry.results.csv', index_col = 'volid')
print(nbp.shape)
nbp['nationbalancedprob'] = nbp.logistic
fp = pd.read_csv('results/fullpoetry.results.csv', index_col = 'volid')
fp['originalprob'] = fp.logistic
print(fp.shape)
inboth = nbp.loc[: ,['nationbalancedprob']].join(fp.loc[ :, 'originalprob'], how = 'inner')
print(inboth.shape)
r, p = pearsonr(inboth.nationbalancedprob, inboth.originalprob)
print(r, r**2, p)
fiction.head()
Explanation: Variance shared between balanced models and the original models
End of explanation
<END_TASK>