<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Creating Feature Sets for HIV and Age
Step1:  Read in HIV Case Control Models
Step2:  Read in Age Models
Here we went and ran this notebook 
Now we need to unpack the data and combine it together
Step3:  Nested Screen for Age Associated Probes
First pass is Hannum et al. dataset 
Next we look in the probes passing stage 1 for probes in the EPIC dataset  
The two tiers is for power, we only need to correct in stage II for the number of probes passing stage I  
I'm using a Benjamini-Hochberg adusted p-value cutoff of 0.01
Step4:  Size Normalize Datasets
Step5:  Form Feature Sets
Step6:  Feature sets from rolling statistics
Step7:  Nominal significance thresholds
Step8:  Nominal significance thresholds
Step9:  Bonferroni adjusted thresholds
Step10:  Combinations
Step11:  Second pass to simplify things...
<ASSISTANT_TASK:>
Python Code:
import os 
if os.getcwd().endswith('Parallel'):
    os.chdir('..')
import NotebookImport
from HIV_Age_Advancement import *
def size_normalize(vec, order, s=10001):
    p = int(np.ceil(s / 2.) - 1)
    rank = lambda v: np.mean(v < v[p])
    vec = vec.ix[order.order().index].dropna()
    rolling_rank = pd.rolling_apply(vec, s, rank, center=True)
    return rolling_rank
Explanation: Creating Feature Sets for HIV and Age
End of explanation
outdir = '/cellar/users/agross/Data/tmp'
p = outdir + '/hiv_consented/'
def read_models(suffix):
    df = pd.concat([pd.read_csv(p + f, index_col=0, header=[0,1]) for 
                    f in os.listdir(p) if f.endswith('{}.csv'.format(suffix))])
    return df
r4 = read_models('hiv_age_cc')
gg = bhCorrection(r4.HIV_LR.p.ix[probe_idx]) < .01
gg.value_counts()
Explanation: Read in HIV Case Control Models
End of explanation
outdir = '/cellar/users/agross/Data/tmp'
tables = ['in_set_s1','in_set_s2','in_set_s3']
res = {}
for tp in tables:
    p = outdir + '/' + tp + '/'
    res[tp] = pd.concat([pd.read_csv(p + f, index_col=0, header=[0,1]) for 
                    f in os.listdir(p) if f.endswith('age_out.csv')])
res = pd.concat(res, axis=1)
p_vals = res.xs('age_LR', axis=1, level=1).xs('p', axis=1, level=1)
p_vals = p_vals.ix[probe_idx].dropna()
len(p_vals)
def disorder_frac(direction, beta, test_set):
    ct = pd.crosstab(beta < .5, direction.ix[test_set]>0)
    return (1.*ct[1][1] + ct[0][0]) / ct.sum().sum()
Explanation: Read in Age Models
Here we went and ran this notebook 
Now we need to unpack the data and combine it together
End of explanation
rr = ti(bhCorrection(p_vals.in_set_s1) < .01)
len(rr)
v = p_vals.in_set_s3.ix[rr]
r2 = ti(bhCorrection(v) < .01)
len(r2)
rr = bhCorrection(p_vals.in_set_s1) < .01
rr = rr + pd.Series(1, index=r2).ix[rr.index].fillna(0)
rr.value_counts()
Explanation: Nested Screen for Age Associated Probes
First pass is Hannum et al. dataset 
Next we look in the probes passing stage 1 for probes in the EPIC dataset  
The two tiers is for power, we only need to correct in stage II for the number of probes passing stage I  
I'm using a Benjamini-Hochberg adusted p-value cutoff of 0.01
End of explanation
mm = df_hiv.ix[:, ti(hiv == 'HIV-')].mean(1)
hiv_rolled = size_normalize(r4.HIV_LR.p, mm)
age_rolled = size_normalize(p_vals.in_set_s1, mm)
Explanation: Size Normalize Datasets
End of explanation
g_age = bhCorrection(p_vals.in_set_s1.ix[probe_idx]) < .01
g_age_2 = pd.Series(True, r2).ix[probe_idx].fillna(False)
g_age = g_age_2
g_hiv = bhCorrection(r4.HIV_LR.p.ix[probe_idx]) < .01
combo = combine(g_age, g_hiv) == 'both'
#Do not import
fisher_exact_test(g_age, g_hiv)
combo.value_counts()
Explanation: Form Feature Sets
End of explanation
g_hiv_r = hiv_rolled.rank(ascending=True).dropna() < g_hiv.sum()
g_age_r = age_rolled.rank(ascending=True).dropna() < g_age.sum()
Explanation: Feature sets from rolling statistics
End of explanation
g_hiv_nom = r4.HIV_LR.p < .05
g_age_nom = p_vals.in_set_s1 < .05
Explanation: Nominal significance thresholds
End of explanation
hiv_nom_r = hiv_rolled.rank(ascending=True).dropna() < g_hiv_nom.sum()
age_nom_r = age_rolled.rank(ascending=True).dropna() < g_age_nom.sum()
Explanation: Nominal significance thresholds: rolled
End of explanation
g_hiv_b = (r4.HIV_LR.ix[probe_idx].p * len(r4)) < .01
g_age_b = (p_vals.in_set_s1.ix[probe_idx] * len(p_vals)) < .01
Explanation: Bonferroni adjusted thresholds
End of explanation
combo = combine(g_age, g_hiv) == 'both'
combo_b = combine(g_hiv_b, g_age_b) == 'both'
combo_r = combine(g_age_r, g_hiv_r) == 'both'
features = {'HIV (BH)': g_hiv & (g_age_nom == False), 
            'HIV (adj)': g_hiv_r & (age_nom_r == False), 
            'HIV (Bonf)': g_hiv_b & (g_age_nom == False), 
            'Age (BH)': g_age_2 & (g_hiv_nom == False), 
            'Age (adj)': g_age_r & (hiv_nom_r == False),
            'Age (Bonf)': g_age_b & (g_hiv_nom == False),
            'HIV + Age (BH)': combo, 
            'HIV + Age (adj)': combo_r,
            'HIV + Age (Bonf)': combo_b}
c1 = pd.DataFrame({i:v.value_counts() for i,v in features.iteritems()}).T
c1
Explanation: Combinations
End of explanation
features = {'HIV (BH)': g_hiv & (g_age_2 == False), 
            'HIV (adj)': g_hiv_r & (age_nom_r == False), 
            'HIV (Bonf)': g_hiv_b & (g_age_b == False), 
            'Age (BH)': g_age_2 & (g_hiv == False), 
            'Age (adj)': g_age_r & (g_hiv == False),
            'Age (Bonf)': g_age_b & (g_hiv_b == False),
            'HIV + Age (BH)': combo, 
            'HIV + Age (adj)': combo_r,
            'HIV + Age (Bonf)': combo_b}
c1 = pd.DataFrame({i:v.value_counts() for i,v in features.iteritems()}).T
c1
Explanation: Second pass to simplify things...
End of explanation
<END_TASK>