<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1>Generate Meta Data Files</h1>
<p>Jay Narhan</p>
May 2017
Parse text files that describe DDSM and MIAS datasets. We then augment and create three meta file
Step1:  <h2>Read DDSM Meta Data
Step2:  <h2>Read in MIAS Meta Data
Step3:  <h3>Create meta_data_all.csv</h3>
This is one file representing meta info on all raw csv files for DDSM and MIAS.
Step4:  
Step5:  <h2>Creating Meta Data for Differencing Use</h2>
Step6:  Important
Step7:  Detection Labels
Step8:  Drop unproven cases as uncertain as to whether an abnormality exists.
Step9:  Diagnosis Labels
Step10:  B&M's (92) and Unproven (16) will need to removed from the differencing analysis when it run, as the pathology cannot be proven in the latter case (in one or both of the breasts), and because handling B&M's through differencing is unclear.
<ASSISTANT_TASK:>
Python Code:
__version__   = '0.1.0'
__status__    = 'Development'
__date__      = '2017-May-25'
__author__    = 'Jay Narhan'
import os
import pandas as pd
import numpy as np
from collections import Counter
META_ROOT = os.path.realpath('../../Meta_Data_Files') + '/'
DDSM_META = META_ROOT + 'Ddsm_png.csv'
MIAS_META = META_ROOT + 'MIAS.txt'
Explanation: <h1>Generate Meta Data Files</h1>
<p>Jay Narhan</p>
May 2017
Parse text files that describe DDSM and MIAS datasets. We then augment and create three meta file: 
* meta_data_all.csv: one file represents meta info on all raw csv files forDDSM and MIAS (in long format), 
* meta_data_detection.csv: meta data for differenced images (goes from long to wide based on patient ID and view - MLO or CC) which can be used in detecting abnormality. 
* meta_data_diagnosis.csv: meta data for differenced images (goes from long to wide based on patient ID and view - MLO or CC) which can be used in detecting pathology. 
meta_data_all.csv will be of the following format:
|Patient_ID | Image_Name | Orientation | View | Lesion_Type | Detection_Res |Pathology_Res|
|-----------|------------|-------------|------|-------------|---------------|-------------|
|-----------|------------|-------------|------|-------------|---------------|-------------|
The other two CSVs will be of the following format:
|Patient_ID | Image_Name | View |  Detection_Res |Pathology_Res|
|-----------|------------|------| ---------------|-------------|
|-----------|------------|------| ---------------|-------------|
Orientation refers to whether the image is of the left or right breast.
View refers to whether the image is Cranio-Caudal (Top looking down) or Medio-Lateral-Oblique()
Detection_Res serves to label with an abnormality exists in the mammogram
Pathology serves as the diagnostic classification objective (labels can include NORMAL, MALIGNANT, or  BENIGN).
End of explanation
ddsm = pd.read_csv(DDSM_META, skiprows=[0,2])
ddsm = ddsm.replace(np.NAN, "None")
ddsm = ddsm.drop(['Type', 'AbType', 'Scanner', 'SubFolder'], axis=1)
ddsm['Pathology'].replace('None', 'NORMAL', inplace=True)
ddsm['Detection_Res'] = np.where( ddsm['Pathology'].str.match(r'NORMAL'), 'NORMAL', 'ABNORMAL' )
ddsm['View']          = np.where( ddsm['Name'].str.contains(r'CC'), 'CC', 'MLO')
ddsm['Patient_ID']    = ddsm['Name'].str.extract(r'^([^.]*)', expand=False)
ddsm['Orientation']   = np.where( ddsm['Name'].str.contains(r'LEFT'), 'LEFT', 'RIGHT')
ddsm  = ddsm.rename(columns= {'Name': 'Image_Name', 'LesionType': 'Lesion_Type', 'Pathology': 'Pathology_Res'})
order = ['Patient_ID', 'Image_Name', 'Orientation', 'View', 'Lesion_Type', 'Detection_Res', 'Pathology_Res']
ddsm = ddsm[order]
print 'Length of DDSM meta information (not necessarily files available): {}'.format(len(ddsm))
ddsm.head(n=8)
pat_ids = [pat_id for pat_id in ddsm['Patient_ID']]
if len(pat_ids) % 4 != 0: # this = number of images and should be divisible by 4!
    print 'Missing DDSM data!'
    print 'Number of patients: {}'.format(len(set(pat_ids)))
    print 'Number of images: {}'.format(len(pat_ids))
    # Missing data:
    counts = Counter(pat_ids)
    for k,v in counts.iteritems():
        if v != 4:
            print 'DDSM Patient: {0} only has {1} images'.format(k, v)
Explanation: <h2>Read DDSM Meta Data:</h2>
End of explanation
with open(MIAS_META) as f:
    content = f.readlines()
mias = pd.DataFrame(columns=order)
mias_patient = 0
for i, row in enumerate(content):
    line = row.split(' ')
    
    img_name = 'mdb' + str(i+1).zfill(3) + '.png' # text file has error in names i.e. line[0] - do not use
    
    lesion =  line[2] 
    if lesion == 'NORM':
        lesion = 'None'
    elif lesion == 'CALC':
        lesion = 'CALCIFICATION'
        
    if line[3] != '\n':
        pathology = line[3]
        if pathology == 'B':
            pathology = 'BENIGN'
        else:
            pathology = 'MALIGNANT'
    else:
        pathology = 'NORMAL'
    
    if pathology == 'NORMAL':
        detection = 'NORMAL'
    else:
        detection = 'ABNORMAL'
        
    patient_id = 'MIAS_' + str(mias_patient)
    
    if i%2 == 0:
        mias.loc[i] = [patient_id, img_name, 'LEFT', 'MLO', lesion, detection, pathology]
    else:
        mias.loc[i] = [patient_id, img_name, 'RIGHT', 'MLO', lesion, detection, pathology]
        mias_patient += 1
mias.head(n=8)
print 'Length of MIAS meta information (not necessarily files): {}'.format(len(mias))
Explanation: <h2>Read in MIAS Meta Data:</h2>
End of explanation
all_data = pd.DataFrame(columns=order)
all_data = all_data.append(ddsm)
all_data = all_data.append(mias)
all_data.to_csv(path_or_buf=META_ROOT +'meta_data_all.csv', index=False)
Explanation: <h3>Create meta_data_all.csv</h3>
This is one file representing meta info on all raw csv files for DDSM and MIAS.
End of explanation
all_data.head()
Explanation: 
End of explanation
print 'Unique Detection_Res values: {}'.format(set(all_data.Detection_Res))
print 'Unique Pathology_Res values: {}'.format(set(all_data.Pathology_Res))
meta_left  = all_data.query('Orientation == "LEFT"')
meta_right = all_data.query('Orientation == "RIGHT"')
meta = meta_left.merge(meta_right, how='inner', on=['Patient_ID', 'View']) # Long to wide on Patients and type of view
del meta['Image_Name_x']
del meta['Image_Name_y']
meta['Image_Name'] = meta.Patient_ID + '_' + meta.View + '.png'
print 'Number of records in meta: {:>10}'.format(meta['Patient_ID'].count())
meta.head()
meta = pd.DataFrame(meta, columns=('Patient_ID', 'Image_Name', 'View', 
                                   'Orientation_x', 'Lesion_Type_x', 'Detection_Res_x', 'Pathology_Res_x', 
                                   'Orientation_y', 'Lesion_Type_y', 'Detection_Res_y', 'Pathology_Res_y'))
meta.head()
malignants = np.where( ((meta['Pathology_Res_x'] == 'MALIGNANT') & (meta['Pathology_Res_y'] == 'MALIGNANT')) |
                       ((meta['Pathology_Res_x'] == 'NORMAL') & (meta['Pathology_Res_y'] == 'MALIGNANT')) |
                       ((meta['Pathology_Res_x'] == 'MALIGNANT') & (meta['Pathology_Res_y'] == 'NORMAL')) )
benigns = np.where( ((meta['Pathology_Res_x'] == 'BENIGN') & (meta['Pathology_Res_y'] == 'BENIGN')) |
                    ((meta['Pathology_Res_x'] == 'NORMAL') & (meta['Pathology_Res_y'] == 'BENIGN')) |
                    ((meta['Pathology_Res_x'] == 'BENIGN') & (meta['Pathology_Res_y'] == 'NORMAL')) |
                    ((meta['Pathology_Res_x'] == 'BENIGN_WITHOUT_CALLBACK') & (meta['Pathology_Res_y'] == 'BENIGN_WITHOUT_CALLBACK')) |
                    ((meta['Pathology_Res_x'] == 'BENIGN_WITHOUT_CALLBACK') & (meta['Pathology_Res_y'] == 'NORMAL')) |
                    ((meta['Pathology_Res_x'] == 'NORMAL') & (meta['Pathology_Res_y'] == 'BENIGN_WITHOUT_CALLBACK')) |
                    ((meta['Pathology_Res_x'] == 'BENIGN') & (meta['Pathology_Res_y'] == 'BENIGN_WITHOUT_CALLBACK')) |
                    ((meta['Pathology_Res_x'] == 'BENIGN_WITHOUT_CALLBACK') & (meta['Pathology_Res_y'] == 'BENIGN')) )
            
both = np.where( ((meta['Pathology_Res_x'] == 'BENIGN') & (meta['Pathology_Res_y'] == 'MALIGNANT')) |
                 ((meta['Pathology_Res_x'] == 'MALIGNANT') & (meta['Pathology_Res_y'] == 'BENIGN')) |
                 ((meta['Pathology_Res_x'] == 'BENIGN_WITHOUT_CALLBACK') & (meta['Pathology_Res_y'] == 'MALIGNANT')) |
                 ((meta['Pathology_Res_x'] == 'MALIGNANT') & (meta['Pathology_Res_y'] == 'BENIGN_WITHOUT_CALLBACK')) )
                
normals = np.where( (meta['Pathology_Res_x'] == 'NORMAL') & (meta['Pathology_Res_y'] == 'NORMAL') )
unproven = np.where( (meta['Pathology_Res_x'] == 'UNPROVEN') | (meta['Pathology_Res_y'] == 'UNPROVEN') )
def add_diagnosis(row_indxs, df, label):
    for item in row_indxs:
        df.loc[item, 'Pathology_Res'] = label
    # Pass by reference, no return needed
add_diagnosis(malignants, meta, 'MALIGNANT')
add_diagnosis(benigns, meta, 'BENIGN')
add_diagnosis(both, meta, 'BENIGN+MALIGNANT')
add_diagnosis(normals, meta, 'NORMAL')
add_diagnosis(unproven, meta, 'UNPROVEN')
meta[9:15]
meta['Pathology_Res'].value_counts()
meta['Detection_Res'] = np.where(meta['Pathology_Res']=='NORMAL', 'NORMAL', 'ABNORMAL')
meta['Detection_Res'].value_counts()
meta['Detection_Res'] = np.where(meta['Pathology_Res']=='UNPROVEN', 'UNPROVEN', meta['Detection_Res'])
meta['Detection_Res'].value_counts()
meta[9:15]
cols = ['Patient_ID', 'Image_Name', 'View', 'Detection_Res', 'Pathology_Res']
meta = meta[cols]
meta[9:15]
Explanation: <h2>Creating Meta Data for Differencing Use</h2>
End of explanation
mask = []
for f in meta.Image_Name:
    mask.append(os.path.isfile('/Users/jnarhan/Projects/CUNY_698/Docker-Shared/Data_Diff_Images/ALL_IMGS/' + f))
    
meta = meta[mask]
print 'Number of records in meta: {:>10}'.format(meta['Patient_ID'].count())
Explanation: Important: Only keeping meta data on differenced images that we have.
End of explanation
print 'Number of Abnormals: {:>4}'.format(sum((meta.Detection_Res == 'ABNORMAL')))
print 'Number of Normals: {:>6}'.format(sum((meta.Detection_Res == 'NORMAL')))
print 'Number of Unproven: {:>5}'.format(sum((meta.Detection_Res == 'UNPROVEN')))
print 'Total: {:18}'.format(meta.Detection_Res.count())
Explanation: Detection Labels: Two-Case Abnormal and Normal
End of explanation
mask = meta['Detection_Res'].isin(['ABNORMAL', 'NORMAL'])
detect_meta = meta[mask]
set(detect_meta.Detection_Res)
print 'Number of Abnormals: {:>4}'.format(sum((detect_meta.Detection_Res == 'ABNORMAL')))
print 'Number of Normals: {:>6}'.format(sum((detect_meta.Detection_Res == 'NORMAL')))
print 'Number of Unproven: {:>5}'.format(sum((detect_meta.Pathology_Res == 'UNPROVEN')))
print 'Total: {:18}'.format(detect_meta.Detection_Res.count())
detect_meta.to_csv(path_or_buf=META_ROOT +'meta_data_detection.csv', index=False)
Explanation: Drop unproven cases as uncertain as to whether an abnormality exists.
End of explanation
print 'Number of Normals: {:>6}'.format(sum((meta.Pathology_Res == 'NORMAL')))
print 'Number of B&Ms: {:>9}'.format(sum((meta.Pathology_Res == 'BENIGN+MALIGNANT')))
print 'Number of Bs: {:>11}'.format(sum((meta.Pathology_Res == 'BENIGN')))
print 'Number of Ms: {:>11}'.format(sum((meta.Pathology_Res == 'MALIGNANT')))
print 'Number of Unproven: {:>5}'.format(sum((meta.Pathology_Res == 'UNPROVEN')))
print 'Total: {:18}'.format(meta.Pathology_Res.count())
Explanation: Diagnosis Labels: Two-Case Benign and Malignant
End of explanation
#mask = meta['Detection_Res'].isin(['BENIGN', 'MALIGNANT', 'NORMAL'])
mask = meta['Pathology_Res'].isin(['BENIGN', 'MALIGNANT'])
diagnosis_meta = meta[mask]
set(diagnosis_meta.Pathology_Res)
print 'Number of Benigns: {:>7}'.format(sum((diagnosis_meta.Pathology_Res == 'BENIGN')))
print 'Number of Malignants: {:}'.format(sum((diagnosis_meta.Pathology_Res == 'MALIGNANT')))
print 'Total: {:19}'.format(diagnosis_meta.Pathology_Res.count())
diagnosis_meta.to_csv(path_or_buf=META_ROOT +'meta_data_diagnosis.csv', index=False)
Explanation: B&M's (92) and Unproven (16) will need to removed from the differencing analysis when it run, as the pathology cannot be proven in the latter case (in one or both of the breasts), and because handling B&M's through differencing is unclear.
End of explanation
<END_TASK>