<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Redcap Form Status Table – All-in-One​
Consolidating all REDCap forms and their status into a single table
Import Libraries
Libraries necessary to run
Step1:  Parametrization
Define parameters to be utilized within the notebook -- hint, they should not be equal to anything!
Step2:  Create Special Columns
Column names
Step3:  Create Data Dictionary
Parse through each site + arm + form, and then create an appropriate data dictionary that can be displayed as a table
Step4:  Functions to Style Table
Functions to help with styling, in accompaniment with .apply()
Step5:  Display Table
Using the subjects dictionary, convert the data into a transposed datafram end display the table
Step6:  Main Function
Define sites and arms, create subjects dictionary, and display table
Step7:  Run main
Run the main function and watch it do its magic
<ASSISTANT_TASK:>
Python Code:
import csv
from datetime import datetime
from IPython.display import display, Markdown, Latex, HTML
import json
import math
import pandas as pd
from pathlib import Path
Explanation: Redcap Form Status Table – All-in-One​
Consolidating all REDCap forms and their status into a single table
Import Libraries
Libraries necessary to run
End of explanation
site: str
arm: str
Explanation: Parametrization
Define parameters to be utilized within the notebook -- hint, they should not be equal to anything!
End of explanation
def get_special_columns(file_path):
    f = open(file_path, "r")
    data = json.load(f)
    return data[0]
new_column_names = get_special_columns("/fs/ncanda-share/beta/chris/ncanda-data-integration/scripts/dashboards/reference/redcap_col_names.json")
Explanation: Create Special Columns
Column names
End of explanation
def create_dictionary(sites, arms):
    # Create blank dictionary
    subjects = {}
    form_statuses = {}
    completeness_statuses = {}
    visit_dates = {}
    missingness_statuses = {}
    # Parse through all sites, arms, and forms
    for site in sites:
        p = Path('/fs/ncanda-share/log/make_all_inventories/inventory_by_site/' + str(site))
        for arm_year in p.iterdir():
            if (arm_year.is_dir() and arm_year.stem in arms):
                for form in arm_year.iterdir():
                    dates_df = pd.read_csv("/fs/ncanda-share/log/status_reports/inventory_dates/" + arm_year.stem + ".csv")
                    if (form.is_dir() == False):
                        df = pd.read_csv(form)
                        
                        # Create blank dictionary if not already a part, or simply set the count
                        for index, row in df.iterrows():
                            missing_status: Float # For use in setting the missing status
                            
                            # Define a new row in each dictionary if not already been parsed
                            if row['study_id'] not in subjects:
                                date_row = (dates_df.loc[dates_df['study_id'] == row['study_id']])
                                missingness_statuses[row['study_id']] = {
                                    'year': arm_year.stem
                                }
                                form_statuses[row['study_id']] = {
                                    'year': arm_year.stem
                                }
                                subjects[row['study_id']] = {
                                    'year': arm_year.stem,
                                    'date': (date_row['visit_date'].values[0])
                                }
                                completeness_statuses[row['study_id']] = {
                                    'year': arm_year.stem
                                }
                            
                            # Update all individual fields
                            subjects[row['study_id']][new_column_names[form.stem]] = int(row['non_nan_count'])
                            form_statuses[row['study_id']][new_column_names[form.stem]] = row['status']
                            completeness_statuses[row['study_id']][new_column_names[form.stem]] = row['complete']
                            
                            # Try to get missing status
                            try:
                                if (type(row['missing']) == str):
                                    missing_status = 0
                                try:
                                    missing_status = int(row['missing'])
                                except ValueError:
                                    missing_status = 0
                            except KeyError:
                                missing_status = 0
                            
                            # Resolve nan errors
                            if (missing_status != missing_status):
                                missing_status = 0
                            missingness_statuses[row['study_id']][new_column_names[form.stem]] = missing_status
    return subjects, form_statuses, completeness_statuses, visit_dates, missingness_statuses
Explanation: Create Data Dictionary
Parse through each site + arm + form, and then create an appropriate data dictionary that can be displayed as a table
End of explanation
success_color = "#39b33f"
light_success_color = "#29f705"
neutral_color = "#b8b8b8"
failure_color = "#e03647"
def determine_cell_style(v, completeness, missingness, form_name):
    style_str: str = "border-style: solid; border-width: thin;" # Initial style string
    # All cases regarding status
    if ('PRESENT' in v): # If Present, Color Green
        style_str += " background-color: " + success_color + ";"
    elif ('EMPTY' in v): # If Empty, mark Gray
        style_str += " background-color: " + neutral_color + ";"
    elif ('MISSING' in v and 0 in v): # If Missing with no content, mark light green
        style_str += " background-color: " + light_success_color + ";"
    elif ('MISSING' in v and 0 not in v ): # If Missing with content, mark Red
        style_str += " background-color: " + failure_color + ";"
    elif ('EXCLUDED' in v and 0 not in v and (1 in v or 2 in v) and form_name == "visit_date"): # If visit_date is excluded with 1, light green
        style_str += " background-color: " + light_success_color + ";"
    elif ('EXCLUDED' in v and 0 not in v): # If Excluded and non-zero, mark Red
        style_str += " background-color: " + failure_color + ";"
    elif ('EXCLUDED' in v and 0 in v): # If Excluded and zero, mark light Green
        style_str += " background-color: " + light_success_color + ";"
    
    return style_str
    
    
def style_row(x, form_statuses, completeness_statuses, missingness_statuses):
    # Create array of style as well as dataframe for data
    style = [None] * x.size
    sample_df = pd.DataFrame(data=x)
    
    # Initialize beginning to be clear background
    style[0] = 'background-color: #ffffff;'
    
    # Loop through and change each style
    for i in range(1, x.size):
        # Grab value and form name
        value = sample_df.iloc[i][0]
        form_name = sample_df.iloc[i].name
        # Get status, completeness, and missingness
        form_status = form_statuses[x.name][form_name]
        completeness = completeness_statuses[x.name][form_name]
        missingness = missingness_statuses[x.name][form_name]
        style[i] = determine_cell_style((value, form_status), completeness, missingness, form_name)
    return style
Explanation: Functions to Style Table
Functions to help with styling, in accompaniment with .apply()
End of explanation
def display_table(subjects, form_statuses, completeness_statuses, missingness_statuses):
    # Convert to Pandas Dataframe and sort by date
    final_df = pd.DataFrame(data=subjects).T
    final_df['date'] = pd.to_datetime(final_df['date'], format='%Y-%m %d', errors='ignore')
    final_df = final_df.sort_values(by="date", ascending=False)
    
    # Sort columns
    first_columns = ['year', 'date', 'visit_date']
    column_order = first_columns + [x for x in sorted(final_df.columns.tolist()) if x not in first_columns]
    final_df = final_df[column_order]
    final_df = final_df.rename(columns=new_column_names)
    final_df = final_df.drop(['date'], axis=1)
    final_df = final_df.applymap(lambda x: x[:2] if "_visit_arm_1" in str(x) else x)
    
    # Add Pandas.style
    s3 = final_df.style.apply(lambda x: style_row(x, form_statuses, completeness_statuses, missingness_statuses), axis=1).set_table_styles(
    [dict(selector="th",props=[('max-width', '50px')]),
     dict(selector="th.col_heading",props=[("writing-mode", "vertical-lr")]),
     dict(selector="td", props=[('margin', '3px')])])
    
    # Display Dashboard Heading 
    display(Markdown('## Sample Dashboard'))
    display(Markdown('For ' + site.capitalize() + ' and arm ' + arm))
    
    # Display Cell Coloring Key
    display(Markdown('### Cell Coloring Key'))
    display(HTML('<div style="background-color: ' + success_color + '; float: left;">"PRESENT"</div>'))
    display(HTML('<div style="background-color: ' + neutral_color + '; float: left;">"EMPTY"</div>'))
    display(HTML('<div style="background-color: ' + failure_color + '; float: left;">"MISSING"</div>'))
    display(HTML('<div style="background-color: ' + light_success_color + '; float: left;">"MISSING WITH NO CONTENT"</div>'))
    display(HTML('<div style="background-color: ' + failure_color + '; float: left;">"EXCLUDED WITH CONTENT"</div>'))
    display(HTML('<div style="background-color: ' + light_success_color + '; float: left;">"EXCLUDED WITH NO CONTENT"</div>'))
    display(HTML('<div style="background-color: ' + light_success_color + '; float: left;">"EXCLUDED FOR VISIT_DATE WITH 1 OR 2 VALUES"</div>'))
    
    # Display 
    display(Markdown('### Table'))
    display(s3)
Explanation: Display Table
Using the subjects dictionary, convert the data into a transposed datafram end display the table
End of explanation
def main():
    sites = [site]
    arms = [arm]
    subjects, form_statuses, completeness_statuses, visit_dates, missingness_statuses = create_dictionary(sites, arms)
    display_table(subjects, form_statuses, completeness_statuses, missingness_statuses)
Explanation: Main Function
Define sites and arms, create subjects dictionary, and display table
End of explanation
main()
Explanation: Run main
Run the main function and watch it do its magic
End of explanation
<END_TASK>