<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 CSCS530 Winter 2015
Complex Systems 530 - Computer Modeling of Complex Systems (Winter 2015)
Course ID
Step1:  Loading Model Results
First, we need to find the list of all directories in our model output folder from the 001-storing-model-results notebook.  We can do this using the glob and os  modules, which will allow us to work with directories and list their contents.
Step2:  Using os.path.join and os.path.basename
We can also create paths and navigate directory trees using os.path.join.  This method helps build file and directory paths, like we see below.
Step3:  Iterating through model run directories
Next, once we are able to "find" all model run directories, we need to iterate through them and read all data from them.  In the cells, we create data frames for each CSV output file from out 001-storing-model-results notebook.
<ASSISTANT_TASK:>
Python Code:
# Imports
import matplotlib.pyplot as plt
import numpy
import pandas
import scipy
import scipy.stats
Explanation: CSCS530 Winter 2015
Complex Systems 530 - Computer Modeling of Complex Systems (Winter 2015)
Course ID: CMPLXSYS 530
Course Title: Computer Modeling of Complex Systems
Term: Winter 2015
Schedule: Wednesdays and Friday, 1:00-2:30PM ET
Location: 120 West Hall (http://www.lsa.umich.edu/cscs/research/computerlab)
Teachers: Mike Bommarito and Sarah Cherng
View this repository on NBViewer
Reading Model Results
In this notebook, we'll learn a common pattern for reading the results of a model run.  Specifically, we'll:
find all model output directories
read the model output into a new dataset
visualize the results
perform a statistical test
End of explanation
import os
# Using os.listdir to show the current directory
os.listdir("./")
# Using os.listdir to show the output directory
os.listdir("output")[0:5]
import glob
# Using glob to list the output directory
glob.glob("output/run-*")[0:5]
Explanation: Loading Model Results
First, we need to find the list of all directories in our model output folder from the 001-storing-model-results notebook.  We can do this using the glob and os  modules, which will allow us to work with directories and list their contents.
End of explanation
run_directory = os.listdir("output")[0]
print(run_directory)
print(os.path.join(run_directory,
                   "parameters.csv"))
             
print(run_directory)
print(os.path.basename(run_directory))
Explanation: Using os.path.join and os.path.basename
We can also create paths and navigate directory trees using os.path.join.  This method helps build file and directory paths, like we see below.
End of explanation
# Create "complete" data frames
run_data = []
all_timeseries_data = pandas.DataFrame()
all_interaction_data = pandas.DataFrame()
# Iterate over all directories
for run_directory in glob.glob("output/run*"):
    # Get the run ID from our directory name
    run_id = os.path.basename(run_directory)
    
    # Load parameter and reshape
    run_parameter_data = pandas.read_csv(os.path.join(run_directory, "parameters.csv"))
    run_parameter_data.index = run_parameter_data["parameter"]
    
    # Load timeseries and interactions
    run_interaction_data = pandas.read_csv(os.path.join(run_directory, "interactions.csv"))
    run_interaction_data["run"] = run_id
    run_ts_data = pandas.read_csv(os.path.join(run_directory, "timeseries.csv"))
    run_ts_data["run"] = run_id
    
    # Flatten parameter data into interaction and TS data
    for parameter_name in run_parameter_data.index:
        run_ts_data.loc[:, parameter_name] = run_parameter_data.loc[parameter_name, "value"]
        
    if run_interaction_data.shape[0] > 0:
        for parameter_name in run_parameter_data.index:
            run_interaction_data.loc[:, parameter_name] = run_parameter_data.loc[parameter_name, "value"]
    
    # Store raw run data
    run_data.append({"parameters": run_parameter_data,
                    "interactions": run_interaction_data,
                    "timeseries": run_ts_data})
    
    # Update final steps
    all_timeseries_data = all_timeseries_data.append(run_ts_data)
    all_interaction_data = all_interaction_data.append(run_interaction_data)
# let's see how many records we have.
print(all_timeseries_data.shape)
print(all_interaction_data.shape)
# Let's see what the data looks like.
all_timeseries_data.head()
all_interaction_data.head()
%matplotlib inline
# let's use groupby to find some information.
last_step_data = all_timeseries_data.groupby("run").tail(1)
# Simple plot
f = plt.figure()
plt.scatter(last_step_data["min_subsidy"],
           last_step_data["num_infected"],
            alpha=0.5)
plt.xlabel("Subsidy")
plt.ylabel("Number infected")
plt.title("Subsidy vs. number infected")
# Let's use groupby with **multiple** variables now.
mean_infected_by_subsidy = all_timeseries_data.groupby(["run", "min_subsidy", "min_prob_hookup"])["num_infected"].mean()
std_infected_by_subsidy = all_timeseries_data.groupby(["run", "min_subsidy", "min_prob_hookup"])["num_infected"].std()
infected_by_subsidy = pandas.concat((mean_infected_by_subsidy,
                                    std_infected_by_subsidy), axis=1)
infected_by_subsidy.columns = ["mean", "std"]
infected_by_subsidy.head()
# Plot a distribution
f = plt.figure()
_ = plt.hist(last_step_data["num_infected"].values,
         color="red",
         alpha=0.5)
plt.xlabel("Number infected")
plt.ylabel("Frequency")
plt.title("Distribution of number infected")
# Perform distribution tests for no subsidy vs. some subsidy
no_subsidy_data = last_step_data.loc[last_step_data["min_subsidy"] == 0,
                                     "num_infected"]
some_subsidy_data = last_step_data.loc[last_step_data["min_subsidy"] > 0,
                                     "num_infected"]
# Plot a distribution
f = plt.figure()
_ = plt.hist(no_subsidy_data.values,
         color="red",
         alpha=0.25)
_ = plt.hist(some_subsidy_data.values,
            color="blue",
            alpha=0.25)
plt.xlabel("Number infected")
plt.ylabel("Frequency")
plt.title("Distribution of number infected")
# Test for normality
print(scipy.stats.shapiro(no_subsidy_data))
print(scipy.stats.shapiro(some_subsidy_data))
# Test for equal variances
print(scipy.stats.levene(no_subsidy_data, some_subsidy_data))
# Perform t-test
print(scipy.stats.ttest_ind(no_subsidy_data, some_subsidy_data))
# Perform rank-sum test
print(scipy.stats.ranksums(no_subsidy_data, some_subsidy_data))
Explanation: Iterating through model run directories
Next, once we are able to "find" all model run directories, we need to iterate through them and read all data from them.  In the cells, we create data frames for each CSV output file from out 001-storing-model-results notebook.
End of explanation
<END_TASK>