<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data Cleaning Challenge!
Goals of this lesson
Students will test their understanding of
Step1:  Open and visually inspect the data
Read the data into Python. The file is called "Study2_Subjects.csv". Check out the separate "README_Study2.txt" file for information on the study and variables.
Step2:  View the first 5 rows of the data frame
Step3:  Examine the data for issues. Some things to look for
Step4:  Correcting data issues
Remember to check the README for additional information on the data that may help you decide what needs to be fixed.
The Condition variable has a missing data point for sub 9
Step5:  The variables Q1, Q2 and Q3 don't have useful names, update them to something useful
Step6:  The sex variable is coded in a non-intuitive way, change it to male/female
Step7:  Create new Variables
Each subject in this study filled out three related questionnaires, but these scores can be combined into a composite score but taking the sum. Create a new variable for this score
Step8:  Combine two datasets together and reshape
You see there's another .csv file for Study 2, this one's called Study2_Trials.csv. Open it and take a look. This frame has the same 10 subjects, but now it has reaction time data from a task with 30 trials. Create a new data frame that combines this with your subject level information
Step9:  Now that you have a one frame with all the data, put it into long format. Each subject should have one row for each trial.
Step10:  You want to do a separate analysis only on the people who had a high score on the anxiety questionnaire. Create a new data frame (in long format) with just the individuals who got 25 or higher
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
sns.set_context("poster")
sns.set(style="ticks",font="Arial",font_scale=2)
Explanation: Data Cleaning Challenge!
Goals of this lesson
Students will test their understanding of:
How to open data in Python
How to check for missing or problematic data and address issues.
How to filter, rearrange and shape data in preparation for analysis.
Your turn!
Now it's your turn to try opening and cleaning a new data set. There are issues with the raw data, so make sure to inspect the variables and make changes when needed. Below is a rough outline to follow, try to get as far as you can.
NOTE: This is the key for the data challenge, so it has all the code already. If you haven't done the challenge, go to the blank version and try doing it on your own. If you are using this key to check your work, remember that there are many ways to do things in Python, so what you did might not be what we did here, but it can still be right (it might even be better!).
End of explanation
os.getcwd()
os.listdir()
subjects = pd.read_table("Study2_Subjects.csv", sep=",")
Explanation: Open and visually inspect the data
Read the data into Python. The file is called "Study2_Subjects.csv". Check out the separate "README_Study2.txt" file for information on the study and variables.
End of explanation
subjects.head()
Explanation: View the first 5 rows of the data frame:
End of explanation
subjects["Age"].hist();
subjects["Sex"]
subjects["Q1"].hist();
subjects["Q2"].hist();
subjects["Q3"].hist();
subjects["Condition"]
#check for missing values
subjects.isnull()
Explanation: Examine the data for issues. Some things to look for: are the column names logical? Are there missing values? Do the variable types make sense? (ie does everything that's a factor make sense as a factor, and does everything that's numeric make sense as numeric?) Are there values that don't make sense? Fix any issues you find:
End of explanation
subjects.loc[subjects["ID"]==9, "Condition"] = "C"
Explanation: Correcting data issues
Remember to check the README for additional information on the data that may help you decide what needs to be fixed.
The Condition variable has a missing data point for sub 9:
End of explanation
subjects = subjects.rename({'Q1':'Anxiety',
                            'Q2':'Depression',
                            'Q3':'Stress'}, axis="columns")
Explanation: The variables Q1, Q2 and Q3 don't have useful names, update them to something useful:
End of explanation
subjects["Sex"] = subjects["Sex"].replace({1:"Male",2:"Female"})
Explanation: The sex variable is coded in a non-intuitive way, change it to male/female:
End of explanation
subjects["Composite"] = subjects[["Anxiety","Depression","Stress"]].sum(axis="columns")
Explanation: Create new Variables
Each subject in this study filled out three related questionnaires, but these scores can be combined into a composite score but taking the sum. Create a new variable for this score:
End of explanation
trials = pd.read_table("Study2_Trials.csv", sep=",")
data = subjects.merge(trials,on="ID")
data.head()
Explanation: Combine two datasets together and reshape
You see there's another .csv file for Study 2, this one's called Study2_Trials.csv. Open it and take a look. This frame has the same 10 subjects, but now it has reaction time data from a task with 30 trials. Create a new data frame that combines this with your subject level information:
End of explanation
id_cols, value_cols = [], []
for column in data.columns:
    if 'RT' in column:
        value_cols.append(column)
    else:
        id_cols.append(column)
    
data_long = pd.melt(data,id_vars=id_cols,var_name="Trial",value_vars=value_cols,value_name="RT")
data_long.head()
Explanation: Now that you have a one frame with all the data, put it into long format. Each subject should have one row for each trial.
End of explanation
subset = data_long[data_long["Anxiety"] >= 25]
Explanation: You want to do a separate analysis only on the people who had a high score on the anxiety questionnaire. Create a new data frame (in long format) with just the individuals who got 25 or higher:
End of explanation
<END_TASK>