<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Analysis of survey evaluations
This Jupyter notebook examines the evaluations recorded in data/evaluations.csv.
The data
We start by loading the CSV file into a pandas DataFrame and print some information on the size and structure of the dataset.
Step1:  The dataset has 400 samples with 27 columns. Some of these columns are not necessary for further analysis
Step2:  The above two rows exemplify an experimental (top row) and a theoretical (bottom row) paper. Values with a NaN value appear for theoretical papers for all columns that are specific to experimental papers. For the affiliation column, 0 represents academia, 1 represents collaboration and 2 represents industry authors. The remaining columns are boolean, 1 if documented and 0 if not. Note that some experimental papers have no value (NaN) for training and/or validation data if a train/validation/test split is not applicable.
Miscellaneous statistics
Step3:  Extracting experimental papers
Analysis of the reproducibility is relevant for experimental papers, as such we will filter out the experimental papers.
Step4:  $R3(e) = Method(e)$
Step5:  $R2(e) = Method(e) \land Data(e)$
Step6:  $R1(e) = Method(e) \land Data(e) \land Exp(e)$
Step7:  Versions
Here's a generated output to keep track of software versions used to run this Jupyter notebook.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
pd.options.mode.chained_assignment = None  # default='warn', hides SettingWithCopyWarning
file = 'data/evaluations.csv'
conversion_dict = {'research_type': lambda x: int(x == 'E')}
evaluation_data = pd.read_csv(file, sep=',', header=0, index_col=0, converters=conversion_dict)
print('Samples per conference\n{}'.format(evaluation_data.groupby('conference').size()), end='\n')
column_headers = evaluation_data.columns.values
print('\nColumn headers: {}'.format(column_headers))
Explanation: Analysis of survey evaluations
This Jupyter notebook examines the evaluations recorded in data/evaluations.csv.
The data
We start by loading the CSV file into a pandas DataFrame and print some information on the size and structure of the dataset.
End of explanation
evaluation_data.drop(['title', 'authors', 'link', 'comments'], axis=1, inplace=True)
column_headers = evaluation_data.columns.values
evaluation_data.head(2)
Explanation: The dataset has 400 samples with 27 columns. Some of these columns are not necessary for further analysis: title, authors, link, comments. Dropping these leaves us with a numerical index for each paper, the conference it was published to, and survey related data. The lambda function above converts the research_type data from E (experimental) and T (theoretical) to 1 and 0 respectively, making it easier to work with in pandas.
End of explanation
print('Samples per affiliation\n{}'.format(evaluation_data.groupby('affiliation').size()), end='\n\n')
print('Affiliation by conference\n{}'.format(evaluation_data.groupby(['conference', 'affiliation']).size()), end='\n\n')
print('Samples per research type\n{}'.format(evaluation_data.groupby('research_type').size()), end='\n\n')
print('Research type by conference\n{}'.format(evaluation_data.groupby(['conference', 'research_type']).size()), end='\n\n')
print('Samples per research outcome\n{}'.format(evaluation_data.groupby('result_outcome').size()), end='\n\n')
print('Research outcome by conference\n{}'.format(evaluation_data.groupby(['conference', 'result_outcome']).size()), end='\n\n')
print('Samples with contribution\n{}'.format(evaluation_data.groupby('contribution').size()), end='\n\n')
print('Contribution by conference\n{}'.format(evaluation_data.groupby(['conference', 'contribution']).size()), end='\n\n')
Explanation: The above two rows exemplify an experimental (top row) and a theoretical (bottom row) paper. Values with a NaN value appear for theoretical papers for all columns that are specific to experimental papers. For the affiliation column, 0 represents academia, 1 represents collaboration and 2 represents industry authors. The remaining columns are boolean, 1 if documented and 0 if not. Note that some experimental papers have no value (NaN) for training and/or validation data if a train/validation/test split is not applicable.
Miscellaneous statistics
End of explanation
experimental_data = evaluation_data[evaluation_data.research_type == 1]
early_years_index = (experimental_data.conference == 'AAAI 14') | (experimental_data.conference == 'IJCAI 13')
late_years_index = (experimental_data.conference == 'AAAI 16') | (experimental_data.conference == 'IJCAI 16')
Explanation: Extracting experimental papers
Analysis of the reproducibility is relevant for experimental papers, as such we will filter out the experimental papers.
End of explanation
method = ['conference','problem_description','goal/objective','research_method',
        'research_question','pseudocode']
r3_columns = method
experimental_data.loc[:, 'R3'] = experimental_data[r3_columns].all(axis=1)
print('R3(e)\nTotal = {}'.format(experimental_data['R3'].sum()))
display(experimental_data[['R3', 'conference']].groupby('conference').sum())
experimental_data.loc[:, 'R3D'] = experimental_data[r3_columns].mean(axis=1)
print('\n\nR3D\nTotal: {:.4f}, variance = {:.4f}\nBy conference, followed by variance'
      .format(experimental_data['R3D'].mean(), experimental_data['R3D'].var()))
display(experimental_data[['R3D', 'conference']].groupby('conference').mean())
display(experimental_data[['R3D', 'conference']].groupby('conference').var())
print('\n\nYear\tR3D\tVariance\n2013/14\t{:.4f}\t{:.4f}'.format(
    experimental_data[early_years_index].R3D.mean(),
    experimental_data[early_years_index].R3D.var()))
print('2016\t{:.4f}\t{:.4f}'.format(
    experimental_data[late_years_index].R3D.mean(),
    experimental_data[late_years_index].R3D.var()))
Explanation: $R3(e) = Method(e)$
End of explanation
data = ['train', 'validation', 'test', 'results']
r2_columns = r3_columns + data
experimental_data.loc[:, 'Data'] = experimental_data[data].all(axis=1)
print('Data(e)\nTotal = {:}'.format(experimental_data['Data'].sum()))
display(experimental_data[['Data', 'conference']].groupby('conference').sum())
experimental_data.loc[:, 'DataD'] = experimental_data[data].mean(axis=1)
print('\n\nDataDegree(e)\nTotal = {:.4f}, variance = {:.4f}\nBy conference, followed by variance'
      .format(experimental_data['DataD'].mean(), experimental_data['DataD'].var()))
display(experimental_data[['DataD', 'conference']].groupby('conference').mean())
display(experimental_data[['DataD', 'conference']].groupby('conference').var())
print('\n\nYear\tDataD\tVariance\n2013/14\t{:.4f}\t{:.4f}'.format(
    experimental_data[early_years_index].DataD.mean(),
    experimental_data[early_years_index].DataD.var()))
print('2016\t{:.4f}\t{:.4f}'.format(
    experimental_data[late_years_index].DataD.mean(),
    experimental_data[late_years_index].DataD.var()))
experimental_data.loc[:, 'R2'] = experimental_data[r2_columns].all(axis=1)
print('\n\nR2(e)\nTotal = {}'.format(experimental_data['R2'].sum()))
display(experimental_data[['R2', 'conference']].groupby('conference').sum())
experimental_data.loc[:, 'R2D'] = experimental_data[r2_columns].mean(axis=1)
print('\n\nR2D(e)\nTotal = {:.4f}, variance = {:.4f}\nBy conference, followed by variance'
      .format(experimental_data['R2D'].mean(), experimental_data['R2D'].var()))
display(experimental_data[['R2D', 'conference']].groupby('conference').mean())
display(experimental_data[['R2D', 'conference']].groupby('conference').var())
print('\n\nYear\tR2D\tVariance\n2013/14\t{:.4f}\t{:.4f}'.format(
    experimental_data[early_years_index].R2D.mean(),
    experimental_data[early_years_index].R2D.var()))
print('2016\t{:.4f}\t{:.4f}'.format(
    experimental_data[late_years_index].R2D.mean(),
    experimental_data[late_years_index].R2D.var()))
Explanation: $R2(e) = Method(e) \land Data(e)$
End of explanation
experiment = ['hypothesis', 'prediction',
        'open_source_code', 'open_experiment_code',
        'hardware_specification', 'software_dependencies',
        'experiment_setup', 'evaluation_criteria']
r1_columns = r2_columns + experiment
experimental_data.loc[:, 'Exp'] = experimental_data[experiment].all(axis=1)
print('Exp(e)\nTotal = {:.4f}'.format(experimental_data['Exp'].sum()))
display(experimental_data[['Exp', 'conference']].groupby('conference').sum())
experimental_data.loc[:, 'ExpD'] = experimental_data[experiment].mean(axis=1)
print('\n\nExpDegree(e)\nTotal = {:.4f}, variance = {:.4f}\nBy conference, followed by variance'
      .format(experimental_data['ExpD'].mean(), experimental_data['ExpD'].var()))
display(experimental_data[['ExpD', 'conference']].groupby('conference').mean())
display(experimental_data[['ExpD', 'conference']].groupby('conference').var())
print('\n\nYear\tExpD\tVariance\n2013/14\t{:.4f}\t{:.4f}'.format(
    experimental_data[early_years_index].ExpD.mean(),
    experimental_data[early_years_index].ExpD.var()))
print('2016\t{:.4f}\t{:.4f}'.format(
    experimental_data[late_years_index].ExpD.mean(),
    experimental_data[late_years_index].ExpD.var()))
experimental_data.loc[:, 'R1'] = experimental_data[r1_columns].all(axis=1)
print('\n\nR1(e)\nTotal = {:.4f}'.format(experimental_data['R1'].sum()))
display(experimental_data[['R1', 'conference']].groupby('conference').sum())
experimental_data.loc[:, 'R1D'] = experimental_data[r1_columns].mean(axis=1)
print('\n\nR1D(e)\nTotal = {:.4f}, variance = {:.4f}\nBy conference, followed by variance'
      .format(experimental_data['R1D'].mean(), experimental_data['R1D'].var()))
display(experimental_data[['R1D', 'conference']].groupby('conference').mean())
display(experimental_data[['R1D', 'conference']].groupby('conference').var())
print('\n\nYear\tR1D\tVariance\n2013/14\t{:.4f}\t{:.4f}'.format(
    experimental_data[early_years_index].R1D.mean(),
    experimental_data[early_years_index].R1D.var()))
print('2016\t{:.4f}\t{:.4f}'.format(
    experimental_data[late_years_index].R1D.mean(),
    experimental_data[late_years_index].R1D.var()))
Explanation: $R1(e) = Method(e) \land Data(e) \land Exp(e)$
End of explanation
import IPython
import platform
print('Python version: {}'.format(platform.python_version()))
print('IPython version: {}'.format(IPython.__version__))
print('pandas version: {}'.format(pd.__version__))
Explanation: Versions
Here's a generated output to keep track of software versions used to run this Jupyter notebook.
End of explanation
<END_TASK>