<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Reading in the datasets
We'll read in the test into the variable test and the training dataset into the variable train.
Step1:  Normalizing the Data
We normalize each element x by subtracting the row minimum and dividing by (row maximum minus row minimum).
x = (x - row_min) / (row_max - row_min)
Step2:  K-nearest neighbours
We will try out various different selections of the data and train a separate model for each.
Let's start with the whole data set.
Step3:  Let's score on the training data set to see whether our model is reasonably trained.
Step4:  Let's score on the test set
Step5:  K-nearest Neighbours Bagging Ensemble
In the following lines we'll try out the BaggingClassifier from scikit-learn and we will populate it with a K-nearest Neighbours Classifier.
Step6:  Non-normalized Data
Let's try the same things with only a subset of genes and no normalization besides setting all values below 0 to 1.
Step7:  To separate the rubbish genes from the more predictive genes, we are doing independent sample Welch's t-tests for all genes, comparing the value for each gene in the AML samples to that value for the same gene in the ALL samples.
Step8:  The violin plot below shows the distribution of the results of the ttests. The values plotted are 1-pvalue for all the genes.
One can see that most genes actually have pretty good p-values.
Step9:  Now we'll select only the top 3000 genes based on the 1-pvalue scores, the lowest genesin this selection having a value of around 0.8
Step10:  Plotting the Results
Below are plots of the different combinations of data sets and classifiers.
The data sets are non-negative, i.e. negative expression values are set to 1 and only the top 3000 genes were considered, and normalized, i.e. for each expression value x
<ASSISTANT_TASK:>
Python Code:
test = pd.read_csv('data_set_ALL_AML_independent.tsv', sep='\t', header=0, index_col=0)
train = pd.read_csv('data_set_ALL_AML_train.tsv', sep='\t', header=0, index_col=0)
train.drop(train.columns[len(train.columns)-1], axis=1, inplace=True)
#The sample_table contains the labels AML/ ALL 
sample_table = pd.read_csv('table_ALL_AML_samples.txt', sep='\t')
sample_table['Idealized'] = 0
sample_table['Idealized'].loc[sample_table['ALL/AML']=='ALL'] = 1
Explanation: Reading in the datasets
We'll read in the test into the variable test and the training dataset into the variable train.
End of explanation
norm_train = train.iloc[:,1::2].apply(lambda x: (x - np.min(x))/(np.max(x)-np.min(x)), axis=1)
norm_test = test.iloc[:,1::2].apply(lambda x: (x - np.min(x))/(np.max(x)-np.min(x)), axis=1)
norm_test = norm_test.reindex_axis(sorted(norm_test.columns, key=int), axis=1)
# Sorting the columns numerically
norm_train = norm_train.reindex_axis(sorted(list(norm_train.columns.values), key=int), axis=1)
Explanation: Normalizing the Data
We normalize each element x by subtracting the row minimum and dividing by (row maximum minus row minimum).
x = (x - row_min) / (row_max - row_min)
End of explanation
norm_train_transp = norm_train.T
neigh = KNeighborsClassifier()
neigh.fit(norm_train_transp, sample_table.iloc[:38,1:2].values.reshape(38))
Explanation: K-nearest neighbours
We will try out various different selections of the data and train a separate model for each.
Let's start with the whole data set.
End of explanation
neigh.score(norm_train.T, sample_table.iloc[:38,1:2].values.reshape(38))
Explanation: Let's score on the training data set to see whether our model is reasonably trained.
End of explanation
neigh.score(norm_test.T, sample_table.iloc[38:, 1:2].values.reshape(34))
results = results.append(pd.Series({'Classifier':'K-nearest Neighbours', 'Dataset':'Normalized', 
                          'Training_Error':neigh.score(norm_train.T, sample_table.iloc[:38,1:2].values.reshape(38)), 
                          'Testing_Error':neigh.score(norm_test.T, sample_table.iloc[38:, 1:2].values.reshape(34))}),
              ignore_index=True)
Explanation: Let's score on the test set:
End of explanation
bagging_neigh = BaggingClassifier(KNeighborsClassifier(), 
                                 max_samples=0.5, max_features=0.5)
bagging_neigh.fit(norm_train.T, sample_table.iloc[:38,1:2].values.reshape(38))
#
results = results.append(pd.Series({'Classifier':'Bagging Ensemble of K-nearest Neighbours', 'Dataset':'Normalized', 
                                    'Training_Error':bagging_neigh.score(norm_train.T, sample_table.iloc[:38,1:2].values.reshape(38)), 
                                    'Testing_Error':bagging_neigh.score(norm_test.T, sample_table.iloc[38:, 1:2].values.reshape(34))})
                         ,ignore_index=True)
Explanation: K-nearest Neighbours Bagging Ensemble
In the following lines we'll try out the BaggingClassifier from scikit-learn and we will populate it with a K-nearest Neighbours Classifier.
End of explanation
non_negative_train = train.iloc[:,1::2].clip_lower(1)
non_negative_train 
non_negative_test = test.iloc[:, 1::2].clip_lower(1)
Explanation: Non-normalized Data
Let's try the same things with only a subset of genes and no normalization besides setting all values below 0 to 1.
End of explanation
#The following line computes the independent t-tests between the AML and ALL samples for every gene.
non_negative_train_ttest_pvals = non_negative_train.apply(lambda x: 1-ttest_ind(x.iloc[:26].values.reshape(26,1), x.iloc[26:].values.reshape(12,1), equal_var=False).pvalue , axis=1).iloc[:,:1]
Explanation: To separate the rubbish genes from the more predictive genes, we are doing independent sample Welch's t-tests for all genes, comparing the value for each gene in the AML samples to that value for the same gene in the ALL samples.
End of explanation
sns.violinplot( y="1", data=non_negative_train_ttest_pvals);
Explanation: The violin plot below shows the distribution of the results of the ttests. The values plotted are 1-pvalue for all the genes.
One can see that most genes actually have pretty good p-values.
End of explanation
top3000_ttest_mask = non_negative_train_ttest_pvals.sort_values('1', axis=0, ascending=False).head(3000).index
top3000_non_negative_train = non_negative_train.loc[top3000_ttest_mask]
top3000_non_negative_test = non_negative_test.loc[top3000_ttest_mask]
topneigh = KNeighborsClassifier()
topneigh.fit(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38))
print('Training Prediction Performance: ', 
      topneigh.score(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38)),
      ' Testing Prediction Performance: ',
      topneigh.score(top3000_non_negative_test.T, sample_table.iloc[38:, 1:2].values.reshape(34)))
results = results.append(pd.Series({'Classifier':'K-nearest Neighbours', 'Dataset':'Non-negative', 
                          'Training_Error':topneigh.score(top3000_non_negative_train.T, sample_table.iloc[:38, 1:2].values.reshape(38)), 
                          'Testing_Error' :topneigh.score(top3000_non_negative_test.T,  sample_table.iloc[38:, 1:2].values.reshape(34))}),
                        ignore_index=True)
top_bagging_neigh = BaggingClassifier(KNeighborsClassifier(), 
                                 max_samples=0.5, max_features=0.5)
top_bagging_neigh.fit(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38))
print('Training Prediction Performance: ', 
      top_bagging_neigh.score(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38)),
      ' Testing Prediction Performance: ',
      top_bagging_neigh.score(top3000_non_negative_test.T, sample_table.iloc[38:, 1:2].values.reshape(34)))
results = results.append(pd.Series({'Classifier':'Bagging Ensemble of K-nearest Neighbours', 'Dataset':'Non-negative', 
                          'Training_Error':top_bagging_neigh.score(top3000_non_negative_train.T, sample_table.iloc[:38, 1:2].values.reshape(38)), 
                          'Testing_Error' :top_bagging_neigh.score(top3000_non_negative_test.T,  sample_table.iloc[38:, 1:2].values.reshape(34))}),
                        ignore_index=True)
top_bagging_tree = BaggingClassifier(max_samples=0.5, max_features=0.5)
top_bagging_tree.fit(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38))
print('Training Prediction Performance: ', 
      top_bagging_tree.score(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38)),
      ' Testing Prediction Performance: ',
      top_bagging_tree.score(top3000_non_negative_test.T, sample_table.iloc[38:, 1:2].values.reshape(34)))
results = results.append(pd.Series({'Classifier':'Bagging Ensemble of Decision Trees', 'Dataset':'Non-negative', 
                          'Training_Error': top_bagging_tree.score(top3000_non_negative_train.T, sample_table.iloc[:38,1:2].values.reshape(38)), 
                          'Testing_Error' :top_bagging_tree.score(top3000_non_negative_test.T, sample_table.iloc[38:, 1:2].values.reshape(34))}),
                        ignore_index=True)
 results = results.iloc[1:,:]
results['Training_Error'] = 1- results['Training_Error']
results['Testing_Error']  = 1-results['Testing_Error']
results
Explanation: Now we'll select only the top 3000 genes based on the 1-pvalue scores, the lowest genesin this selection having a value of around 0.8
End of explanation
ax = sns.factorplot(x='Dataset', y='Training_Error', kind='bar', hue='Classifier', data=results)
ax = sns.factorplot(x='Dataset', y='Testing_Error', kind='bar', hue='Classifier', data=results)
Explanation: Plotting the Results
Below are plots of the different combinations of data sets and classifiers.
The data sets are non-negative, i.e. negative expression values are set to 1 and only the top 3000 genes were considered, and normalized, i.e. for each expression value x:
x --> (x - row_min) / (row_max - row_min)
End of explanation
<END_TASK>