<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Metrics vs Predictions
In PhiSpy there are lots of metrics that you can adapt to suit your needs, but how important is each metric in each prediction?
We have a snakefile that will make all possible combinations of metrics and run them for you. 
The parsing code generates two files
Step1:  Read the metrics
Step2:  Read the results file.
Note that we deliberately set the data types here, because pandas was not guessing them correctly.
Step3:  Find the option with no metrics
We need to find the option with --metrics none, --phage_genes 0, and --ignore_annotations. These are coded as none, pg0, and noannotation respectively.
Step4:  But no metrics == no phages!
If we turn everything off, (code 45
Step5:  Find the option with all metrics
Similarly, we need to find the option that includes orf_length_med, shannon_slope, at_skew, gc_skew, and max_direction phmms, but includes neither noannotation nor pg0
Step6:  Plot none and all metrics
These sort of represent "none" and "all". These are the two options from above
Step7:  Plot some selected measures
We have too much data to plot them all, so lets just plot a few selected measures. Chosen almost at random, but not really
Step8:  Plot each metric alone
Note that for each metric, we need none plus the metric, otherwise the results are misleading. Especially for --phmms and --phagegenes where the other metrics are set to the default.
So here we plot each metric alone, together with a version of all metrics.
Step9:  Summary of the metrics
How do the F<sub>1</sub> scores vary across the metrics?
Step10:  Plot the different metrics
Well there are a lot of colours. 
The more metrics we add, the better we get!
Step11:  Find some good, and bad, metrics
Lets see which combinations of metrics are good and bad.
Note that we join the tables here, so we get the definitions of the metrics, not just the numbers.
<ASSISTANT_TASK:>
Python Code:
import os, sys
import itertools
import re
import json
%matplotlib inline
from random import randint
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import gzip
from math import log, e
from scipy import stats
from math import sqrt
Explanation: Metrics vs Predictions
In PhiSpy there are lots of metrics that you can adapt to suit your needs, but how important is each metric in each prediction?
We have a snakefile that will make all possible combinations of metrics and run them for you. 
The parsing code generates two files: the first is a list of a code (just an int) and the combination of metrics. This is just so the other table is not so big. 
The second output is the table of Accuracy, Precision, Recall, Specificity, and F<sub>1</sub> score for each metric.
End of explanation
mdf = pd.read_csv('metric_codes.tsv', delimiter="\t", index_col=0, names=['Code', 'Metric'], header=None)
mdf.head(30)
mdf.tail(30)
Explanation: Read the metrics
End of explanation
dtypes={'Genome': object, 'Metric':np.int64, 'Accuracy':np.float64, 'Precision':np.float64, 'Recall':np.float64, 'Specificity':np.float64, 'F1 score':np.float64}
measures = pd.read_csv("phispy_metrics_tptn.tsv", delimiter="\t", dtype=dtypes, header=None, names=["Genome", "Metric", "Accuracy", "Precision", "Recall", "Specificity", 'F1 score'])
measures.head()
Explanation: Read the results file.
Note that we deliberately set the data types here, because pandas was not guessing them correctly.
End of explanation
mdf[mdf['Metric'].str.contains('none') & mdf['Metric'].str.contains('noannotation') & mdf['Metric'].str.contains('pg0')]
Explanation: Find the option with no metrics
We need to find the option with --metrics none, --phage_genes 0, and --ignore_annotations. These are coded as none, pg0, and noannotation respectively.
End of explanation
measures[(measures['Metric'] == 45)].head()
Explanation: But no metrics == no phages!
If we turn everything off, (code 45: no metrics, phage genes, or hmms) we do not find any phages! (remove the .head() to see more)
End of explanation
mdf[~mdf['Metric'].str.contains('none') & ~mdf['Metric'].str.contains('noannotation') & ~mdf['Metric'].str.contains('pg0') & mdf['Metric'].str.contains('orf_length_med') & mdf['Metric'].str.contains('shannon_slope') & mdf['Metric'].str.contains('phmms')] 
Explanation: Find the option with all metrics
Similarly, we need to find the option that includes orf_length_med, shannon_slope, at_skew, gc_skew, and max_direction phmms, but includes neither noannotation nor pg0
End of explanation
fig, ax = plt.subplots(figsize=(22,16))
m_an = measures[(measures['Metric'] == 45) | (measures['Metric'] == 464)]
sns.violinplot(ax = ax, x="Metric", y="F1 score", data=m_an, scale="count" )
sns.stripplot(ax = ax,  x="Metric", y="F1 score", data=m_an, jitter=True, color="Black")
ax.set_xticklabels(['None', 'All'])
Explanation: Plot none and all metrics
These sort of represent "none" and "all". These are the two options from above
End of explanation
m_some = measures[(measures['Metric'] == 254) | (measures['Metric'] == 0) | (measures['Metric'] == 223)
                  | (measures['Metric'] == 57)
                  | (measures['Metric'] == 464)
                 ]
fig, ax = plt.subplots(figsize=(22,16))
sns.violinplot(ax = ax, x="Metric", y="F1 score", data=m_some, scale="count" )
sns.stripplot(ax = ax,  x="Metric", y="F1 score", data=m_some, jitter=True, color="Black")
ax.set_xticklabels(list(mdf.iloc[m_some['Metric'].unique(),0]), rotation=45)
Explanation: Plot some selected measures
We have too much data to plot them all, so lets just plot a few selected measures. Chosen almost at random, but not really
End of explanation
m_alone = measures[((measures['Metric'] >8) & (measures['Metric'] <17))
                   | (measures['Metric'] == 464)
                 ]
fig, ax = plt.subplots(figsize=(22,16))
labels = list(mdf.iloc[m_alone['Metric'].unique(),0])
labels[-1] = "All"
labels = [i.replace('none ', '') for i in labels]
sns.violinplot(ax = ax, x="Metric", y="F1 score", data=m_alone, scale="count" )
sns.stripplot(ax = ax,  x="Metric", y="F1 score", data=m_alone, jitter=True, color="Black")
ax.set_xticklabels(labels)
fig.savefig('metrics_alone.png')
Explanation: Plot each metric alone
Note that for each metric, we need none plus the metric, otherwise the results are misleading. Especially for --phmms and --phagegenes where the other metrics are set to the default.
So here we plot each metric alone, together with a version of all metrics.
End of explanation
st = measures[['Metric', 'F1 score']].groupby('Metric').describe()
st.head()
Explanation: Summary of the metrics
How do the F<sub>1</sub> scores vary across the metrics?
End of explanation
fig, ax = plt.subplots(figsize=(22,16))
sns.barplot(ax = ax, x=st.index.values, y=('F1 score',  'mean'), data=st)
Explanation: Plot the different metrics
Well there are a lot of colours. 
The more metrics we add, the better we get!
End of explanation
f_good = st[st[('F1 score',  'mean')] > 0.84]
f_good
pd.merge(f_good, mdf, left_index=True, right_index=True)
f_bad = st[st[('F1 score',  'mean')] < 0.5]
f_bad_code = pd.merge(f_bad, mdf, left_index=True, right_index=True)
f_bad_code
max(st[('F1 score',  'mean')])
Explanation: Find some good, and bad, metrics
Lets see which combinations of metrics are good and bad.
Note that we join the tables here, so we get the definitions of the metrics, not just the numbers.
End of explanation
<END_TASK>