<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Check the effect of the bugs in the code
Load the libraries
Step1:  Load the data
Step2:  Check the shape of the data
Step3:  Compute the difference between the LR
Step4:  Explore the differences
Plot of the difference in LR with respect to the new LR
Step5:  Plot of the absolute difference in LR with respect to the new LR in log space.
Step6:  Thresholds
Step7:  Plot of the percentage of absolute difference in LR with respect to the new LR with the y axis in log space. Sources always below the threshold are plotted in orange.
Step8:  Plot of the percentage of absolute difference in LR with respect to the new LR in log space. Sources always below the threshold are plotted in orange and the galaxies that are not matched now are plotted as green crosses.
Step9:  The same plot than before but using the new threshol
Step10:  Direct comparison of the LR of the sources that changed. Most of them are close to the threshold change but in 4 cases there are big differences.
Step11:  Number of sources with respect to the thresholds
Step12:  The percentage of changes due to the bug
<ASSISTANT_TASK:>
Python Code:
import numpy as np
from astropy.table import Table, join
import matplotlib.pyplot as plt
%matplotlib inline
Explanation: Check the effect of the bugs in the code
Load the libraries
End of explanation
idp = "idata/main_pdf_v0.8-b"
idp_old = "idata/main_pdf_v0.8"
lnew = Table.read(idp+"/lofar_m5.fits")
lold = Table.read(idp_old+"/lofar_m5.fits")
merged = join(lnew, lold, keys=["Source_Name"], table_names=["new", "old"])
Explanation: Load the data
End of explanation
len(lnew)
len(lold)
Explanation: Check the shape of the data
End of explanation
diff_lr = merged['lr_5_old']-merged['lr_5_new']
len(diff_lr)
Explanation: Compute the difference between the LR
End of explanation
plt.plot(merged['lr_5_new'], diff_lr, marker=",", ls="")
plt.xlabel("LR (new)")
plt.ylabel("LR diff")
Explanation: Explore the differences
Plot of the difference in LR with respect to the new LR
End of explanation
plt.loglog(merged['lr_5_new'], np.abs(diff_lr), marker=",", ls="")
plt.xlabel("log LR (new)")
plt.ylabel("log abs. LR diff")
Explanation: Plot of the absolute difference in LR with respect to the new LR in log space.
End of explanation
told = 0.69787
tnew = 0.71684
Explanation: Thresholds
End of explanation
cond_sel = ((merged['lr_5_old'] >= told) | (merged['lr_5_new'] >= tnew))
cond_nosel = ((merged['lr_5_old'] < told) & (merged['lr_5_new'] < tnew))
plt.semilogy(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="")
plt.semilogy(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="")
plt.xlabel("LR (new)")
plt.ylabel("log percent abs. LR diff")
Explanation: Plot of the percentage of absolute difference in LR with respect to the new LR with the y axis in log space. Sources always below the threshold are plotted in orange.
End of explanation
cond_sel = ((merged['lr_5_old'] >= told) | (merged['lr_5_new'] >= tnew)) & (~np.isnan(merged['category_old']))
cond_nosel = ((merged['lr_5_old'] < told) & (merged['lr_5_new'] < tnew)) & (~np.isnan(merged['category_old']))
cond_change = ((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew)) & (~np.isnan(merged['category_old']))
plt.loglog(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="", label="Selected")
plt.loglog(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="", label="Rejected")
plt.loglog(merged['lr_5_new'][cond_change], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_change], marker="x", ls="", label="Changes")
plt.xlabel("log LR (new)")
plt.ylabel("log percent abs. LR diff")
plt.legend()
Explanation: Plot of the percentage of absolute difference in LR with respect to the new LR in log space. Sources always below the threshold are plotted in orange and the galaxies that are not matched now are plotted as green crosses.
End of explanation
# Use new threshold for old data as well
cond_sel = ((merged['lr_5_old'] >= tnew) | (merged['lr_5_new'] >= tnew))
cond_nosel = ((merged['lr_5_old'] < tnew) & (merged['lr_5_new'] < tnew))
cond_change = ((merged['lr_5_old'] >= tnew) & (merged['lr_5_new'] < tnew))
plt.loglog(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="", label="Selected")
plt.loglog(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="", label="Rejected")
plt.loglog(merged['lr_5_new'][cond_change], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_change], marker="x", ls="", label="Changes")
plt.xlabel("log LR (new)")
plt.ylabel("log percent abs. LR diff")
plt.legend()
Explanation: The same plot than before but using the new threshol
End of explanation
cond_change = ((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew))
plt.subplot(1,2,1)
plt.semilogx(merged['lr_5_new'][cond_change], merged['lr_5_old'][cond_change], marker="x", ls="")
plt.xlabel("log LR (new)")
plt.ylabel("LR (old)")
plt.subplot(1,2,2)
plt.plot(merged['lr_5_new'][cond_change], merged['lr_5_old'][cond_change], marker="x", ls="")
plt.xlabel("LR (new)")
plt.ylabel("LR (old)")
Explanation: Direct comparison of the LR of the sources that changed. Most of them are close to the threshold change but in 4 cases there are big differences.
End of explanation
np.sum((merged['lr_5_old'] >= told) & (merged['lr_5_new'] >= tnew))
np.sum((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew))
np.sum((merged['lr_5_old'] < told) & (merged['lr_5_new'] >= tnew))
np.sum((merged['lr_5_old'] < told) & (merged['lr_5_new'] < tnew))
Explanation: Number of sources with respect to the thresholds
End of explanation
4/91332*100
cond_change = ((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew) & (np.abs(diff_lr)/merged['lr_5_new']*100 > 10))
merged[cond_change]
cond = (np.abs(diff_lr)/merged['lr_5_new']*100 > 10)
merged[cond]
# ['Source_Name','ML_LR_new','AllWISE_new',
#  'objID_new','category_new','W1mag_new','i_new','lr_index_5_new',
#  'lr_dist_5_new','lr_5_new','lr_index_sel_5_new','ML_LR_old','AllWISE_old',
#  'objID_old','category_old','W1mag_old','i_old','lr_index_5_old',
#  'lr_dist_5_old','lr_5_old','lr_index_sel_5_old']
cond = (np.abs(diff_lr)/merged['lr_5_new']*100 > 1)
merged[cond & (merged['lr_5_old'] >= told)]['Source_Name','ML_LR_new','AllWISE_new',
  'objID_new','category_new','W1mag_new','i_new','lr_index_5_new',
  'lr_dist_5_new','lr_5_new','lr_index_sel_5_new','ML_LR_old','AllWISE_old', 
  'objID_old','category_old','W1mag_old','i_old','lr_index_5_old',
  'lr_dist_5_old','lr_5_old','lr_index_sel_5_old']
cond = (np.abs(diff_lr)/merged['lr_5_new']*100 > 1)
merged[cond & (merged['lr_5_old'] >= told) & (merged['lr_index_5_old'] != merged['lr_index_5_new'])]['Source_Name','ML_LR_new','AllWISE_new',
  'objID_new','category_new','W1mag_new','i_new','lr_index_5_new',
  'lr_dist_5_new','lr_5_new','lr_index_sel_5_new','ML_LR_old','AllWISE_old', 
  'objID_old','category_old','W1mag_old','i_old','lr_index_5_old',
  'lr_dist_5_old','lr_5_old','lr_index_sel_5_old']
Explanation: The percentage of changes due to the bug
End of explanation
<END_TASK>