<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data
Step1:  Correlation analysis
Pearson
Step2:  P-value非常小，而ks statistic数值较大，认为FRI/FRII有一定的可分性。即原假设FRI/FRII的射电光学和光度服从统一分布是错误的。
但是，mag的D值，相对来说较小，说明光学数据上可分性没有luminosity高
Chi
<ASSISTANT_TASK:>
Python Code:
lumo_fr1_typical = lumo[idx2_same] * 10**-22
lumo_fr2_typical = lumo[idx3_same] * 10**-22
mag_fr1_typical = mag_abs[idx2_same]
mag_fr2_typical = mag_abs[idx3_same]
lumo_fr1_like = lumo[idx_fr1] * 10**-22
lumo_fr2_like = lumo[idx_fr2] * 10**-22
mag_fr1_like = mag_abs[idx_fr1]
mag_fr2_like = mag_abs[idx_fr2]
mag_fr1 = np.hstack([mag_abs[idx_fr1], mag_abs[idx2_same]])
mag_fr2 = np.hstack([mag_abs[idx_fr2], mag_abs[idx3_same]])
lumo_fr1 = np.hstack([lumo[idx_fr1], lumo[idx2_same]]) * 10 ** -22
lumo_fr2 = np.hstack([lumo[idx_fr2], lumo[idx3_same]]) * 10 ** -22
Explanation: Data
End of explanation
import scipy.stats.stats as stats 
from sklearn.feature_selection import chi2
# ks test
# https://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.stats.ks_2samp.html#scipy.stats.ks_2sam
lumo_ks_D_t,lumo_ks_p_t = stats.ks_2samp(lumo_fr1_typical,lumo_fr2_typical)
print("KS statistic of lumo: typical %.5f" % lumo_ks_D_t)
print("P-value of lumo: typical %.5e" % lumo_ks_p_t)
mag_ks_D_t,mag_ks_p_t = stats.ks_2samp(mag_fr1_typical,mag_fr2_typical)
print("KS statistic of Mr: typical %.5f" % mag_ks_D_t)
print("P-value of Mr: typical %.5e" % mag_ks_p_t)
# FR like
lumo_ks_D_l,lumo_ks_p_l = stats.ks_2samp(lumo_fr1_like,lumo_fr2_like)
print("KS statistic of lumo: like %.5f" % lumo_ks_D_l)
print("P-value of lumo: like %.5e" % lumo_ks_p_l)
mag_ks_D_l,mag_ks_p_l = stats.ks_2samp(mag_fr1_like,mag_fr2_like)
print("KS statistic of Mr: like %.5f" % mag_ks_D_l)
print("P-value of Mr: like %.5e" % mag_ks_p_l)
# FR
lumo_ks_D,lumo_ks_p = stats.ks_2samp(lumo_fr1,lumo_fr2)
print("KS statistic of lumo: %.5f" % lumo_ks_D)
print("P-value of lumo: %.5e" % lumo_ks_p)
mag_ks_D,mag_ks_p = stats.ks_2samp(mag_fr1,mag_fr2)
print("KS statistic of Mr: %.5f" % mag_ks_D)
print("P-value of Mr: %.5e" % mag_ks_p)
Explanation: Correlation analysis
Pearson: http://blog.csdn.net/hjh00/article/details/48230399
p-value: https://stackoverflow.com/questions/22306341/python-sklearn-how-to-calculate-p-values
Kolmogorov-Smirnov test: https://stackoverflow.com/questions/10884668/two-sample-kolmogorov-smirnov-test-in-python-scipy
Scipy.stats.kstest: https://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.stats.kstest.html
End of explanation
x_lumo = np.hstack((lumo_fr1,lumo_fr2))
x_lumo.shape
x_lumo = np.log10(np.hstack((lumo_fr1,lumo_fr2)))
x_mag = np.hstack((mag_fr1,mag_fr2))
x_lumo_norm = (x_lumo - x_lumo.min()) / (x_lumo.max() - x_lumo.min())
x_mag_norm = (x_mag - x_mag.min()) / (x_mag.max() - x_mag.min())
x = np.vstack([x_lumo_norm,x_mag_norm])
x = x.transpose()
y = np.zeros(len(mag_abs))
y[idx2_same] = 1
y[idx_fr1] = 1
y[idx3_same] = 2
y[idx_fr2] = 2
y = y[np.where(y > 0)]
scores, pvalues = chi2(x, y)
pvalues
from scipy.stats import chisquare
chisquare(x_lumo_norm, y)
np.random.seed(12222222)
x = np.random.normal(0,1,size=(20000,))
y = np.random.normal(0,1,size=(20000,))
stats.ks_2samp(x,y)
Explanation: P-value非常小，而ks statistic数值较大，认为FRI/FRII有一定的可分性。即原假设FRI/FRII的射电光学和光度服从统一分布是错误的。
但是，mag的D值，相对来说较小，说明光学数据上可分性没有luminosity高
Chi
End of explanation
<END_TASK>