<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Sveučilište u Zagrebu<br>
Fakultet elektrotehnike i računarstva
Strojno učenje
<a href="http
Step1:  Sadržaj
Matrica zabune
Osnovne mjere
F-mjera
Višeklasna klasifikacija
Procjena pogreške
Statističko testiranje
Usporedba klasifikatora
Matrica zabune
Step2:  [Skica
Step3:  [Skica
Step4:  Osnovne mjere
[Skica
Step5:  Primjer
Step6:  Variranje klasifikacijskog praga
Step7:  Krivulja preciznost-odziv
Step8:  ROC i AUC
ROC = Receiver Operating Characteristics
TPR kao funkcija od FPR
AUC = Area Under the (ROC) Curve
Step9:  F-mjera
F1-mjera
Step10:  Višeklasna klasifikacija
<ASSISTANT_TASK:>
Python Code:
# Učitaj osnovne biblioteke...
import scipy as sp
import sklearn
import pandas as pd
%pylab inline
Explanation: Sveučilište u Zagrebu<br>
Fakultet elektrotehnike i računarstva
Strojno učenje
<a href="http://www.fer.unizg.hr/predmet/su">http://www.fer.unizg.hr/predmet/su</a>
Ak. god. 2015./2016.
Bilježnica 10: Vrednovanje modela
(c) 2015 Jan Šnajder
<i>Verzija: 0.1 (2015-12-19)</i>
End of explanation
y_test = sp.random.choice((0,1), size=10); y_test
y_pred = sp.random.choice((0,1), size=10); y_pred
Explanation: Sadržaj
Matrica zabune
Osnovne mjere
F-mjera
Višeklasna klasifikacija
Procjena pogreške
Statističko testiranje
Usporedba klasifikatora
Matrica zabune
End of explanation
def cm(y_true, y_pred):
    tp = 0
    fp = 0
    fn = 0
    tn = 0
    for (t, p) in zip(y_true, y_pred):
        if t == 0 and p == 1: fp += 1
        elif t == 1 and p == 0: fn += 1
        elif t == 1 and p == 1: tp += 1
        else: tn += 1
    return sp.array([[tp, fp], [fn, tn]])
        
cm(y_test, y_pred)
from sklearn.metrics import confusion_matrix
Explanation: [Skica: retci -> klasifikacija, stupci -> stvarno]
End of explanation
confusion_matrix(y_test, y_pred)
confusion_matrix(y_test, y_pred, labels=[1,0])
Explanation: [Skica: retci -> stvarno, stupci -> klasifikacija]
End of explanation
cm(y_test, y_pred)
from sklearn.metrics import accuracy_score, precision_score, recall_score
accuracy_score(y_test, y_pred)
precision_score(y_test, y_pred)
recall_score(y_test, y_pred)
Explanation: Osnovne mjere
[Skica: TP-FP-TN-FN]
Točnost (engl. accuracy)
$$
\mathrm{Acc} = \frac{\mathrm{TP}+\mathrm{TN}}{N} = \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}
$$
Preciznost (engl. precision)
$$
\mathrm{P} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}
$$
Odziv (engl. recall), true positive rate, specificity
$$
\mathrm{R} = \mathrm{TPR} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}
$$
Fall-out, false positive rate
$$
\mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}}
$$
Primjer
End of explanation
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import Imputer
titanic_df = pd.read_csv("../data/titanic-train.csv")
titanic_df.drop(['PassengerId'], axis=1, inplace=True)
titanic_df1 = titanic_df[['Pclass', 'Sex', 'Age','Survived']]
titanic_X = titanic_df[['Pclass', 'Sex', 'Age']].as_matrix()
titanic_y = titanic_df['Survived'].as_matrix()
le = LabelEncoder()
titanic_X[:,1] = le.fit_transform(titanic_X[:,1])
imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
titanic_X = imp.fit_transform(titanic_X)
titanic_X
titanic_y
shape(titanic_X), shape(titanic_y)
from sklearn import cross_validation
X_train, X_test, y_train, y_test = cross_validation.train_test_split(titanic_X, titanic_y, train_size=2.0/3, random_state=42)
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(C=1)
lr.fit(X_train, y_train)
lr.predict(X_train)
y_pred_lr = lr.predict(X_test); y_pred_lr
y_test
cm(y_test, y_pred_lr)
accuracy_score(y_test, y_pred_lr)
lr.score(X_test, y_test)
lr.score(X_train, y_train)
precision_score(y_test, y_pred_lr, pos_label=1)
recall_score(y_test, y_pred_lr, pos_label=1)
from sklearn.svm import SVC
svm = SVC(C=1)
svm.fit(X_train, y_train)
svm.score(X_test, y_test)
y_pred_svm = svm.predict(X_test); y_pred_svm
cm(y_test, y_pred_svm)
precision_score(y_test, y_pred_svm, pos_label=1)
recall_score(y_test, y_pred_svm, pos_label=1)
Explanation: Primjer: Titanic dataset
End of explanation
y_scores_lr = lr.predict_proba(X_test)[:,1]; y_scores_lr
print precision_score(y_test, y_pred_lr)
print recall_score(y_test, y_pred_lr)
threshold = 0.4
y_pred_lr_tweaked = map(lambda s : 1 if s > threshold else 0, y_scores_lr)
print y_pred_lr_tweaked
print precision_score(y_test, y_pred_lr_tweaked)
print recall_score(y_test, y_pred_lr_tweaked)
Explanation: Variranje klasifikacijskog praga
End of explanation
from sklearn.metrics import precision_recall_curve
pr, re, _ = precision_recall_curve(y_test, y_scores_lr, pos_label=1)
pr
re
plt.plot(re, pr)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.show()
from sklearn.metrics import average_precision_score
average_precision_score(y_test, y_scores_lr)
y_scores_svm = svm.decision_function(X_test)[:,0]
print y_scores_svm
pr_lr, re_lr, _ = precision_recall_curve(y_test, y_scores_lr, pos_label=1)
pr_svm, re_svm, _ = precision_recall_curve(y_test, y_scores_svm, pos_label=1)
plt.plot(re_lr, pr_lr, label='LR')
plt.plot(re_svm, pr_svm, label='SVM')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.legend()
plt.show()
print average_precision_score(y_test, y_scores_lr)
print average_precision_score(y_test, y_scores_svm)
Explanation: Krivulja preciznost-odziv
End of explanation
from sklearn.metrics import roc_curve, auc
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_scores_lr)
roc_auc_lr = auc(fpr_lr, tpr_lr)
fpr_svm, tpr_svm, _ = roc_curve(y_test, y_scores_svm)
roc_auc_svm = auc(fpr_svm, tpr_svm)
plt.plot(fpr_lr, tpr_lr, label='LR ROC curve (area = %0.2f)' % roc_auc_lr)
plt.plot(fpr_svm, tpr_svm, label='SVM ROC curve (area = %0.2f)' % roc_auc_svm)
plt.plot([0, 1], [0, 1], 'k--')
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.legend(loc='lower right')
plt.show()
Explanation: ROC i AUC
ROC = Receiver Operating Characteristics
TPR kao funkcija od FPR
AUC = Area Under the (ROC) Curve
End of explanation
def f_beta(p, r, beta):
    return ((1 + beta**2) * p * r) / (beta**2 * p + r)
f_beta(0.5, 0.9, 1)
f_beta(0.5, 0.9, 0.5)
f_beta(0.5, 0.9, 2)
(0.5 + 0.9) / 2
sqrt(0.5 * 0.9)
2/(1/0.5 + 1/0.9)
r = 0.5
xs = sp.linspace(0, 1)
plt.plot(xs, (xs + r)/2, label='aritm')
plt.plot(xs, sp.sqrt(xs*r), label='geom')
plt.plot(xs, 2/(1/xs + 1/r), label='harm')
plt.legend(loc='lower right')
plt.show()
Explanation: F-mjera
F1-mjera:
$$
F = \frac{2}{\frac{1}{P}+\frac{1}{R}} = \frac{2PR}{P+R}
$$
F-beta:
$$
F_\beta = \frac{(1+\beta^2)PR}{\beta^2 P +R}
$$
$F_{0.5}$ dvostruko naglašava preciznost, $F_{2}$ dvostruko naglašava recall
End of explanation
data = sp.loadtxt("path/do/glass.data", delimiter=",", skiprows=1)
print data
shape(data)
glass_X, glass_y = data[:,1:10], data[:,10]
from sklearn import cross_validation
X_train, X_test, y_train, y_test = cross_validation.train_test_split(glass_X, glass_y, train_size=2.0/3, random_state=42)
X_train.shape, X_test.shape
from sklearn.svm import SVC
m = SVC() # SVC(C=1, gamma='auto')
m.fit(X_train, y_train)
m.classes_
y_pred = m.predict(X_test); y_pred
from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, y_pred)
from sklearn.metrics import f1_score
f1_score(y_test, y_pred, pos_label=l, average=None)
sp.mean(_)
f1_score(y_test, y_pred, average='macro')
f1_score(y_test, y_pred, average='micro')
Explanation: Višeklasna klasifikacija
End of explanation
<END_TASK>