<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  Goal
Step2:  The anomalies are the minority.
Remove the time, since it has no meaning for discovering anomalies for us.
Step3:  In unsupervised approaches, the label is not used
Step4:  All the methods we will use, except iForests, performs best if the dataset is scaled
Step5:  K-means clustering
Step6:  The array clusters contains the cluster id of each sample
Step7:  Check how many elements per cluster
Step8:  If you try to compute the silhouette score with ordinary sklearn functions, it is extremely slow.
Recall that you need to compute the distances between all samples, i.e.
Step9:  We will thus used an alternative implementation from Alexandre Abraham.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score
# We resort to a third party library to plot silhouette diagrams
! pip install yellowbrick
from yellowbrick.cluster import SilhouetteVisualizer
Explanation: <a href="https://colab.research.google.com/github/andreaaraldo/BROKEN-PJ/blob/master/anomaly_detection.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
End of explanation
! wget https://datahub.io/machine-learning/creditcard/r/creditcard.csv
df = pd.read_csv('creditcard.csv')
df.head()
df.info(verbose=True)
df['Class'].value_counts()
Explanation: Goal: 
Find fraudolent credit card transactions
Dataset:
* From DataHub
* Anonimized transactions
* Features have no precise meaning: obtained via Principal Component Analysis (PCA)
* Ground truth: transactions labeled as normal/anomaly
End of explanation
df = df.drop('Time', axis=1)
Explanation: The anomalies are the minority.
Remove the time, since it has no meaning for discovering anomalies for us.
End of explanation
X = df.drop('Class', axis=1)
Explanation: In unsupervised approaches, the label is not used
End of explanation
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
Explanation: All the methods we will use, except iForests, performs best if the dataset is scaled
End of explanation
K =3
model = KMeans(n_clusters=K, random_state=3)
clusters = model.fit_predict(X_scaled)
Explanation: K-means clustering
End of explanation
clusters[0:5]
Explanation: The array clusters contains the cluster id of each sample
End of explanation
plt.hist(clusters)
# Inspired by https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html#sphx-glr-auto-examples-cluster-plot-kmeans-silhouette-analysis-py
fig, (ax1) = plt.subplots()
# The silhouette coefficient can range from -1, 1
ax1.set_xlim([-1, 1])
# The (n_clusters+1)*10 is for inserting blank space between silhouette
# plots of individual clusters, to demarcate them clearly.
ax1.set_ylim([0, len(X) + (K + 1) * 10])
Explanation: Check how many elements per cluster
End of explanation
print("Distances to be computed: ", "{:e}".format( X_scaled.shape[0]**2) )
Explanation: If you try to compute the silhouette score with ordinary sklearn functions, it is extremely slow.
Recall that you need to compute the distances between all samples, i.e.
End of explanation
! wget https://gist.githubusercontent.com/AlexandreAbraham/5544803/raw/221aa797cdbfa9e9f75fc0aabb2322dcc11c8991/unsupervised_alt.py
import unsupervised_alt
# The silhouette_score gives the average value for all the samples.
# This gives a perspective into the density and separation of the formed
# clusters
silhouette_avg = silhouette_score(X_scaled, clusters)
# Compute the silhouette scores for each sample
sample_silhouette_values = silhouette_samples(X_scaled, clusters)
Explanation: We will thus used an alternative implementation from Alexandre Abraham.
End of explanation
<END_TASK>