<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Preparing the data
After we import our CSV of crime data, we need to do a couple things to get it ready for clustering
Step1:  K-means clustering
Here we'll review the idea of k-means clustering you discussed last week and see how it applies to our crime data. We'll start with three clusters.
Step2:  The data comes out in the format of cluster_id,incident_type,lat,lng. If we save it to a csv file, we can load it into Google's simple map viewer tool to see how it looks.
As you can see, segmenting the data into only three clusters doesn't give us anything useful. Let's try a bigger number.
Step3:  These clusters are arguably more useful, but it's also clear that k-means might not be the best tool for figuring out our density-based crime clusters. Let's try another approach.
DBSCAN
Unlike K-Means clustering, which requires us to define the number of clusters we want in advance, DBSCAN is a density-based clustering algorithm that works by finding points that are close together (given an input parameter, known as epsilon).
As we'll see in class, the intuition of the algorithm is relatively simple to understand. A functioning, documented implementation is here if you want to explore further.
Since we'll be using scikit-learn's implementation, though, we can invoke it in almost exactly the same way that we invoked K-Means. We'll start by establishing a constant for epsilon.
Step4:  DBSCAN requires a pre-processing step that K-Means doesn't
Step5:  Each entry in the matrix shows how far each of our 1,995 points is from each of the other points in the dataset, in the unit of latitude and longitude degrees. The distances between points are key for DBSCAN to compute densities. But this also exposes one of its weaknesses
<ASSISTANT_TASK:>
Python Code:
data = list(csv.DictReader(open('data/columbia_crime.csv', 'r').readlines()))
# This part just splits out the latitude and longitude coordinate fields for each incident, which we need for mapping.
coords = [(float(d['lat']), float(d['lng'])) for d in data if len(d['lat']) > 0]
print coords[:10]
# And this creates a matching array of incident types
types = [d['ExtNatureDisplayName'] for d in data]
print types[:10]
Explanation: Preparing the data
After we import our CSV of crime data, we need to do a couple things to get it ready for clustering: extracting the coordinate pairs that we want to cluster, and pulling together some simple labels so we know which indcident each point refers to.
End of explanation
number_of_clusters = 3
kmeans = KMeans(n_clusters=number_of_clusters)
kmeans.fit(coords)
clusters_to_csv(kmeans.labels_, types, coords)
Explanation: K-means clustering
Here we'll review the idea of k-means clustering you discussed last week and see how it applies to our crime data. We'll start with three clusters.
End of explanation
number_of_clusters = 10
kmeans = KMeans(n_clusters=number_of_clusters)
kmeans.fit(coords)
clusters_to_csv(kmeans.labels_, types, coords)
Explanation: The data comes out in the format of cluster_id,incident_type,lat,lng. If we save it to a csv file, we can load it into Google's simple map viewer tool to see how it looks.
As you can see, segmenting the data into only three clusters doesn't give us anything useful. Let's try a bigger number.
End of explanation
# We're dealing in unprojected coordinates, so this basically refers to a fraction of a degree of lat/lng.
EPS = 0.02
Explanation: These clusters are arguably more useful, but it's also clear that k-means might not be the best tool for figuring out our density-based crime clusters. Let's try another approach.
DBSCAN
Unlike K-Means clustering, which requires us to define the number of clusters we want in advance, DBSCAN is a density-based clustering algorithm that works by finding points that are close together (given an input parameter, known as epsilon).
As we'll see in class, the intuition of the algorithm is relatively simple to understand. A functioning, documented implementation is here if you want to explore further.
Since we'll be using scikit-learn's implementation, though, we can invoke it in almost exactly the same way that we invoked K-Means. We'll start by establishing a constant for epsilon.
End of explanation
distance_matrix = distance.squareform(distance.pdist(coords))
print distance_matrix.shape
print distance_matrix
Explanation: DBSCAN requires a pre-processing step that K-Means doesn't: converting our "coords" array into a pairwise distance matrix that shows how far every point is away from every other point. Scipy's pdist and squareform functions can handle this for us:
End of explanation
# Fit DBSCAN in the same way we fit K-Means, using the EPS parameter and distance matrix established above
db = DBSCAN(eps=EPS)
db.fit(distance_matrix)
# Now print the results
clusters_to_csv(db.labels_, types, coords)
Explanation: Each entry in the matrix shows how far each of our 1,995 points is from each of the other points in the dataset, in the unit of latitude and longitude degrees. The distances between points are key for DBSCAN to compute densities. But this also exposes one of its weaknesses: it can take a long time to run DBSCAN on huge datasets.
End of explanation
<END_TASK>