<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  Part 1
Step2:  To cluster the images, we'll need to convert the images into a format we can pass into our KMeans model, which expects 1D feature vectors. For this assignment, we'll just flatten our image. This is akin to cutting each image up into rows, and concatenating the rows end-to-end to form one long, skinny image.
Note that this is a rather naive way to vecotrize our images, and there are definitely better ways to represent images for clustering. We'll stick with this for now to keep things simple.
Step3:  Now run the code box below using different numbers of clusters, and note how the clustering results change.
Step4:  1.1) Effect of number of clusters
Q1
Step5:  2.1) The Importance of Normalization
Scroll through the dataframe generated by the codebox above, and take note of the scales of each of the features. They can vary drastically in magnitude! Let's normalize our data before clustering.
Q1
Step6:  2.2) Interpreting Clusters
Now let's cluster our data! Once again, play around with $k$ to see how the results change.
Step7:  Analyzing Centroids
What characterizes each of the clusters?
It can be a little hard to tell what characteristics each cluster has latched on to from cluster membership alone. One way to characterize clusters is to look at the centroids, which are the average values of each cluster. One can think of the centroids as describing the average group member.
Run the following code box to display the values of the centroids of each cluster. Note
Step8:  Visualizing Centroids
These values can be difficult to compare on their own. One good way to visualize cluster centroids is by using a color-coded heatmap of normalized values. Use the code box below to generate such a heat map.
Step9:  For Q2-Q4, please answer using k=3.
Q2
Step10:  Now, let's see where these countries would actually have clustered.
<ASSISTANT_TASK:>
Python Code:
!pip install datacommons --upgrade --quiet
!pip install datacommons_pandas --upgrade --quiet
import datacommons
import datacommons_pandas
import numpy as np
import pandas as pd
# for visualization
import matplotlib.pyplot as plt
import seaborn as sns
# for clustering
from sklearn.cluster import KMeans
Explanation: <a href="https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/intro_data_science/Introduction_to_Clustering.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
Copyright 2022 Google LLC.
SPDX-License-Identifier: Apache-2.0
Introduction to Clustering
Previously, we've seen ways to develop models for supervised learning, or where we have labels, categories, or some other ground-truth values we can use to learn relationships in our data from. But what if we're interested in unsupervised learning? That is, if we have some data without matching labels, is there a way to elucidate patterns inherent in the data itself, without known what those relationships are a-priori?
One common unsupervised learning technique is clustering, sometimes also known as cluster analysis. The idea behind clustering is to divide your datapoints into groups (clusters), such that data points in each group are more similar to each other than to datapoints in other clusters. Clustering is used in many fields for data exploration.
In this assignment, we'll build some intuition through case studies where you'll apply clustering.
Learning Goals
Understand what clustering is and does
See how clustering can be used to find patterns in the data
Build intuition into how clustering works
Get some hands-on experience clustering real-world data
Part 0: A Brief Primer on K-Means
There are many different algorithms for clustering data. For this assignment, we'll be using the k-means algorithm, one of the simplest and most popular clustering algorithms out there.
Most notably, in k-means, you need to specify how many clusters the algorithm should divide your data into, so the number of clusters you select is a tunable hyperparameter. Then, the k-means algorithm attempts to create that many clusters by minimizing the euclidean distance between points in the same cluster.
Don't worry too much about implementation details right now, the goal of this assignment is to build intuition about clustering. We'll look more into the mechanics of how things work in part 2. If you're interested in the details, you can take a look at the wikipedia entry on k-means in the meantime.
Run the following code blocks to install and load all the packages you'll need for this assignment.
End of explanation
# load dataset
from keras.datasets import fashion_mnist
(fashion_mnist, labels), _ = fashion_mnist.load_data()
# limit to first 2000 images so runtimes are reasonable
fashion_mnist = fashion_mnist[:2000, :, :]
# Show the first 25 images as a sample
print(f"Loaded {fashion_mnist.shape[0]} images. The first 25 are:")
sns.set(rc={'figure.figsize':(11,8)})
for i in range(25):
  # define subplot
  plt.subplot(5, 5, i+1)
  # plot raw pixel data
  plt.imshow(fashion_mnist[i], cmap=plt.get_cmap('gray'))
  plt.axis("off")
plt.show()
Explanation: Part 1: Images - Building Intuition with FashionMNIST
We'll start exploring clustering with a simple, intuitive case: clustering the FashionMNIST dataset.
The FashionMNIST dataset is a collection of over 60,000 (28x28) greyscale images of various clothing items (e.g. shoes, shirts, bags) that's often used as a nice toy dataset in computer vision circles. While this particular dataset already has labels for each image, we'll be working with the images only. That is, we'll see what patterns in FashionMNIST we can recover without any labels.
Let's start by loading the dataset and viewing some sample images.
End of explanation
# squash images into 1D
fashion_data = fashion_mnist.reshape((-1, 28*28))
Explanation: To cluster the images, we'll need to convert the images into a format we can pass into our KMeans model, which expects 1D feature vectors. For this assignment, we'll just flatten our image. This is akin to cutting each image up into rows, and concatenating the rows end-to-end to form one long, skinny image.
Note that this is a rather naive way to vecotrize our images, and there are definitely better ways to represent images for clustering. We'll stick with this for now to keep things simple.
End of explanation
# set number of clusters
k = 2   #Change me!
# cluster the images
kmeans = KMeans(n_clusters=k).fit(fashion_data)
# create a dictionary of clusters to images
clusters = {n:[] for n in range(k)}
for i in range(fashion_data.shape[0]):
  key = kmeans.labels_[i]
  value = fashion_data[i,:].reshape(1, 28, 28).squeeze()
  clusters[key].append(value)
# display images from each cluster
sns.set(rc={'figure.figsize':(6,4)})
for cluster_num, images in clusters.items():
  print(f'Cluster {cluster_num} contains {len(images)} images. The first 25 are:')
  for i in range(min(25, len(images))):
    # define subplot
    plt.subplot(5, 5, i+1)
    # plot raw pixel data
    plt.imshow(images[i], cmap=plt.get_cmap('gray'))
    plt.axis("off")
  # show the figure
  plt.show()
Explanation: Now run the code box below using different numbers of clusters, and note how the clustering results change.
End of explanation
dcids = [
  'country/AGO', 'country/ALB', 'country/ARG', 'country/ARM', 'country/AUS',
  'country/AZE', 'country/BDI', 'country/BGD', 'country/BGR', 'country/BIH',
  'country/BLZ', 'country/BOL', 'country/BRA', 'country/BTN', 'country/BWA',
  'country/CAN', 'country/CHL', 'country/CHN', 'country/CMR', 'country/COD',
  'country/COG', 'country/COL', 'country/CRI', 'country/CZE', 'country/DOM',
  'country/DZA', 'country/ECU', 'country/EGY', 'country/ETH', 'country/FJI',
  'country/GAB', 'country/GEO', 'country/GHA', 'country/GTM', 'country/GUY',
  'country/HND', 'country/IDN', 'country/IND', 'country/IRN', 'country/JAM',
  'country/JOR', 'country/JPN', 'country/KAZ', 'country/KEN', 'country/KGZ',
  'country/KIR', 'country/KOR', 'country/LAO', 'country/LBN', 'country/LCA',
  'country/LSO', 'country/MAR', 'country/MDA', 'country/MDG', 'country/MEX',
  'country/MKD', 'country/MLI', 'country/MMR', 'country/MNE', 'country/MNG',
  'country/MOZ', 'country/MRT', 'country/MWI', 'country/MYS', 'country/NAM',
  'country/NER', 'country/NGA', 'country/NIC', 'country/NPL', 'country/PAK',
  'country/PAN', 'country/PER', 'country/PHL', 'country/PNG', 'country/PRY',
  'country/ROU', 'country/RWA', 'country/SDN', 'country/SLV', 'country/SRB',
  'country/SWZ', 'country/SYR', 'country/THA', 'country/TJK', 'country/TKM',
  'country/TLS', 'country/TON', 'country/TTO', 'country/TUN', 'country/TUR',
  'country/TZA', 'country/UGA', 'country/UKR', 'country/USA', 'country/UZB',
  'country/VNM', 'country/VUT', 'country/WSM', 'country/YEM', 'country/ZAF',
  'country/ZMB', 'country/ZWE'
]
stat_vars_to_query = [
  "Amount_Emissions_CarbonDioxide_PerCapita",
  "LifeExpectancy_Person",
  "Count_Person_IsInternetUser_PerCapita",
  "GrowthRate_Count_Person",
  "Count_Person_Upto4Years_Overweight_AsFractionOf_Count_Person_Upto4Years",
  "GiniIndex_EconomicActivity",
  "Count_Product_MobileCellularSubscription_AsFractionOf_Count_Person",
  "Amount_EconomicActivity_GrossDomesticProduction_Nominal_PerCapita",
  "FertilityRate_Person_Female",
  "Count_Death_AsAFractionOfCount_Person"
]
df = datacommons_pandas.build_multivariate_dataframe(dcids,stat_vars_to_query)
# swap index for names
df.insert(0, 'name', df.index.map(datacommons.get_property_values(df.index, 'name')).str[0])
df.set_index('name', drop=True, inplace=True)
display(df)
Explanation: 1.1) Effect of number of clusters
Q1: Try the settings the settings k=2, k=10, and k=15. What value of k results in clusters that are most meaningful?
A1: We should expect k=10 to work best out of the options given. (FashionMNIST had 10 categories). Students might rightfully point out that the "best" k can depend on what you're looking for (e.g. if k=2 is just shoes vs non-shoes, this might be the right level of nuance for your application!)
Q2: What happens if k is too small? What happens if k is too large?
A2: Too small and the clusters will still look heterogenous. Too large and we'll have clusters that seem to represent the same object (e.g. multiple clusters of shirts).
1.2) What traits get elucidated?
Q3: Let's go back to k=2. What features of the images does KMeans seem to be using to split the data into two groups? Why do you think this was the feaure used?
A3: k=2 tends to cluster based on image intensity (e.g. dark vs light images). Because k-means uses euclidean distance between pixel values, high on average and low on average pixel values would be the most distinguishing feature at the k=2 level.
Q4: Now compare k=10 with k=8. Which items or traits group together in k=8 compared to k=10?
A4: Students should notice that items that look similar (e.g. bag with similar area and shading to shirts) will group first with those items that are most similar. 
1.3) Extension Question
Q5 (Extension): Imagine you have a set of 1 million unlabeled images that look just like these fashion MNIST images, and you want to generate categorical labels for each of the images. However, labeling each of those 1 million images is too time intensive and costly.
Devise a strategy for how you could use clustering to help you label each of the images. 
A5: Cluster the 1 million images. Label each of the clusters, and assign every image the label corresponding to its cluster
This is semi-supervised learning!
Part 2: Real-World Data
Now that we've built up some intuition on what k-means does, let's try clustering on some more complex, real world data. We've provided a list of the DCIDs of just over 100 different countries around the world, along with a list of DCIDs statistical variables to start out with.
In case the DCID names are unclear, we'll be looking at the following statistics for each country:
* CO2 emissions per capita
* Life expectancy
* Number of internet users per capita
* Population growth rate
* Percentage of population that is overweight
* Gini Index (A measure of economic inequality)
* Percentage of population with mobile phone subscriptions
* Gross domestic product per captia
* Fertility rate
* Number of deaths per year, normalized by population
Run the following code boxes to load and cluster the data associated with each country.
End of explanation
# normalize the values
mean_df = df.mean()
std_df = df.std()
normalized_df = ((df-mean_df)/std_df)
display(normalized_df)
Explanation: 2.1) The Importance of Normalization
Scroll through the dataframe generated by the codebox above, and take note of the scales of each of the features. They can vary drastically in magnitude! Let's normalize our data before clustering.
Q1: What do you think would happen if we didn't normalize our data before clustering?
A1:Generally the features with higher magnitudes will end up dominating the clustering.
End of explanation
# Clustering using K-means
n_clusters = 3
kmeans_model = KMeans(n_clusters).fit(normalized_df)
labels_df = pd.DataFrame(data=np.transpose(kmeans_model.labels_), index=normalized_df.index, columns=['cluster'])
# list countries in each cluster:
for i in range(n_clusters):
  print(f'Countries in Cluster {i}:')
  print(labels_df.index[labels_df['cluster']==i].tolist())
  print(len(labels_df.index[labels_df['cluster']==i].tolist()))
Explanation: 2.2) Interpreting Clusters
Now let's cluster our data! Once again, play around with $k$ to see how the results change.
End of explanation
# Get centroids of each cluster.
for i in range(n_clusters):
  print(f'\nCluster {i}:')
  # display non-normalized mean values
  mean_to_display = df[labels_df['cluster']==i].mean()
  display(mean_to_display)
Explanation: Analyzing Centroids
What characterizes each of the clusters?
It can be a little hard to tell what characteristics each cluster has latched on to from cluster membership alone. One way to characterize clusters is to look at the centroids, which are the average values of each cluster. One can think of the centroids as describing the average group member.
Run the following code box to display the values of the centroids of each cluster. Note: We're displaying the non-normalized values for better interpretability. The clustering was still performed on normalized values.
End of explanation
# get normalized values
normalized_means = []
for i in range(n_clusters):
  # calculate normalized values for the next part
  mean_normalized = normalized_df[labels_df['cluster']==i].mean()
  normalized_means.append(mean_normalized.to_frame().transpose())
normalized_means_df = pd.concat(normalized_means)
# For better visualization, we'll multiply the following features by -1
# so that a higher value is associated with more development.
features_to_flip = [
  "GiniIndex_EconomicActivity",
  "GrowthRate_Count_Person",
  "FertilityRate_Person_Female"
]
for column in features_to_flip:
  normalized_means_df[column] *= -1
# Plot heatmap
sns.set(rc={'figure.figsize':(11.7,8.27)})
sns.set(font_scale = 1.5)
ax = sns.heatmap(normalized_means_df.to_numpy(), linewidth=0.5, xticklabels=normalized_means_df.columns, center=0)
ax.set_ylabel("Cluster")
plt.show()
Explanation: Visualizing Centroids
These values can be difficult to compare on their own. One good way to visualize cluster centroids is by using a color-coded heatmap of normalized values. Use the code box below to generate such a heat map.
End of explanation
dcids_holdout = ['country/HTI','country/DEU', 'country/IRQ']
# get values for each holdout
df_holdout = datacommons_pandas.build_multivariate_dataframe(dcids_holdout,stat_vars_to_query)
df_holdout = df_holdout.dropna()
# swap index for names
df_holdout.insert(0, 'name', df_holdout.index.map(datacommons.get_property_values(df_holdout.index, 'name')).str[0])
df_holdout.set_index('name', drop=True, inplace=True)
# show values
display(df_holdout)
# normalized version for clustering later
normalized_df_holdout = ((df_holdout-mean_df)/std_df)
Explanation: For Q2-Q4, please answer using k=3.
Q2: What does each cluster seem to represent?
Now, here's a list of 3 held-out countries (i.e., countries that weren't part of the original countries we used to create the clusters).
Germany
Haiti
Iraq
Q3: For each of the countries listed above, make a prediction for which cluster the country would be a member of.
Run the code box below to get feature values for each country that can help you answer Q3.
End of explanation
# find which cluster centroid is closest
for country in df_holdout.index:
  country_data = normalized_df_holdout.loc[country].to_numpy()
  country_data = country_data[np.newaxis, :]
  difference = normalized_means_df.to_numpy() - country_data
  distance = np.linalg.norm(difference,axis=1)
  cluster_membership = np.argmin(distance, axis=0)
  print(f"{country} belongs to cluster {cluster_membership}")
Explanation: Now, let's see where these countries would actually have clustered.
End of explanation
<END_TASK>