<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Clustering
We use a simple k-means algorithm to demonstrate how clustering can be done. Clustering can help discover valuable, hidden groupings within the data. The dataset is created in the Obtain_dataset Notebook.
Step1:  1. Find the clusters using K-means
We show the simplest use of K-means. You can pick the number of clusters that fits your use case best.
Step2:  It looks like cluster 2 focused on negative reviews, while cluster 0 and 1 focused on positive reviews.
Step3:  Visualization of clusters in a 2d projection. The red cluster clearly represents negative reviews. The blue cluster seems quite different from the others. Let's see a few samples from each cluster.
2. Text samples in the clusters & naming the clusters
Let's show random samples from each cluster. We'll use davinci-instruct-beta-v3 to name the clusters, based on a random sample of 6 reviews from that cluster.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
df = pd.read_csv('output/embedded_1k_reviews.csv')
df['text-similarity-babbage-001'] = df.babbage_similarity.apply(eval).apply(np.array)
matrix = np.vstack(df.babbage_similarity.values)
matrix.shape
Explanation: Clustering
We use a simple k-means algorithm to demonstrate how clustering can be done. Clustering can help discover valuable, hidden groupings within the data. The dataset is created in the Obtain_dataset Notebook.
End of explanation
from sklearn.cluster import KMeans
n_clusters = 4
kmeans = KMeans(n_clusters = n_clusters,init='k-means++',random_state=42)
kmeans.fit(matrix)
labels = kmeans.labels_
df['Cluster'] = labels
df.groupby('Cluster').Score.mean().sort_values()
Explanation: 1. Find the clusters using K-means
We show the simplest use of K-means. You can pick the number of clusters that fits your use case best.
End of explanation
from sklearn.manifold import TSNE
import matplotlib
import matplotlib.pyplot as plt
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)
vis_dims2 = tsne.fit_transform(matrix)
x = [x for x,y in vis_dims2]
y = [y for x,y in vis_dims2]
for category, color in enumerate(['purple', 'green', 'red', 'blue']):
    xs = np.array(x)[df.Cluster==category]
    ys = np.array(y)[df.Cluster==category]
    plt.scatter(xs, ys, color=color, alpha=0.3)
    avg_x = xs.mean()
    avg_y = ys.mean()
    
    plt.scatter(avg_x, avg_y, marker='x', color=color, s=100)
plt.title("Clusters identified visualized in language 2d using t-SNE")
Explanation: It looks like cluster 2 focused on negative reviews, while cluster 0 and 1 focused on positive reviews.
End of explanation
import openai
# Reading a review which belong to each group.
rev_per_cluster = 3
for i in range(n_clusters):
    print(f"Cluster {i} Theme:", end=" ")
    
    reviews = "\n".join(df[df.Cluster == i].combined.str.replace("Title: ", "").str.replace("\n\nContent: ", ":  ").sample(rev_per_cluster, random_state=42).values)
    response = openai.Completion.create(
        engine="davinci-instruct-beta-v3",
        prompt=f"What do the following customer reviews have in common?\n\nCustomer reviews:\n\"\"\"\n{reviews}\n\"\"\"\n\nTheme:",
        temperature=0,
        max_tokens=64,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0
    )
    print(response["choices"][0]["text"].replace('\n',''))
    sample_cluster_rows = df[df.Cluster == i].sample(rev_per_cluster, random_state=42) 
    for j in range(rev_per_cluster):
        print(sample_cluster_rows.Score.values[j], end=", ")
        print(sample_cluster_rows.Summary.values[j], end=":   ")
        print(sample_cluster_rows.Text.str[:70].values[j])
        
    print("-" * 100)
Explanation: Visualization of clusters in a 2d projection. The red cluster clearly represents negative reviews. The blue cluster seems quite different from the others. Let's see a few samples from each cluster.
2. Text samples in the clusters & naming the clusters
Let's show random samples from each cluster. We'll use davinci-instruct-beta-v3 to name the clusters, based on a random sample of 6 reviews from that cluster.
End of explanation
<END_TASK>