<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Get the comments' HTML
Step1:  Extract the comments
Step2:  Create comment stemmer and TFIDF vectorizer
We will do some stemming in the comment text in order to create shorter vectors that represent each comment.
Step3:  Vectorize extracted comments
Step4:  These are the vectorized comments
Step5:  Apply clustering algorithm to vectorized comments
KMeans with 4 clusters
<ASSISTANT_TASK:>
Python Code:
import requests
from bs4 import BeautifulSoup
url = "http://www.theguardian.com/discussion/p/4fqc7"
r = requests.get(url)
html = r.text
soup = BeautifulSoup(html, "html.parser")
comments = soup.select(".d-comment__main")
comment_authors = soup.select(".d-comment__author")
print len (comments), " comments found in first page."
print len (comment_authors), " authors found in first page."
Explanation: Get the comments' HTML
End of explanation
comments_dict = []
parsed_comments = []
parsed_authors = []
for comment, author in zip(comments, comment_authors):
    c = comment.select(".d-comment__body")[0].text
    a = author['title']
    comments_dict.append({"text": c, "author": a})
    parsed_comments.append(c)
    parsed_authors.append(a)
    
print comments_dict[:6]
Explanation: Extract the comments
End of explanation
from sklearn.feature_extraction.text import TfidfVectorizer
import nltk.stem
english_stemmer = nltk.stem.SnowballStemmer('english')
class StemmedTfidfVectorizer(TfidfVectorizer):
    def build_analyzer(self):
        analyzer=super(StemmedTfidfVectorizer,self).build_analyzer()
        return lambda doc:(english_stemmer.stem(w) for w in analyzer(doc))
Explanation: Create comment stemmer and TFIDF vectorizer
We will do some stemming in the comment text in order to create shorter vectors that represent each comment.
End of explanation
stem_vectorizer = StemmedTfidfVectorizer(min_df=1, stop_words='english')
stem_analyze = stem_vectorizer.build_analyzer()
# print [tok for tok in stem_analyze ("When we have a real living wage, there will no longer need to be 'stupid tax credits'. Until then, people need a top up to support themselves, because the companies they work for, don't want to give people their dues.")]
comment_vectors = stem_vectorizer.fit_transform(parsed_comments)
print "%d features found" % (len(stem_vectorizer.get_feature_names()))
print stem_vectorizer.get_feature_names()
Explanation: Vectorize extracted comments
End of explanation
formatted = ["Comment #{0}\n{1}".format(i,cv) for i, cv in enumerate(comment_vectors)]
for f in formatted:
    print f
Explanation: These are the vectorized comments
End of explanation
from sklearn.cluster import KMeans
km = KMeans(n_clusters=4, init='k-means++',
            max_iter=100, n_init=1)
km.fit(comment_vectors)
# Top terms per cluster (out of the 4 clusters)
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = stem_vectorizer.get_feature_names()
for i in range(4):
    print "Cluster %d:"%(i)
    for ind in order_centroids[i, :10]:
        print " %s" % terms[ind]
    print ""
!git add -A && git commit -m "Clusters comments in 4 clusters using kmeans. Now I intend to use agglomerative clustering on the data."
Explanation: Apply clustering algorithm to vectorized comments
KMeans with 4 clusters
End of explanation
<END_TASK>