<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Subreddit Mapping using t-SNE
This was my first effort at subreddit mapping to test if the idea was vaiable. It turns out that this was mostly quite similar to the final analysis, but I spent a while exploring some other options as well.
Step1:  I hadn't bothered to look if the relevant scikit-learn functions actually accepted sparse matrices when I was just playing, so I did the row normalization myself by hand.
Step2:  Again with the hand-rolled normalisation. It was not hard in this case.
Step3:  Instead of LargeVis we can just use t-SNE. Some caveats
Step4:  Clustering looks pretty much the same as it did in the final version. I played with parameters a little here, and also looked at leaf clustering as the cluster extraction method. In practice, however, the standard Excess of Mass approach was more than adequate.
Step5:  Onto the Bokeh plotting. This was still just experimenting with mapping and clustering so I hadn't honed down the plot code much. I don't do nice colormapping, for instance, but instead plot the noise and cluster points separately. There is also no adjustment of alpha channels based on zoom levels. It was good enough to view the map and mouse over regions to see how well things worked.
Step6:  The final real test was simply print out the contents of the clusters and see if they made sense. For the most part they are pretty good, but they are less good than what LargeVis provided, with more clusters for which there aren't clear topics. Feel free to do exactly this for the LargeVis version and you'll see what I mean.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import scipy.sparse as ss
import numpy as np
from sklearn.decomposition import TruncatedSVD
import sklearn.manifold
import tsne
import re
raw_data = pd.read_csv('subreddit-overlap')
raw_data.head()
subreddit_popularity = raw_data.groupby('t2_subreddit')['NumOverlaps'].sum()
subreddits = np.array(subreddit_popularity.sort_values(ascending=False).index)
index_map = dict(np.vstack([subreddits, np.arange(subreddits.shape[0])]).T)
count_matrix = ss.coo_matrix((raw_data.NumOverlaps, 
                              (raw_data.t2_subreddit.map(index_map),
                               raw_data.t1_subreddit.map(index_map))),
                             shape=(subreddits.shape[0], subreddits.shape[0]),
                             dtype=np.float64)
count_matrix
Explanation: Subreddit Mapping using t-SNE
This was my first effort at subreddit mapping to test if the idea was vaiable. It turns out that this was mostly quite similar to the final analysis, but I spent a while exploring some other options as well.
End of explanation
conditional_prob_matrix = count_matrix.tocsr()
row_sums = np.array(conditional_prob_matrix.sum(axis=1))[:,0]
row_indices, col_indices = conditional_prob_matrix.nonzero()
conditional_prob_matrix.data /= row_sums[row_indices]
reduced_vectors = TruncatedSVD(n_components=500,
                               random_state=0).fit_transform(conditional_prob_matrix)
Explanation: I hadn't bothered to look if the relevant scikit-learn functions actually accepted sparse matrices when I was just playing, so I did the row normalization myself by hand.
End of explanation
reduced_vectors /= np.sqrt((reduced_vectors**2).sum(axis=1))[:, np.newaxis]
Explanation: Again with the hand-rolled normalisation. It was not hard in this case.
End of explanation
seed_state = np.random.RandomState(0)
subreddit_map = tsne.bh_sne(reduced_vectors[:10000], perplexity=50.0, random_state=seed_state)
subreddit_map_df = pd.DataFrame(subreddit_map, columns=('x', 'y'))
subreddit_map_df['subreddit'] = subreddits[:10000]
subreddit_map_df.head()
Explanation: Instead of LargeVis we can just use t-SNE. Some caveats: the tnse package is still quite a bit faster than t-SNE in scikit-learn, but iot only works with python 2.
End of explanation
import hdbscan
clusterer = hdbscan.HDBSCAN(min_samples=5, 
                            min_cluster_size=20).fit(subreddit_map)
cluster_ids = clusterer.labels_
subreddit_map_df['cluster'] = cluster_ids
Explanation: Clustering looks pretty much the same as it did in the final version. I played with parameters a little here, and also looked at leaf clustering as the cluster extraction method. In practice, however, the standard Excess of Mass approach was more than adequate.
End of explanation
from bokeh.plotting import figure, show, output_notebook, output_file
from bokeh.models import HoverTool, ColumnDataSource, value
from bokeh.models.mappers import LinearColorMapper
from bokeh.palettes import viridis
from collections import OrderedDict
output_notebook()
color_mapper = LinearColorMapper(palette=viridis(256), low=0, high=cluster_ids.max())
color_dict = {'field': 'cluster', 'transform': color_mapper}
plot_data_clusters = ColumnDataSource(subreddit_map_df[subreddit_map_df.cluster >= 0])
plot_data_noise = ColumnDataSource(subreddit_map_df[subreddit_map_df.cluster < 0])
tsne_plot = figure(title=u'A Map of Subreddits',
                   plot_width = 700,
                   plot_height = 700,
                   tools= (u'pan, wheel_zoom, box_zoom,'
                           u'box_select, resize, reset'),
                   active_scroll=u'wheel_zoom')
tsne_plot.add_tools( HoverTool(tooltips = OrderedDict([('subreddit', '@subreddit'),
                                                       ('cluster', '@cluster')])))
# draw clusters
tsne_plot.circle(u'x', u'y', source=plot_data_clusters,
                 fill_color=color_dict, line_alpha=0.002, fill_alpha=0.1,
                 size=10, hover_line_color=u'black')
# draw noise
tsne_plot.circle(u'x', u'y', source=plot_data_noise,
                 fill_color=u'gray', line_alpha=0.002, fill_alpha=0.05,
                 size=10, hover_line_color=u'black')
# configure visual elements of the plot
tsne_plot.title.text_font_size = value(u'16pt')
tsne_plot.xaxis.visible = False
tsne_plot.yaxis.visible = False
tsne_plot.grid.grid_line_color = None
tsne_plot.outline_line_color = None
show(tsne_plot);
Explanation: Onto the Bokeh plotting. This was still just experimenting with mapping and clustering so I hadn't honed down the plot code much. I don't do nice colormapping, for instance, but instead plot the noise and cluster points separately. There is also no adjustment of alpha channels based on zoom levels. It was good enough to view the map and mouse over regions to see how well things worked.
End of explanation
def is_nsfw(subreddit):
    return re.search(r'(nsfw|gonewild)', subreddit)
for cid in range(cluster_ids.max() + 1):
    subreddits = subreddit_map_df.subreddit[cluster_ids == cid]
    if np.any(subreddits.map(is_nsfw)):
        subreddits = ' ... Censored ...'
    else:
        subreddits = subreddits.values
        
    print '\nCluster {}:\n{}\n'.format(cid, subreddits) 
Explanation: The final real test was simply print out the contents of the clusters and see if they made sense. For the most part they are pretty good, but they are less good than what LargeVis provided, with more clusters for which there aren't clear topics. Feel free to do exactly this for the LargeVis version and you'll see what I mean.
End of explanation
<END_TASK>