<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Interactive UMAP visualization with Graphistry for scale and explainability
UMAP is a great algorithm for clustering rich data. It is commonly used for turning data with many columns into more approachable 2D/3D visualizations, finding clusters, and making cluster classifiers. You may be familiar with earlier dimensionality-reduction algorithms like PCA, K-Means, and t-SNE that UMAP generally improves upon. As great as UMAP is, unfortunately, traditional UMAP results are still hard to understand for questions like what elements are in a cluster or why, and hard to interact with as you tweak their many settings. This notebook shows how to use Graphistry and graphs to quickly understand and interact with the results. 
We demonstrate working the leading CPU implementation of UMAP, umap_learn. You may also enjoy our end-to-end GPU tutorial that uses the GPU-accelerated RAPIDS cuML implementaton of UMAP, though it currently takes an extract step of having to manually compute the k-nn.
For an ongoing example, we use a security event log (IPs, timestamps, counts, alert names, ...), and we've seen similarly great results for areas like fraud, genomics, and misinformation
Step1:  Prep 2
Step2:  Prep 3
Step3:  Prep 4
Step4:  UMAP
UMAP has many options -- run help(umap.UMAP) for details. It returns two things we use
Step5:  Visualize 1
Step6:  Fascinatingly, when Graphistry's force-directed graph layout algorithm reuses UMAP's inferred edge connectivities, the layout does not significantly change from what UMAP computes. Try hitting the "play" button in the tool to see for yourself! That means the graph-based intuitions for subsequent interactions, such as removing key nodes/edges and reclustering, should be consistent.
Visualize 2
<ASSISTANT_TASK:>
Python Code:
# Already installed in Graphistry & RAPIDS distros
# ! pip install --user umap-learn
# ! pip install --user graphistry
import graphistry, pandas as pd, umap
# To specify Graphistry account & server, use:
# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')
# For more options, see https://github.com/graphistry/pygraphistry#configure
Explanation: Interactive UMAP visualization with Graphistry for scale and explainability
UMAP is a great algorithm for clustering rich data. It is commonly used for turning data with many columns into more approachable 2D/3D visualizations, finding clusters, and making cluster classifiers. You may be familiar with earlier dimensionality-reduction algorithms like PCA, K-Means, and t-SNE that UMAP generally improves upon. As great as UMAP is, unfortunately, traditional UMAP results are still hard to understand for questions like what elements are in a cluster or why, and hard to interact with as you tweak their many settings. This notebook shows how to use Graphistry and graphs to quickly understand and interact with the results. 
We demonstrate working the leading CPU implementation of UMAP, umap_learn. You may also enjoy our end-to-end GPU tutorial that uses the GPU-accelerated RAPIDS cuML implementaton of UMAP, though it currently takes an extract step of having to manually compute the k-nn.
For an ongoing example, we use a security event log (IPs, timestamps, counts, alert names, ...), and we've seen similarly great results for areas like fraud, genomics, and misinformation:
Prep 1: Install
Prep 2: Load and clean data
Prep 3: Featurization
Prep 4: Normalize & weight
UMAP
Visualize 1: UMAP as a graph
Visualize 2: Explaining UMAP connections
Prep 1: Install
Install umap and graphistry if you have not already
If you are not running a graphistry server, you can use a free Hub account via the username/password option
End of explanation
df = pd.read_csv('../../data/honeypot.csv')
df['victimPort'] = df['victimPort'].astype('uint32')
df['time(max)'] = pd.to_datetime(df['time(max)'] * 1000 * 1000 * 1000)
df['time(min)'] = pd.to_datetime(df['time(min)'] * 1000 * 1000 * 1000)
print(df.info())
df.sample(5)
Explanation: Prep 2: Load and clean data
UMAP works with most tabular data. You can use it with rows that have strings, numbers, dates and more!
The below small example is server logs of security honeypots getting hacked.
End of explanation
dummmies = [
    pd.get_dummies(df[c], prefix=f'{c}_oh')
    for c in ['victimIP', 'victimPort', 'vulnName']
]
encoded_ips = ([
    df[[]].assign(
        attackerIP_a = df['attackerIP'].str.extract("^(\d+)\.").astype('uint8'),
        attackerIP_b = df['attackerIP'].str.extract("^\d+\.(\d+)\.").astype('uint8'),
        attackerIP_c = df['attackerIP'].str.extract("^\d+\.\d+\.(\d+)\.").astype('uint8'),
        attackerIP_d = df['attackerIP'].str.extract("^\d+\.\d+\.\d+\.(\d+)$").astype('uint8'),
        victimIP_a = df['victimIP'].str.extract("^(\d+)\.").astype('uint8'),
        victimIP_b = df['victimIP'].str.extract("^\d+\.(\d+)\.").astype('uint8'),
        victimIP_c = df['victimIP'].str.extract("^\d+\.\d+\.(\d+)\.").astype('uint8'),
        victimIP_d = df['victimIP'].str.extract("^\d+\.\d+\.\d+\.(\d+)$").astype('uint8')
    )
])
orig_continuous = [
    df[['victimPort', 'count', 'time(max)', 'time(min)']].assign(
        duration=df['time(max)'] - df['time(min)']
    )
]
df2 = pd.concat(encoded_ips + dummmies + orig_continuous, axis=1)
print('new shape:', df2.info())
df2.sample(5)
Explanation: Prep 3: Featurization
UMAP operates on numeric columns, so we create a new table of numeric values using several common feature encodings:
* Replace categorical values like specific IPs, ports, and alert names with many one-hot encoded columns. Ex: For column "victimIP", and many columns like "victimIP_oh_127.0.0.1" whose values are 0/1
* Component columns: Split IPs like "172.31.13.124" into parts like "172" vs "31" in case there are phenomena like coordinated IP ranges
* Compute derived and entangled columns, like augmenting the min/max times of an alert being seen with the duration (max - min)
While the original data only had 7 columns, the new one has 33. We've worked with 10K+ columns in GPU-accelerated use cases.
You may benefit from using libraries to streamline the normalization. We only use pandas calls to be clear, and in a way that is directly translatable to cuDF for automatic GPU acceleration on bigger workloads.
End of explanation
df3 = df2.copy()
for c in df3:
    #print(c)
    df3[c] = ((df3[c] - df3[c].min())/(df3[c].max() - df3[c].min())).fillna(0)
print(df3.info())
df3.sample(5)
Explanation: Prep 4: Normalize & weight
Once you have numeric data, UMAP is still sensitive to how you normalize each column. We do a simple conversion of each column to values between 0-1.
Fancier normalizations of some columns might also try to adjust for aspects like the distribution. Likewise, you can try increasing specific columns to being 0-10 to increase their relative weight.
End of explanation
# see help(umap.UMAP)
umap_options = {
    'n_components': 2,
    'metric': 'euclidean'
}
%%time
embedding = umap.UMAP(**umap_options).fit(df3)
embedding
%%time
coo = embedding.graph_.tocoo()
print('coo lens', len(coo.row), len(coo.col), len(coo.data))
print(coo.row[0:5], coo.col[0:5], coo.data[0:5])
weighted_edges_df = pd.DataFrame({
    's': coo.row,
    'd': coo.col,
    'w': coo.data
})
weighted_edges_df.sample(3)
nodes_df = pd.concat([
    df,
    pd.DataFrame(embedding.embedding_).rename(columns={0: 'x', 1: 'y'})
], axis=1)
nodes_df['x'] = nodes_df['x'] * 100
nodes_df['y'] = nodes_df['y'] * 100
nodes_df = nodes_df.reset_index().rename(columns={'index': 'n'})
print(nodes_df.info())
nodes_df.sample(5)
Explanation: UMAP
UMAP has many options -- run help(umap.UMAP) for details. It returns two things we use:
* An (x,y) position pair for each record
* A weighted edgelist (sparse matrix) listing higher-value similarities between records
We enrich our original data frame with the x/y positions and create a new one with the edges. Note that we throw away most of the features: we'll get explainable summaries later.
End of explanation
# Most of the settings are optional and can be changed on-the-fly in the UI
g = (
    graphistry
    .nodes(nodes_df, 'n')
    .edges(weighted_edges_df, 's', 'd')
    .bind(point_x='x', point_y='y', edge_weight='w')
    .settings(url_params={'play': 0, 'edgeInfluence': 5})
    .encode_edge_color('w', ['maroon', 'pink', 'white'], as_continuous=True)
    .encode_point_size('count')
)
g.plot()
Explanation: Visualize 1: Interactive UMAP using graphs
We first use UMAP data for an interactive graph visualization you can inspect and manipulate
Nodes: Represents the original records
Position: From the UMAP embedding
Size: Bind to the original 'count' column
Color: Use Graphistry's default to autoinfer a community label based on edges (below)
Edges: Shows UMAP's inferred connectivites (correlations)
Color: Edge weight, cold to hot
Weights: From UMAP's inferred connectivities
You can think of UMAP's edges as being the most important pairwise weighted votes stating "these records should be close together". In force-directed graph layout algorithms, they act as elastic springs that prevent the nodes from drifting too far apart.
The visualization lets you interactively explore phenomena like coloring by alert name and time and drilling into specific clusters. As UMAP is fairly fast here (and can be faster via the cuML flow), we have both a fast visual interaction loop and decently fast coding loop.
End of explanation
#triple: src_node_EDGE_dst_node
edge_triples = (g
 ._edges
 .merge(g._nodes, left_on=g._source, right_on=g._node)
 .rename(columns={c: f'src_{c}' for c in g._nodes})
 .merge(g._nodes, left_on=g._destination, right_on=g._node)
 .rename(columns={c: f'dst_{c}' for c in g._nodes})
)
#print(edge_triplescolumns)
equivs = []
for c in g._nodes:
    equiv = edge_triples[ edge_triples[f'src_{c}'] == edge_triples[f'dst_{c}'] ]
    if len(equiv) > 0:
        equiv = equiv[[g._source, g._destination]].assign(
            type=c,
            match_val=edge_triples[f'src_{c}'],
            w=0.1)
        equiv[c] = edge_triples[f'src_{c}']
        print('adding', c, len(equiv))
        equivs.append(equiv)
    else:
        print('no hits on col', c)
equivs_df = pd.concat(equivs)
equivs_df['match_val'] = equivs_df['match_val'].astype(str)  # ensure arrow works  
#equivs_df.sample(10)
edges2 = pd.concat([g._edges.assign(type='umap', match_val='1'), equivs_df])
g2 = (g
      .edges(edges2)
      #.edges(edges2[edges2['type'] == 'attackerIP'])
      .bind(edge_label='match_val')
      .encode_edge_color('type', categorical_mapping={
          'umap': 'grey',
          'victimIP': 'blue',
          'attackerIP': 'lightblue',
          'victimPort': 'green',
          'vulnName': 'yellow',
          'count': 'white'
      })
)
g2.plot()
print(g2._edges.info())
g2._edges.sample(3)
Explanation: Fascinatingly, when Graphistry's force-directed graph layout algorithm reuses UMAP's inferred edge connectivities, the layout does not significantly change from what UMAP computes. Try hitting the "play" button in the tool to see for yourself! That means the graph-based intuitions for subsequent interactions, such as removing key nodes/edges and reclustering, should be consistent.
Visualize 2: Explainable UMAP connections
When nodes have many features, even having UMAP's edges showing their nearest neighbors does not clarify which attributes best explain why they are being clustered. For example, the primary partitioning largely follows alert name, but those split into subclusters with interesting designs based on secondary factors like IP address and time.
To visually clarify which features a pair of nodes have in commmon, we add edges between them, one for each attribute in common. The more common features between two nodes, the more edges. This is similar to how graphistry.hypergraph(df, ...)['graph'].plot() works. For simplicity, the below algorithm computes new edges far exact feature matches.
For initial intuition, we color the edges based on the type -- IP, alert name, etc. -- but it can also may make sense to color them by specific values, like a particular IP address or alert name.
End of explanation
<END_TASK>