<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Notebook Title <a class="tocSkip">
Make some notes here for the thought process, steps taken, TODOs.
Imports
Import deps
Map all dependencies under categories, for easier tracking / readability.
Also check
Step2:  Inserts for Jupyter
Any kind of IPython/Jupyter related stuff
Step3:  Import data
Make sure to check if the data is present in the targeted scope, and that the size is readable with current RAM.
Feel free to have subheadings for multiple datasets, data descriptions etc.
Step4:  Main
Pick a more flat structure (h1 headings with #) or nested structure appropriately - not too nested, not too cluttered.
Use the bash, Luke
Step5:  Make use of IPython stuff
F-strings (>= Python 3.6) can be combined with IPython's <kbd>display</kbd> module for fun and profit.
Step6:  For more nested json's or dictionaries, it's best to use something interactive like RenderJSON.
Step8:  Use slides, and decouple declaration from run (if you're not using "Hide codecell inputs").
Step9:  Use tqdm for every run that takes more than a couple of seconds and can be tracked by some iterator.
<ASSISTANT_TASK:>
Python Code:
# BASE ------------------------------------
from datetime import datetime as dt
nb_start = dt.now()
# Be mindful when you have this activated.
# import warnings
# warnings.filterwarnings('ignore')
import json
from pathlib import Path
from time import sleep
# Display libs
from IPython.display import display, HTML
from tqdm import tqdm, tqdm_notebook
tqdm.pandas()
SEED = 24
%%time
# ETL ------------------------------------
import numpy as np
import pandas as pd
# VIZ ------------------------------------
import matplotlib.cm as cm
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import plotly.graph_objs as go
import plotly.figure_factory as ff
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode(connected=True)
import plotly.express as px
import plotly.io as pio
from plotly.tools import mpl_to_plotly
# NETWORK ANALYSIS ------------------------------------
import networkx as nx
import community as community_louvain
Explanation: Notebook Title <a class="tocSkip">
Make some notes here for the thought process, steps taken, TODOs.
Imports
Import deps
Map all dependencies under categories, for easier tracking / readability.
Also check: https://github.com/xR86/ml-stuff/tree/master/scripts
End of explanation
# https://stackoverflow.com/a/37124230
import uuid
from IPython.display import display_javascript, display_html, display
import json
class RenderJSON(object):
    def __init__(self, json_data):
        if isinstance(json_data, dict):
            self.json_str = json.dumps(json_data)
        else:
            self.json_str = json_data
        self.uuid = str(uuid.uuid4())
    def _ipython_display_(self):
        display_html('<div id="{}" style="height: 100%; width:100%;"></div>'.format(self.uuid), raw=True)
        display_javascript(
        require(["https://rawgit.com/caldwell/renderjson/master/renderjson.js"], function() {
        document.getElementById('%s').appendChild(renderjson(%s))
        });
         % (self.uuid, self.json_str), raw=True)
%%javascript
/*Increase timeout to load properly*/
var rto = 120;
console.log('[Custom]: Increase require() timeout to', rto, 'seconds.');
window.requirejs.config({waitSeconds: rto});
%%html
<style>
    /* font for TODO */
    @import url('https://fonts.googleapis.com/css?family=Oswald&display=swap');
    
    .hl {
        padding: 0.25rem 0.3rem;
        border-radius: 5px;
    }
    /* used: https://www.color-hex.com/color-palette/87453 */
    .hl.hl-yellow  { background-color: rgba(204,246,43,0.5); /*#fdef41;*/ }
    .hl.hl-orange  { background-color: rgba(255,150,42,0.5); }
    .hl.hl-magenta { background-color: rgba(244,73,211,0.5); }
    .hl.hl-blue    { background-color: rgba(80,127,255,0.5); }
    .hl.hl-violet  { background-color: rgba(149,47,255,0.5); }
    
    .todo {
        font-family: 'Oswald', sans-serif;
        font-size: 2rem;
    }
    
    input.checkmark {
        height: 1.5rem;
        margin-right: 0.5rem;
    }
                    
    kbd.cr {
        padding: 2px 3px;
        background-color: red;
        color: #FFF;
        border-radius: 5px;
    }
    kbd.xmltag {
        background-color: #ff8c8c;
        color: #FFF;
    }
    kbd.xmltag.xmltag--subnode {
        background-color: #9f8cff;
        color: #FFF;
    }
    kbd.xmltag.xmltag--subsubnode {
        background-color: #de8cff;
        color: #FFF;
    }
</style>
<!-- ========================================== -->
<h3 style="margin-top:1rem; margin-bottom:2rem"> Examples: </h3>
    
<div>Highlighted text in:
    <span class="hl hl-yellow">yellow</span>,
    <span class="hl hl-orange">orange</span>,
    <span class="hl hl-magenta">magenta</span>,
    <span class="hl hl-blue">blue</span>,
    <span class="hl hl-violet">violet</span>,
</div>
<br/>
<div class="todo">TODO</div>  
<input class="checkmark" type="checkbox" checked="checked" disabled>Finished TODO text.  
<input class="checkmark" type="checkbox" disabled>TODO text.
<br/><br/>
Tags: <kbd class="cr">CR</kbd> (CR for Camera-Ready, graphs/sections that are important)
Explanation: Inserts for Jupyter
Any kind of IPython/Jupyter related stuff:
- classes that leverage the <kbd>display</kbd> module,
- javascript inserts,
- HTML/CSS inserts to be reused in multiple displays for highlighting,
- table of contents marking, etc.
End of explanation
%%bash
ls -l tests/
%%time
# df = pd.read_csv()
# df.info()
# df.head()
Explanation: Import data
Make sure to check if the data is present in the targeted scope, and that the size is readable with current RAM.
Feel free to have subheadings for multiple datasets, data descriptions etc.
End of explanation
%%bash
ls -l tests/
%%bash
ls data/raw | wc -l | xargs printf '%s files'
du -h data/raw | cut -f1 | xargs printf ', total of %s'
ls data/raw/ | head -n 4 | xargs printf '\n\t%s'
ls data/raw/ | tail -n 4 | xargs printf '\n\t%s'
Explanation: Main
Pick a more flat structure (h1 headings with #) or nested structure appropriately - not too nested, not too cluttered.
Use the bash, Luke
End of explanation
tm = (dt.now() - nb_start).total_seconds()
display(HTML(f'Started notebook <span class="hl hl-yellow">{tm:.0f}s</span> ago.'))
# If you use type specifiers, don't put space after the specifier
# display(HTML(f'{ tm:.0f}'))  # works
# display(HTML(f'{ tm:.0f }')) # breaks
Explanation: Make use of IPython stuff
F-strings (>= Python 3.6) can be combined with IPython's <kbd>display</kbd> module for fun and profit.
End of explanation
RenderJSON({
    'a': {
        'c': 0
    },
    'b': 1
})
Explanation: For more nested json's or dictionaries, it's best to use something interactive like RenderJSON.
End of explanation
slide_1 = HTML(
<h3>Lex Fridman<br/><br/>
Deep Learning Basics: Introduction and Overview<br/>&nbsp;</h3>
<iframe width="560" height="315" src="https://www.youtube.com/embed/O5xeyoRL95U" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
)
slide_1
Explanation: Use slides, and decouple declaration from run (if you're not using "Hide codecell inputs").
End of explanation
# For most simple stuff, use tqdm
files = list(range(10))
for file in tqdm_notebook(files):
    sleep(0.1)
# When you need more control over the progress bar, 
# use decoupled tqdm
with tqdm_notebook(total=len(files)) as pbar:
    for file in files:
        sleep(0.1)
        pbar.update(1)
nb_end = dt.now()
print('Time elapsed: %s' % (nb_end - nb_start))
'Time elapsed: %.2f minutes' % (
    (nb_end - nb_start).total_seconds() / 60
)
Explanation: Use tqdm for every run that takes more than a couple of seconds and can be tracked by some iterator.
End of explanation
<END_TASK>