<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 moving the work to jupyter
This notebook demonstrates the use of single-line calls to D3 visualizations via the simple d3_lib.py file and referenced css and js files.
Configuration for D3
Step1:  Aogaeru load
Step2:  LINUX_VMSTAT
Step3:  convert the dataframe data into Json format
to_csv does not work since it is necessary to parse it in d3 where it ends up as json format anyways
Step4:  TODO
Step5:  Access logs
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
from korg import korg
from korg.pattern import PatternRepo
import tarfile
from loganalyser import plot
Explanation: moving the work to jupyter
This notebook demonstrates the use of single-line calls to D3 visualizations via the simple d3_lib.py file and referenced css and js files.
Configuration for D3
End of explanation
pr = PatternRepo(['./korg_patterns/'], False)
lg = korg.LineGrokker('%{AOGAERU_LOAD}', pr)
df = pd.DataFrame()
# now grok the aogaeru load log
with open("./data/results.csv") as infile:
    for line in infile:
        #print line
        #print lg.grok(line)
        df = df.append(lg.grok(line), ignore_index=True)
df.timestamp = pd.to_datetime(df.timestamp)  # convert to timestamp
df.head()
df.dtypes
Explanation: Aogaeru load
End of explanation
pr = PatternRepo(['./korg_patterns/'], False)
lg = korg.LineGrokker('%{LINUX_VMSTAT}', pr)
vm = pd.DataFrame()
with tarfile.open('data/supercars-oscounters-13102912.tgz', mode="r:gz") as tar:
    #infile = tar.extractfile('./vmstat.txt.%s' % testrun)
    infile = tar.extractfile('./vmstat.txt.13102912')
    # read first line start and interval
    # 20131029 124500 interval 5 sec
    for line in infile:
        #print line
        #print lg.grok(line)
        vm = vm.append(lg.grok(line), ignore_index=True)
#vm.timestamp = pd.to_datetime(vm.timestamp)  # convert to timestamp
# tz is timezone of data
rng = pd.date_range('10/29/2013 12:45:00', periods=400, freq='5S', tz="Europe/Berlin")
#rng = pd.date_range('10/29/2013 12:45:00', periods=400, freq='5S')
cpu = pd.DataFrame()
cpu['idle'] = vm['id'].astype(int)
cpu['system'] = vm['sy'].astype(int)
cpu['user'] = vm['us'].astype(int)
cpu['wait'] = vm['wa'].astype(int)
# we do not use index since we "to_json" next
#cpu['timestamp'] = rng
#cpu.set_index(['timestamp'], inplace=True)
#cpu.index = pd.to_datetime(rng, utc=True)
cpu['timestamp'] = pd.to_datetime(rng, utc=True) # timestamp in UTC !
cpu.head()
cpu.dtypes
Explanation: LINUX_VMSTAT
End of explanation
#print  cpu.to_csv(orient='records')
print cpu.to_json(orient='records', date_format='epoch')
Explanation: convert the dataframe data into Json format
to_csv does not work since it is necessary to parse it in d3 where it ends up as json format anyways
End of explanation
#HTML(plot.set_styles(['stacked',]))
plot.set_styles(['stacked',])
#HTML(plot.draw_graph('stacked', {'data': cpu.to_json(orient='records')}))
plot.draw_graph('stacked', cpu.to_json(orient='records'))
Explanation: TODO: add from - to (with timezone!)
End of explanation
pr = PatternRepo(['./korg_patterns/'], True)
lg = korg.LineGrokker('%{COMMONAPACHELOG}', pr)
req = pd.DataFrame()
# now grok the aogaeru load log
with open("./data/access_log2") as infile:
    for line in infile:
        req = req.append(lg.grok(line), ignore_index=True)
req.head()
req.timestamp = pd.to_datetime(req.timestamp)  # convert to timestamp
Explanation: Access logs
End of explanation
<END_TASK>