<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Questions on the (Non-)Interruptibility of Sarsa(λ) and Q-learning
Author
Step1:  Results
%load_md se_results.md
Step2:  Q-learning
See Sarsa(λ) section above for an explanation of the plots.
Step4:  Questions
%load_md questions.md
Ideas for further research
%load_md se_ideas.md
Acknowledgements
Thanks to Rafael Cosman, Stuart Armstrong, Patrick LaVictoire and Satoshi Ono for their comments and advice, also on the old notebook! Thanks to Joel Lehman and Christian Kamm for their ideas for further research!
Bibliography
%load_md se_bib.md
You can ignore the following.
Step5:  Interesting
<ASSISTANT_TASK:>
Python Code:
from mpl_toolkits.mplot3d import Axes3D
import matplotlib
from matplotlib import pyplot as plt
import numpy as np
import numpy.ma as ma
import sys
sys.path.append("..")
from hiora_cartpole import interruptibility
import saveloaddata
import stats_experiments
import stats_experiments as se
data_dir_p = "../data"
Explanation: Questions on the (Non-)Interruptibility of Sarsa(λ) and Q-learning
Author: Richard Möhn, &lt;my first name&gt;.moehn@posteo.de
⭧repo
Abstract
%load_md se_abstract.md
Introduction
%load_md se_intro.md
Related Work
%load_md se_related.md
Method
%load_md se_method.md
End of explanation
plt.close('all')
algo = 'Sarsa'
fig, axes = se.arrange_algo_full()
clim, clim2 = se.load_plot_all(algo, '-rand-tiebreak', 1, axes, fig, data_dir_p)
se.load_plot_all(algo, '-rand-tiebreak', 0, axes, fig, data_dir_p, clim, clim2)
fig
Explanation: Results
%load_md se_results.md
End of explanation
algo = 'Q-learning'
fig, axes = se.arrange_algo_full()
clim, clim2 = se.load_plot_all(algo, '-drt', 1, axes, fig, data_dir_p) 
se.load_plot_all(algo, '-drt', 0, axes, fig, data_dir_p, clim, clim2)
fig
Explanation: Q-learning
See Sarsa(λ) section above for an explanation of the plots.
End of explanation
# Credits: https://nbviewer.jupyter.org/gist/HHammond/7a78d35b34d85406aa60
from IPython import paths
from IPython.core.display import HTML
import os
def css_styling():
    Load default custom.css file from ipython profile
    base = paths.get_ipython_dir()
    styles = "<style>\n%s\n</style>" % (open('custom.css','r').read())
    return HTML(styles)
css_styling()
algo = 'Q-learning'
fig, ax = se.arrange_algo_full()
with saveloaddata.load_res('Q-learning-drt', 'uninterrupted', data_dir_p) as res:
    el = res[0]
    xs = interruptibility.rsxs2nparray(*res)
se.plot_episode_lengths(el[:10], ax.el[0])
se.plot_xs_hist(interruptibility.mask_after_cross(xs).flatten(), ax_comp[0])
before_cross = interruptibility.mask_after_cross(xs)
se.plot_xs_hist(before_cross.compressed(), ax_comp[0])
np.all(before_cross.compressed() <= 1.0)
before_cross.compressed()
se.plot_xs_hist(interruptibility.mask_after_cross(xs).flatten(), ax_comp[0], bins=25)
np.mean(before_cross.flatten()), np.mean(before_cross.compressed())
del before_cross
with saveloaddata.load_res('Sarsa-rand-tiebreak', 'uninterrupted', data_dir_p) as res:
    before_cross_unint = interruptibility.mask_after_cross(interruptibility.rsxs2nparray(*res))
mesh = se.plot_xss_cum_hist_devel(before_cross_int, ax.devel2[1], bins=2)
fig.colorbar(mesh, ax=ax.devel2[1])
fig.colorbar(mesh, ax=ax.devel[1])
mesh = se.plot_xss_cum_hist_devel(before_cross_int, ax.devel[1])
ax.devel[1].colorbar(mesh)
fig
se.plot_xs_hist(interruptibility.mask_after_cross(xs).compressed(), ax, label='uninterrupted')
with saveloaddata.load_res('Sarsa-rand-tiebreak', 'interrupted', data_dir_p) as res:
    before_cross_int = interruptibility.mask_after_cross(interruptibility.rsxs2nparray(*res))
   
se.plot_xs_hist(interruptibility.mask_after_cross(xs).compressed(), ax, label='interrupted')
ax.legend()
fig
stats_experiments.plot_mean_std_change(before_cross_unint, label='uninterrupted')
stats_experiments.plot_mean_std_change(before_cross_int, label='interrupted')
plt.legend()
plt.show()
se.plot_jsd_devel(before_cross_unint)
plt.show()
%debug
fig, ax = plt.subplots()
ax.set_xscale('log')
se.plot_jsd_comp_final(before_cross_unint, ax=ax)
ax.set_xlim([0.0, 0.03])
ax.get_xlim()
plt.show()
Explanation: Questions
%load_md questions.md
Ideas for further research
%load_md se_ideas.md
Acknowledgements
Thanks to Rafael Cosman, Stuart Armstrong, Patrick LaVictoire and Satoshi Ono for their comments and advice, also on the old notebook! Thanks to Joel Lehman and Christian Kamm for their ideas for further research!
Bibliography
%load_md se_bib.md
You can ignore the following.
End of explanation
fig, ax = plt.subplots()
mesh = stats_experiments.plot_xss_cum_hist_change(xs, ax, bins=25)
#plt.colorbar(mesh)
plt.show()
del xs
p = '../data/Sarsa-disc-uninterrupted-xe-170221003432.pickle'
with open(p, 'rb') as f:
    res = pickle.load(f)
    
with open(p, 'wb') as f:
    pickle.dump(res[0:2], f)
Explanation: Interesting: Interruptions don't move the whole thing to the left, but limit variance.
End of explanation
<END_TASK>