<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Chicken Example -- Off Function Approximation
Lambda=0
Step1:  Average errors
Step2:  Lambda=1.0
Step3:  Commentary
As expected, with lambda set to one, the solution found by TD is the same as the least squares solution.
On average, emphasis TD seems to diverge from the least-squares solution most in the latter states.
These states are the ones that will tend to have the highest emphasis
Sometimes, it appears like the algorithm is trying to get those values right, since they are the ones it is emphasizing.
This case doesn't seem to be a correction-- when using the on-policy distribution, the difference disappears (as would be expected from following the derivation of ETD)
I do not have a good intuition for why there is less discrepancy towards the "middle", that is, about half way between the start state and the "wall"
Get name of notebook and export it, in a roundabout way
<ASSISTANT_TASK:>
Python Code:
ns = 8
states = np.array([i for i in range(ns)])
# Target policy transition matrix
P_pi = np.diag(np.ones(ns-1), 1)
P_pi[-1,0] = 1
# Target policy distribution
d_pi = stationary(P_pi)
# Behavior policy transition matrix
P_mu = np.array([
        [0.0, 1, 0, 0, 0, 0, 0, 0],
        [0.0, 0, 1, 0, 0, 0, 0, 0],
        [0.0, 0, 0, 1, 0, 0, 0, 0],
        [0.0, 0, 0, 0, 1, 0, 0, 0],
        [0.5, 0, 0, 0, 0, 0.5, 0, 0],
        [0.5, 0, 0, 0, 0, 0, 0.5, 0],
        [0.5, 0, 0, 0, 0, 0, 0, 0.5],
        [1.0, 0, 0, 0, 0, 0, 0, 0],
], dtype=np.float)
# Behavior policy distribution
d_mu = stationary(P_mu)
# Reward vector under target policy
rvec = np.zeros(ns)
rvec[-1] = 1
# State-dependent parameters
gamma = 0.9*np.ones(ns)
gamma[0] = 0
lmbda = 0.0*np.ones(ns)
interest = np.ones(ns)
# True values
v_true = ls_solver(P_pi, rvec, np.eye(ns), gamma, d=d_pi)
print("Baseline: RMSE if all values are zero")
print(rmse(v_true, np.zeros_like(v_true), d_pi))
Explanation: Chicken Example -- Off Function Approximation
Lambda=0
End of explanation
num_trials = 5000
phi_length = 6
row_sum = 3
# Store the values
ls_lst = []
td_lst = []
etd_lst = []
for i in range(num_trials):
    print("Trial: %d"%i, end="\r")
    
    # Feature matrix
    X = random_binary(ns, phi_length, row_sum)
    # Least-squares solution
    w_ls = ls_solver(P_pi, rvec, X, gamma, d=d_pi)
    v_ls = np.copy(X @ w_ls)
    # TD solution
    w_td = td_solver(P_pi, rvec, X, gamma, lmbda, d=d_pi)
    v_td = np.copy(X @ w_td)
    # ETD solution
    w_etd = etd_solver(P_pi, rvec, X, gamma, lmbda, interest, d_pi=d_pi, d_mu=d_mu)
    v_etd = np.copy(X @ w_etd)
    
    # Store the data
    ls_lst.append(v_ls)
    td_lst.append(v_td)
    etd_lst.append(v_etd)
# Average the results
ls_avg = np.mean(ls_lst, axis=0)
td_avg = np.mean(td_lst, axis=0)
etd_avg = np.mean(etd_lst, axis=0)
# Average the errors
ls_errors = pd.Series([rmse(v_true, v, d_pi) for v in ls_lst])
td_errors = pd.Series([rmse(v_true, v, d_pi) for v in td_lst])
etd_errors = pd.Series([rmse(v_true, v, d_pi) for v in etd_lst])
# Describe the data
df = pd.DataFrame()
df['least-squares'] = ls_errors
df['TD'] = td_errors
df['ETD'] = etd_errors
df.describe()
print(df.describe().to_string())
# Plot the results
fig, ax = plt.subplots()
width = 0.3
ax.bar(states, ls_avg, width, color='r')
ax.bar(states + width, td_avg, width, color='g')
ax.bar(states + 2*width, etd_avg, width, color='b')
# formatting 
ax.set_title('Approximate state values')
plt.show()
# Plot the results
fig, ax = plt.subplots()
width = 0.3
ax.bar(states, ls_avg - v_true, width, color='r')
ax.bar(states + width, td_avg - v_true, width, color='g')
ax.bar(states + 2*width, etd_avg - v_true, width, color='b')
# formatting 
ax.set_title('Approximate values less true values')
plt.show()
# Average over the values computed for each state
print("Least squares approximation error")
print(rmse(ls_avg, v_true, d_pi))
print("TD approximation error")
print(rmse(td_avg, v_true, d_pi))
print("ETD approximation error")
print(rmse(etd_avg, v_true, d_pi))
# Plot the results
fig, ax = plt.subplots()
width = 0.3
# ax.bar(states, ls_avg - v_true, width, color='r')
ax.bar(states + width, td_avg - ls_avg, width, color='g')
ax.bar(states + 2*width, etd_avg - ls_avg, width, color='b')
# formatting 
ax.set_title('Approximate values less true values')
plt.show()
print("RMS difference between TD and LS solution")
print(rmse(ls_avg, td_avg, d_pi))
print("RMS difference between ETD and LS solution")
print(rmse(ls_avg, etd_avg, d_pi))
Explanation: Average errors
End of explanation
ns = 8
states = np.array([i for i in range(ns)])
# Target policy transition matrix
P_pi = np.diag(np.ones(ns-1), 1)
P_pi[-1,0] = 1
# Target policy distribution
d_pi = stationary(P_pi)
# Behavior policy transition matrix
P_mu = np.array([
        [0.0, 1, 0, 0, 0, 0, 0, 0],
        [0.0, 0, 1, 0, 0, 0, 0, 0],
        [0.0, 0, 0, 1, 0, 0, 0, 0],
        [0.0, 0, 0, 0, 1, 0, 0, 0],
        [0.5, 0, 0, 0, 0, 0.5, 0, 0],
        [0.5, 0, 0, 0, 0, 0, 0.5, 0],
        [0.5, 0, 0, 0, 0, 0, 0, 0.5],
        [1.0, 0, 0, 0, 0, 0, 0, 0],
], dtype=np.float)
# Behavior policy distribution
d_mu = stationary(P_mu)
# Reward vector under target policy
rvec = np.zeros(ns)
rvec[-1] = 1
# State-dependent parameters
gamma = 0.9*np.ones(ns)
gamma[0] = 0
lmbda = 1.0*np.ones(ns)
interest = np.ones(ns)
# True values
v_true = ls_solver(P_pi, rvec, np.eye(ns), gamma, d=d_pi)
num_trials = 5000
phi_length = 6
row_sum = 3
# Store the values
ls_lst = []
td_lst = []
etd_lst = []
for i in range(num_trials):
    print("Trial: %d"%i, end="\r")
    # Feature matrix
    X = random_binary(ns, phi_length, row_sum)
    # Least-squares solution
    w_ls = ls_solver(P_pi, rvec, X, gamma, d=d_pi)
    v_ls = np.copy(X @ w_ls)
    # TD solution
    w_td = td_solver(P_pi, rvec, X, gamma, lmbda, d=d_pi)
    v_td = np.copy(X @ w_td)
    # ETD solution
    w_etd = etd_solver(P_pi, rvec, X, gamma, lmbda, interest, d_pi=d_pi, d_mu=d_mu)
    v_etd = np.copy(X @ w_etd)
    
    # Store the data
    ls_lst.append(v_ls)
    td_lst.append(v_td)
    etd_lst.append(v_etd)
# Average the results
ls_avg = np.mean(ls_lst, axis=0)
td_avg = np.mean(td_lst, axis=0)
etd_avg = np.mean(etd_lst, axis=0)
# Average the errors
ls_errors = pd.Series([rmse(v_true, v, d_pi) for v in ls_lst])
td_errors = pd.Series([rmse(v_true, v, d_pi) for v in td_lst])
etd_errors = pd.Series([rmse(v_true, v, d_pi) for v in etd_lst])
# Describe the data
df = pd.DataFrame()
df['least-squares'] = ls_errors
df['TD'] = td_errors
df['ETD'] = etd_errors
df.describe()
# Plot the results
fig, ax = plt.subplots()
width = 0.3
ax.bar(states, ls_avg, width, color='r')
ax.bar(states + width, td_avg, width, color='g')
ax.bar(states + 2*width, etd_avg, width, color='b')
# formatting 
ax.set_title('Approximate state values')
plt.show()
# Plot the results
fig, ax = plt.subplots()
width = 0.3
ax.bar(states, ls_avg - v_true, width, color='r')
ax.bar(states + width, td_avg - v_true, width, color='g')
ax.bar(states + 2*width, etd_avg - v_true, width, color='b')
# formatting 
ax.set_title('Approximate values less true values')
plt.show()
print("Least squares approximation error")
print(rmse(ls_avg, v_true, d_pi))
print("TD approximation error")
print(rmse(td_avg, v_true, d_pi))
print("ETD approximation error")
print(rmse(etd_avg, v_true, d_pi))
# Plot the results
fig, ax = plt.subplots()
width = 0.3
ax.bar(states + width, td_avg - ls_avg, width, color='g')
ax.bar(states + 2*width, etd_avg - ls_avg, width, color='b')
# formatting 
ax.set_title('RMS Difference between LS solution')
plt.show()
print("RMS difference between TD and LS solution")
print(rmse(ls_avg, td_avg, d_pi))
print("RMS difference between ETD and LS solution")
print(rmse(ls_avg, etd_avg, d_pi))
Explanation: Lambda=1.0
End of explanation
%%javascript
var attribs = document.body.attributes;
var cmd = "theNotebook = " + "'" + attribs['data-notebook-name'].value + "'";
IPython.notebook.kernel.execute(cmd);
%env __NOTEBOOK_NAME $theNotebook
%%bash
echo "ipython nbconvert --to html "$__NOTEBOOK_NAME""
ipython nbconvert --to html "$__NOTEBOOK_NAME"
Explanation: Commentary
As expected, with lambda set to one, the solution found by TD is the same as the least squares solution.
On average, emphasis TD seems to diverge from the least-squares solution most in the latter states.
These states are the ones that will tend to have the highest emphasis
Sometimes, it appears like the algorithm is trying to get those values right, since they are the ones it is emphasizing.
This case doesn't seem to be a correction-- when using the on-policy distribution, the difference disappears (as would be expected from following the derivation of ETD)
I do not have a good intuition for why there is less discrepancy towards the "middle", that is, about half way between the start state and the "wall"
Get name of notebook and export it, in a roundabout way
End of explanation
<END_TASK>