How to understand the RLstep in Keepaway (Compare with Sarsa)
<p>In "Stone, Peter, Richard S. Sutton, and Gregory Kuhlmann. "Reinforcement learning for robocup soccer keepaway." Adaptive Behavior 13.3 (2005): 165-188.", the RLstep pseudocode seems quite a bit different from Sarsa(λ), which the authors say RLStep implements. </p>

<p>Here is the <a href="https://i.stack.imgur.com/rpp8P.png" rel="nofollow">RLstep pseudocode</a> and here is the <a href="https://i.stack.imgur.com/3cFXy.png" rel="nofollow">Sarsa(lambda) pseudocode</a>.</p>

<p>The areas of confusion are:</p>

<ul>
<li><p>Line 10 in the Sarsa(λ) pseudocode updates the Q value for each
state-action pair <em>after</em> adding 1 to the <code>e(s,a)</code>. But in the RLstep
pseudocode the eligibility trace update (line 19) doesn't happen
until after the value update (line 17). </p></li>
<li><p>Lines 18 and 19 in RLstep seem quite different from the Sarsa(λ) pseudocode.</p></li>
<li><p>What are lines 20-25 doing with the eligibility trace?</p></li>
</ul>