I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural network model.

Suppose there are 2 valid moves in the action space (UP or DOWN)

Suppose the batch size is 5, and the 5 states are this:

$$[s_1, s_2, s_3, s_4, s_5]$$

We put this batch into the neural network model and output Q values. Then we put $[s_1', s_2', s_3', s_4', s_5']$ into a target network.

What I'm confused about is this:

Each state in $[s_1, s_2, s_3, s_4, s_5]$ is different.

Are we computing Q values for UP and DOWN for ALL 5 states after they go through the neural network?

For example, $$[Q_{s_1}(\text{UP}), Q_{s_1}(\text{DOWN})], \\ [Q_{s_2} (\text{UP}), Q_{s_2}(\text{DOWN})], \\ [Q_{s_3}(\text{UP}), Q_{s_3}(\text{DOWN})], \\ [Q_{s_4}(\text{UP}), Q_{s_4}(\text{DOWN})], \\ [Q_{s_5}(\text{UP}), Q_{s_5}(\text{DOWN})]$$


The way the states are used is as follows:

Typically your $Q$-network will state a state as input and output scores over the action space. I.e. $Q : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$. So, in your replay buffer you should store $s_t, a_t, r_{t+1}, s_{t+1}, \mbox{done}$ (note that done just represents where the episode ended on this transition and I add for completeness.

Now, when you are doing your batch updates you sample uniformly at random from this replay buffer. This means you get $B$ tuples of $s_t, a_t, r_{t+1}, s_{t+1}, \mbox{done}$. Now, I will assume $B=1$ as it is easier to explain and the extension to $B > 1$ should be easy to see.

For our state-action tuple $s_t, a_t$ we want to shift what the network predicts for this pair to be closer to $r_{t+1} + \gamma \arg\max_a Q(s,a)$. However, our neural network only takes the state as input, and outputs a vector of scores for each action. That means we want to shift the output of our network for the state $s_t$ towards the target I just mentioned, but only for the action $a_t$ that we took. To do this we just calculate the target, i.e. we calculate $r_{t+1} + \gamma \arg\max_a Q(s,a)$, and then we do gradient ascent like we would a normal neural network where the target vector is the same as the predicted vector everywhere except the $a_t$th element, which we will change to $r_{t+1} + \gamma \arg\max_a Q(s,a)$. This way, our network moves closer to our Q-learning update for only the action we want, in line with how Q-learning works.

It is also worth nothing that you can parameterise your Neural Network to be a function $Q: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ which would make training more in line with how tabular Q-learning but is seldom used in practice as it becomes much more expensive to compute (you have to do a forward pass for each action, rather than one forward pass per state).

  • $\begingroup$ Maybe worth noting that it is valid to model $\hat{q}(s,a, \theta)$ directly with NN mapping $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. That leads to a different approach using minibatches over all actions in each state for max and argmax operations (inefficient) but a simpler construction of training data. I think the approach you outline is more often used in practice though $\endgroup$ – Neil Slater Aug 12 '20 at 15:47
  • $\begingroup$ @NeilSlater I had to do this recently... can get quite messy to programme (or maybe that was just the specific paper I was having to implement...) $\endgroup$ – David Ireland Aug 12 '20 at 15:51
  • $\begingroup$ You might make a neural network for $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ if the number of valid actions in any state is low compared to full action space and/or actions have useful traits that can be generalised over. Also of course if you are working with an RL process that doesn't need max or argmax over Q (e.g. variations of Actor-Critic) $\endgroup$ – Neil Slater Aug 12 '20 at 15:55
  • 1
    $\begingroup$ @mamauwu ah, apologies. It just becomes a typical batch update like in normal neural networks. You could either loop through your batch and do this individually or you can do it as one update. Either way, I think the common method is to accumulate the loss for each sample in the batch, take the average, and then do backprop with it. $\endgroup$ – David Ireland Aug 12 '20 at 16:32
  • 1
    $\begingroup$ Perfect! That's the answer I was looking for. Thank you so much! $\endgroup$ – THAT_AI_GUY Aug 12 '20 at 16:37

Your Answer

