Does a form of reinforcement learning exist where an agent can only receive reward based on its current state, rather than a perceived future reward assessed by reasoning over the agent's possible future actions? The reward must still result in the evolution of near optimal policy chains.

  • $\begingroup$ The agent has no control of the reward system (or function), so it is a bit unclear what you're asking. Are you asking if is there an agent that acts based solely on the current state and the possible immediately reward he can receive after having taken an action from the current state? Furthemore, your question in the title is a little bit different than the questions that you ask in the body of the post (provided they mean what I think). $\endgroup$ – nbro Jun 14 '19 at 16:12
  • $\begingroup$ I'm assuming the agent senses its environment, and based on that carries out an action for which it receives a reward which it could not predict before selecting the action. The reward received should be fed back historically through the prior decision chain to strengthen the chain, or weaken it, in proportion to the reward received. I'm allowing for the existence of multiple agents, with a selection of the current agent being made based on sensor input and prior reward. $\endgroup$ – Nick Jun 14 '19 at 16:46
  • $\begingroup$ if you cant rollout or perceive future rewards and you just want a model that maps state to present reward, doesnt that just become a normal supervised learning regression problem? $\endgroup$ – mshlis Jun 14 '19 at 16:47
  • $\begingroup$ Ok. So, what's your question? $\endgroup$ – nbro Jun 14 '19 at 16:47
  • $\begingroup$ Ok, I'll use an example. I want to learn a policy to escape a maze with 1 exit 1 entrance. An agent can move forward, turn left or right and sense if a wall is in front of it. An ideal meta-policy is always to keep 'its hand' on the wall and move so it never loses contact. The selected agent receives immediate reward according to how it moves, and partial reward from future agent-action pairs to reinforce the agent-sensor-action chain. The ideal meta-policy may not be optimal for any particular maze, but is guaranteed to find the exit for any matrix. $\endgroup$ – Nick Jun 14 '19 at 17:05

The literature knows two different techniques: model-based reinforcement learning and model-free reinforcement learning.[1] To make things more complicated, the model-free approach can be enhanced with a reward model.[2] A reward model is equal to a reward function which can be learned with learning from demonstration.[3] Sometimes, it's called inverse reinforcement learning, because at the beginning the reward model isn't available.

To answer the initial question directly, the answer is yes; reinforcement learning without a model is possible. The reward model has to be created on the fly with incoming data. In theory, this technique is able to solve more complex problems, which are going beyond a line following robot.

[1] What's the difference between model-free and model-based reinforcement learning?, What's the difference between model-free and model-based reinforcement learning?

[2] Section Injecting Human Knowledge, https://ai.stackexchange.com/a/11821/11571

[3] Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA), Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)


Your Answer

