Deep Reinforcement Learning with Explicit Context Representation
Francisco Munguia-Galeano Ah-Hwee Tan Ze Ji
Reinforcement learning (RL) has shown an outstanding capability for solving
complex computational problems. However, most RL algorithms lack an explicit
method that would allow learning from contextual information. Humans use
context to identify patterns and relations among elements in the environment,
along with how to avoid making wrong actions. On the other hand, what may seem
like an obviously wrong decision from a human perspective could take hundreds
of steps for an RL agent to learn to avoid. This paper proposes a framework for
discrete environments called Iota explicit context representation (IECR). The
framework involves representing each state using contextual key frames (CKFs),
which can then be used to extract a function that represents the affordances of
the state; in addition, two loss functions are introduced with respect to the
affordances of the state. The novelty of the IECR framework lies in its
capacity to extract contextual information from the environment and learn from
the CKFs' representation. We validate the framework by developing four new
algorithms that learn using context: Iota deep Q-network (IDQN), Iota double
deep Q-network (IDDQN), Iota dueling deep Q-network (IDuDQN), and Iota dueling
double deep Q-network (IDDDQN). Furthermore, we evaluate the framework and the
new algorithms in five discrete environments. We show that all the algorithms,
which use contextual information, converge in around 40,000 training steps of
the neural networks, significantly outperforming their state-of-the-art
equivalents.