Neural Policy Style Transfer
Raul Fernandez-Fernandez Juan G. Victores Jennifer J. Gago David Estevez Carlos Balaguer
Style Transfer has been proposed in a number of fields: fine arts, natural
language processing, and fixed trajectories. We scale this concept up to
control policies within a Deep Reinforcement Learning infrastructure. Each
network is trained to maximize the expected reward, which typically encodes the
goal of an action, and can be described as the content. The expressive power of
deep neural networks enables encoding a secondary task, which can be described
as the style. The Neural Policy Style Transfer (NPST) algorithm is proposed to
transfer the style of one policy to another, while maintaining the content of
the latter. Different policies are defined via Deep Q-Network architectures.
These models are trained using demonstrations through Inverse Reinforcement
Learning. Two different sets of user demonstrations are performed, one for
content and other for style. Different styles are encoded as defined by user
demonstrations. The generated policy is the result of feeding a content policy
and a style policy to the NPST algorithm. Experiments are performed in a
catch-ball game inspired by the Deep Reinforcement Learning classical Atari
games; and a real-world painting scenario with a full-sized humanoid robot,
based on previous works of the authors. The implementation of three different
Q-Network architectures (Shallow, Deep and Deep Recurrent Q-Network) to encode
the policies within the NPST framework is proposed and the results obtained in
the experiments with each of these architectures compared.