Model-based adaptation for sample efficient transfer in reinforcement
  learning control of parameter-varying systems
Ibrahim Ahmed Marcos Quinones-Grueiro Gautam Biswas
In this paper, we leverage ideas from model-based control to address the
sample efficiency problem of reinforcement learning (RL) algorithms.
Accelerating learning is an active field of RL highly relevant in the context
of time-varying systems. Traditional transfer learning methods propose to use
prior knowledge of the system behavior to devise a gradual or immediate
data-driven transformation of the control policy obtained through RL. Such
transformation is usually computed by estimating the performance of previous
control policies based on measurements recently collected from the system.
However, such retrospective measures have debatable utility with no guarantees
of positive transfer in most cases. Instead, we propose a model-based
transformation, such that when actions from a control policy are applied to the
target system, a positive transfer is achieved. The transformation can be used
as an initialization for the reinforcement learning process to converge to a
new optimum. We validate the performance of our approach through four benchmark
examples. We demonstrate that our approach is more sample-efficient than
fine-tuning with reinforcement learning alone and achieves comparable
performance to linear-quadratic-regulators and model-predictive control when an
accurate linear model is known in the three cases. If an accurate model is not
known, we empirically show that the proposed approach still guarantees positive
transfer with jump-start improvement.