Transfer of Reinforcement Learning-Based Controllers from Model- to
  Hardware-in-the-Loop
Mario Picerno Lucas Koch Kevin Badalian Marius Wegener Joschka Schaub Charles Robert Koch Jakob Andert
The process of developing control functions for embedded systems is
resource-, time-, and data-intensive, often resulting in sub-optimal cost and
solutions approaches. Reinforcement Learning (RL) has great potential for
autonomously training agents to perform complex control tasks with minimal
human intervention. Due to costly data generation and safety constraints,
however, its application is mostly limited to purely simulated domains. To use
RL effectively in embedded system function development, the generated agents
must be able to handle real-world applications. In this context, this work
focuses on accelerating the training process of RL agents by combining Transfer
Learning (TL) and X-in-the-Loop (XiL) simulation. For the use case of transient
exhaust gas re-circulation control for an internal combustion engine, use of a
computationally cheap Model-in-the-Loop (MiL) simulation is made to select a
suitable algorithm, fine-tune hyperparameters, and finally train candidate
agents for the transfer. These pre-trained RL agents are then fine-tuned in a
Hardware-in-the-Loop (HiL) system via TL. The transfer revealed the need for
adjusting the reward parameters when advancing to real hardware. Further, the
comparison between a purely HiL-trained and a transferred agent showed a
reduction of training time by a factor of 5.9. The results emphasize the
necessity to train RL agents with real hardware, and demonstrate that the
maturity of the transferred policies affects both training time and
performance, highlighting the strong synergies between TL and XiL simulation.