On Task-Relevant Loss Functions in Meta-Reinforcement Learning and
  Online LQR
Jaeuk Shin Giho Kim Howon Lee Joonho Han Insoon Yang
Designing a competent meta-reinforcement learning (meta-RL) algorithm in
terms of data usage remains a central challenge to be tackled for its
successful real-world applications. In this paper, we propose a
sample-efficient meta-RL algorithm that learns a model of the system or
environment at hand in a task-directed manner. As opposed to the standard
model-based approaches to meta-RL, our method exploits the value information in
order to rapidly capture the decision-critical part of the environment. The key
component of our method is the loss function for learning the task inference
module and the system model that systematically couples the model discrepancy
and the value estimate, thereby facilitating the learning of the policy and the
task inference module with a significantly smaller amount of data compared to
the existing meta-RL algorithms. The idea is also extended to a non-meta-RL
setting, namely an online linear quadratic regulator (LQR) problem, where our
method can be simplified to reveal the essence of the strategy. The proposed
method is evaluated in high-dimensional robotic control and online LQR
problems, empirically verifying its effectiveness in extracting information
indispensable for solving the tasks from observations in a sample efficient
manner.