Boosting Long-Delayed Reinforcement Learning with Auxiliary
  Short-Delayed Task
Qingyuan Wu Simon Sinong Zhan Yixuan Wang Chung-Wei Lin Chen Lv Qi Zhu Chao Huang
Reinforcement learning is challenging in delayed scenarios, a common
real-world situation where observations and interactions occur with delays.
State-of-the-art (SOTA) state-augmentation techniques either suffer from the
state-space explosion along with the delayed steps, or performance degeneration
in stochastic environments. To address these challenges, our novel
Auxiliary-Delayed Reinforcement Learning (AD-RL) leverages an auxiliary
short-delayed task to accelerate the learning on a long-delayed task without
compromising the performance in stochastic environments. Specifically, AD-RL
learns the value function in the short-delayed task and then employs it with
the bootstrapping and policy improvement techniques in the long-delayed task.
We theoretically show that this can greatly reduce the sample complexity
compared to directly learning on the original long-delayed task. On
deterministic and stochastic benchmarks, our method remarkably outperforms the
SOTAs in both sample efficiency and policy performance.