Policy Learning for Off-Dynamics RL with Deficient Support
Linh Le Pham Van Hung The Tran Sunil Gupta
Reinforcement Learning (RL) can effectively learn complex policies. However,
learning these policies often demands extensive trial-and-error interactions
with the environment. In many real-world scenarios, this approach is not
practical due to the high costs of data collection and safety concerns. As a
result, a common strategy is to transfer a policy trained in a low-cost, rapid
source simulator to a real-world target environment. However, this process
poses challenges. Simulators, no matter how advanced, cannot perfectly
replicate the intricacies of the real world, leading to dynamics discrepancies
between the source and target environments. Past research posited that the
source domain must encompass all possible target transitions, a condition we
term full support. However, expecting full support is often unrealistic,
especially in scenarios where significant dynamics discrepancies arise. In this
paper, our emphasis shifts to addressing large dynamics mismatch adaptation. We
move away from the stringent full support condition of earlier research,
focusing instead on crafting an effective policy for the target domain. Our
proposed approach is simple but effective. It is anchored in the central
concepts of the skewing and extension of source support towards target support
to mitigate support deficiencies. Through comprehensive testing on a varied set
of benchmarks, our method's efficacy stands out, showcasing notable
improvements over previous techniques.