Efficient Duple Perturbation Robustness in Low-rank MDPs
Yang Hu Haitong Ma Bo Dai Na Li
The pursuit of robustness has recently been a popular topic in reinforcement
learning (RL) research, yet the existing methods generally suffer from
efficiency issues that obstruct their real-world implementation. In this paper,
we introduce duple perturbation robustness, i.e. perturbation on both the
feature and factor vectors for low-rank Markov decision processes (MDPs), via a
novel characterization of $(\xi,\eta)$-ambiguity sets. The novel robust MDP
formulation is compatible with the function representation view, and therefore,
is naturally applicable to practical RL problems with large or even continuous
state-action spaces. Meanwhile, it also gives rise to a provably efficient and
practical algorithm with theoretical convergence rate guarantee. Examples are
designed to justify the new robustness concept, and algorithmic efficiency is
supported by both theoretical bounds and numerical simulations.