We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies. We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies. Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies. This framework for training deep RL policies involve a zero-sum  dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region. Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm. Without loss of generality, we demonstrate that deep RL policies trained in this fashion will be maximally robust to a ``worst" possible adversarial disturbances. Deep reinforcement learning (RL) for complex agent behavior in realistic environments usually combines function approximation techniques with learning-based control. A good RL controller should guarantee fulfillment of performance specifications under external disturbances, or modeling errors. Quite often in practice, however, this is not the case -with deep RL policies not often generalizing well to real-world scenarios. This can be attributed to the inherent differences between the training and testing environments. Recently, there have been efforts at integrating function approximation techniques with learning-based control, in an end-to-end fashion, in order to have systems that optimize objectives while guaranteeing generalization to environmental uncertainties. Examples include trajectory-based optimization for known dynamics ( BID16 BID25 ), or trajectory optimization for unknown dynamics such as guided policy search algorithms BID0 BID13 BID15 .While these methods produce performance efficiency for agent tasks in the real world, there are sensitivity questions of such policies that need to be addressed such as, how to guarantee maximally robust deep RL policies in the presence of external disturbances, or modeling errors. A typical approach employed in minimizing sample inefficiency is to engineer an agent's policy in a simulated environment, and later transfer such policies to physical environments. However, questions of robustness persist in such scenarios as the agent often has to cope with modeling errors and new sensory inputs from a different environment. For continuous control tasks, learned policies may become brittle in the presence of external perturbations, or a slight change in the system dynamics may significantly affect the performance of the learned controller BID20 -defeating the purpose of having a robust policy that is learned through environmental interaction .The contribution of this paper is two-fold:• first, we provide a framework that demonstrates the brittleness of a state-of-the-art deep RL policy; specifically, given a trained RL policy, we pose an adversarial agent against the fixed trained policy; the goal is to perturb the parameter space of the learned policy. We demonstrate that the most sophisticated deep policies fail in the presence of adversarial perturbations.• second, we formulate an iterative dynamic zero-sum, two player game, where each agent executes an opposite reaction to its pair: a concave-convex problem follows explicitly, and our goal is to achieve a saddle point equilibrium, where the state is everywhere defined but possibly infinite-valued).Noting that lack of generalization of learned reward functions to the real-world can be thought of as external disturbance that perturb the system dynamics, we formulate the learning of robust control policies as a zero-sum two player Markov game -an iterative dynamic game (iDG) -that pits an adversarial agent against a protagonist agent.The controller aims to minimize a given cost while the second agent, an adversary aims to maximize the given cost in the presence of an additive disturbance. We run the algorithm in finite episodic settings and show a dynamic game approach aimed at generating policies that are maximally robust.The content of this paper is thus organized: we review relevant literature to our contribution in Sec. 2; we then provide an H ∞ background in Sec. BID2 . This H ∞ technical introduction will be used in formulating the design of perturbation signals in Sec. BID3 . Without loss of generality, we provide a formal treatment of the iDG algorithm within the guided policy search framework in Sec. 5. Experimental evaluation on multiple robots is provided in Sec. 6 followed by conclusions in Sec. 7. We have evaluated the sensitivity of select deep reinforcement learning algorithms and shown that despite the most carefully designed policies, such policies implemented on real-world agents exhibit a potential for disastrous performances when unexpected such as when there exist modeling errors and discrepancy between training environment and real-world roll-outs (as evidenced by the results from the two dynamics the agent faces in our sensitivity experiment). We then test the dynamic trajectory optimization two-player algorithm on a robot motor task using Levine et al's BID13 's guided policy search algorithm. In our implementation of the dynamic game algorithm, we focus on the robustness parameters that cause the robot's policy to fail in the presence of the erstwhile sensysensitivity parameter. We demonstrate that our two-player game framework allows agents operating under nonlinear dynamics to learn the underlying dynamics under significantly more finite samples than vanilla GPS algorithm does -thus improving upon the Gaussian model mixture method used in BID0 and BID13 .Having agents that are robust to unmodeled nonlinearities, dynamics, and high frequency modes in a nonlinear dynamical system has long been a fundamental question that control theory strives to achieve. To the best of our knowledge, we are not aware of other works that addresses the robustness of deep policies that are trained end-to-end from a maximal robustness perspective. In future work, we hope to replace the crude Gaussian Mixture Model for the dynamics with a more sophisticated nonlinear model, and evaluate how the agent behaves in the presence of unknown dynamics.
This paper demonstrates how H-infinity control theory can help better design robust deep policies for robot motor taks Proposes to incorporate elements of robust control into guided policy research in order to devise a method that is resilient to perturbations and model mismatch. The paper presents a method for evaluating the sensitivity and robustness of deep RL policies, and proposes a dynamic game approach for learning robust policies.