Learning Risk-Aware Quadrupedal Locomotion using Distributional
  Reinforcement Learning
Lukas Schneider Jonas Frey Takahiro Miki Marco Hutter
Deployment in hazardous environments requires robots to understand the risks
associated with their actions and movements to prevent accidents. Despite its
importance, these risks are not explicitly modeled by currently deployed
locomotion controllers for legged robots. In this work, we propose a risk
sensitive locomotion training method employing distributional reinforcement
learning to consider safety explicitly. Instead of relying on a value
expectation, we estimate the complete value distribution to account for
uncertainty in the robot's interaction with the environment. The value
distribution is consumed by a risk metric to extract risk sensitive value
estimates. These are integrated into Proximal Policy Optimization (PPO) to
derive our method, Distributional Proximal Policy Optimization (DPPO). The risk
preference, ranging from risk-averse to risk-seeking, can be controlled by a
single parameter, which enables to adjust the robot's behavior dynamically.
Importantly, our approach removes the need for additional reward function
tuning to achieve risk sensitivity. We show emergent risk sensitive locomotion
behavior in simulation and on the quadrupedal robot ANYmal.