Improving Autonomous Separation Assurance through Distributed
  Reinforcement Learning with Attention Networks
Marc W. Brittain Luis E. Alvarez Kara Breeden
Advanced Air Mobility (AAM) introduces a new, efficient mode of
transportation with the use of vehicle autonomy and electrified aircraft to
provide increasingly autonomous transportation between previously underserved
markets. Safe and efficient navigation of low altitude aircraft through highly
dense environments requires the integration of a multitude of complex
observations, such as surveillance, knowledge of vehicle dynamics, and weather.
The processing and reasoning on these observations pose challenges due to the
various sources of uncertainty in the information while ensuring cooperation
with a variable number of aircraft in the airspace. These challenges coupled
with the requirement to make safety-critical decisions in real-time rule out
the use of conventional separation assurance techniques. We present a
decentralized reinforcement learning framework to provide autonomous
self-separation capabilities within AAM corridors with the use of speed and
vertical maneuvers. The problem is formulated as a Markov Decision Process and
solved by developing a novel extension to the sample-efficient, off-policy soft
actor-critic (SAC) algorithm. We introduce the use of attention networks for
variable-length observation processing and a distributed computing architecture
to achieve high training sample throughput as compared to existing approaches.
A comprehensive numerical study shows that the proposed framework can ensure
safe and efficient separation of aircraft in high density, dynamic environments
with various sources of uncertainty.