Privacy-Engineered Value Decomposition Networks for Cooperative
  Multi-Agent Reinforcement Learning
Parham Gohari Matthew Hale Ufuk Topcu
In cooperative multi-agent reinforcement learning (Co-MARL), a team of agents
must jointly optimize the team's long-term rewards to learn a designated task.
Optimizing rewards as a team often requires inter-agent communication and data
sharing, leading to potential privacy implications. We assume privacy
considerations prohibit the agents from sharing their environment interaction
data. Accordingly, we propose Privacy-Engineered Value Decomposition Networks
(PE-VDN), a Co-MARL algorithm that models multi-agent coordination while
provably safeguarding the confidentiality of the agents' environment
interaction data. We integrate three privacy-engineering techniques to redesign
the data flows of the VDN algorithm, an existing Co-MARL algorithm that
consolidates the agents' environment interaction data to train a central
controller that models multi-agent coordination, and develop PE-VDN. In the
first technique, we design a distributed computation scheme that eliminates
Vanilla VDN's dependency on sharing environment interaction data. Then, we
utilize a privacy-preserving multi-party computation protocol to guarantee that
the data flows of the distributed computation scheme do not pose new privacy
risks. Finally, we enforce differential privacy to preempt inference threats
against the agents' training data, past environment interactions, when they
take actions based on their neural network predictions. We implement PE-VDN in
StarCraft Multi-Agent Competition (SMAC) and show that it achieves 80% of
Vanilla VDN's win rate while maintaining differential privacy levels that
provide meaningful privacy guarantees. The results demonstrate that PE-VDN can
safeguard the confidentiality of agents' environment interaction data without
sacrificing multi-agent coordination.