Environmental-Impact Based Multi-Agent Reinforcement Learning
Farinaz Alamiyan-Harandi Pouria Ramazi
To promote cooperation and strengthen the individual impact on the collective
outcome in social dilemmas, we propose the Environmental-impact Multi-Agent
Reinforcement Learning (EMuReL) method where each agent estimates the
"environmental impact" of every other agent, that is, the difference in the
current environment state compared to the hypothetical environment in the
absence of that other agent. Inspired by the Inequity Aversion model, the agent
then compares its own reward with those of its fellows multiplied by their
environmental impacts. If its reward exceeds the scaled reward of one of its
fellows, the agent takes "social responsibility" toward that fellow by reducing
its own reward. Therefore, the less influential an agent is in reaching the
current state, the more social responsibility is taken by other agents.
Experiments in the Cleanup (resp. Harvest) test environment demonstrate that
agents trained based on EMuReL learn to cooperate more effectively and obtain
$54\%$ ($39\%$) and $20\%$ ($44\%$) more total rewards while preserving the
same cooperation levels compared to when they are trained based on the two
state-of-the-art reward reshaping methods inequity aversion and social
influence.