Kindness in Multi-Agent Reinforcement Learning
Farinaz Alamiyan-Harandi Mersad Hassanjani Pouria Ramazi
In human societies, people often incorporate fairness in their decisions and
treat reciprocally by being kind to those who act kindly. They evaluate the
kindness of others' actions not only by monitoring the outcomes but also by
considering the intentions. This behavioral concept can be adapted to train
cooperative agents in Multi-Agent Reinforcement Learning (MARL). We propose the
KindMARL method, where agents' intentions are measured by counterfactual
reasoning over the environmental impact of the actions that were available to
the agents. More specifically, the current environment state is compared with
the estimation of the current environment state provided that the agent had
chosen another action. The difference between each agent's reward, as the
outcome of its action, with that of its fellow, multiplied by the intention of
the fellow is then taken as the fellow's "kindness". If the result of each
reward-comparison confirms the agent's superiority, it perceives the fellow's
kindness and reduces its own reward. Experimental results in the Cleanup and
Harvest environments show that training based on the KindMARL method enabled
the agents to earn 89\% (resp. 37\%) and 44% (resp. 43\%) more total rewards
than training based on the Inequity Aversion and Social Influence methods. The
effectiveness of KindMARL is further supported by experiments in a traffic
light control problem.