Agent based modelling for continuously varying supply chains
Wan Wang Haiyan Wang Adam J. Sobey
Problem definition: Supply chains are constantly evolving networks.
Reinforcement learning is increasingly proposed as a solution to provide
optimal control of these networks. Academic/practical: However, learning in
continuously varying environments remains a challenge in the reinforcement
learning literature.Methodology: This paper therefore seeks to address whether
agents can control varying supply chain problems, transferring learning between
environments that require different strategies and avoiding catastrophic
forgetting of tasks that have not been seen in a while. To evaluate this
approach, two state-of-the-art Reinforcement Learning (RL) algorithms are
compared: an actor-critic learner, Proximal Policy Optimisation(PPO), and a
Recurrent Proximal Policy Optimisation (RPPO), PPO with a Long Short-Term
Memory(LSTM) layer, which is showing popularity in online learning
environments. Results: First these methods are compared on six sets of
environments with varying degrees of stochasticity. The results show that more
lean strategies adopted in Batch environments are different from those adopted
in Stochastic environments with varying products. The methods are also compared
on various continuous supply chain scenarios, where the PPO agents are shown to
be able to adapt through continuous learning when the tasks are similar but
show more volatile performance when changing between the extreme tasks.
However, the RPPO, with an ability to remember histories, is able to overcome
this to some extent and takes on a more realistic strategy. Managerial
implications: Our results provide a new perspective on the continuously varying
supply chain, the cooperation and coordination of agents are crucial for
improving the overall performance in uncertain and semi-continuous
non-stationary supply chain environments without the need to retrain the
environment as the demand changes.