Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges
Ahmed Haj Yahmed Altaf Allah Abbassi Amin Nikanjam Heng Li Foutse Khomh
Deep reinforcement learning (DRL), leveraging Deep Learning (DL) in
reinforcement learning, has shown significant potential in achieving
human-level autonomy in a wide range of domains, including robotics, computer
vision, and computer games. This potential justifies the enthusiasm and growing
interest in DRL in both academia and industry. However, the community currently
focuses mostly on the development phase of DRL systems, with little attention
devoted to DRL deployment. In this paper, we propose an empirical study on
Stack Overflow (SO), the most popular Q&A forum for developers, to uncover and
understand the challenges practitioners faced when deploying DRL systems.
Specifically, we categorized relevant SO posts by deployment platforms:
server/cloud, mobile/embedded system, browser, and game engine. After filtering
and manual analysis, we examined 357 SO posts about DRL deployment,
investigated the current state, and identified the challenges related to
deploying DRL systems. Then, we investigate the prevalence and difficulty of
these challenges. Results show that the general interest in DRL deployment is
growing, confirming the study's relevance and importance. Results also show
that DRL deployment is more difficult than other DRL issues. Additionally, we
built a taxonomy of 31 unique challenges in deploying DRL to different
platforms. On all platforms, RL environment-related challenges are the most
popular, and communication-related challenges are the most difficult among
practitioners. We hope our study inspires future research and helps the
community overcome the most common and difficult challenges practitioners face
when deploying DRL systems.