Exploring the Robustness of Decentralized Training for Large Language
  Models
Lin Lu Chenxi Dai Wangcheng Tao Binhang Yuan Yanan Sun Pan Zhou
Decentralized training of large language models has emerged as an effective
way to democratize this technology. However, the potential threats associated
with this approach have not been carefully discussed, which would hinder the
development of decentralized training infrastructures. This paper aims to
initiate discussion towards this end by exploring the robustness of
decentralized training from three main perspectives. First, we demonstrate the
vulnerabilities inherent in decentralized training frameworks in terms of
hardware, data, and models. Second, we highlight the fundamental difference
between decentralized foundation model training and vanilla federated learning,
where the security techniques employed in federated learning cannot be applied
directly. Third, we discuss the essential components required for a robust and
efficient decentralized training framework and present a case study by modeling
a concrete threat model. Our objective in this vision paper is to emphasize the
importance of addressing security concerns in the context of decentralized
training for large language models.