MP-SL: Multihop Parallel Split Learning
Joana Tirana Spyros Lalis Dimitris Chatzopoulos
Federated Learning (FL) stands out as a widely adopted protocol facilitating
the training of Machine Learning (ML) models while maintaining decentralized
data. However, challenges arise when dealing with a heterogeneous set of
participating devices, causing delays in the training process, particularly
among devices with limited resources. Moreover, the task of training ML models
with a vast number of parameters demands computing and memory resources beyond
the capabilities of small devices, such as mobile and Internet of Things (IoT)
devices. To address these issues, techniques like Parallel Split Learning (SL)
have been introduced, allowing multiple resource-constrained devices to
actively participate in collaborative training processes with assistance from
resourceful compute nodes. Nonetheless, a drawback of Parallel SL is the
substantial memory allocation required at the compute nodes, for instance
training VGG-19 with 100 participants needs 80 GB. In this paper, we introduce
Multihop Parallel SL (MP-SL), a modular and extensible ML as a Service (MLaaS)
framework designed to facilitate the involvement of resource-constrained
devices in collaborative and distributed ML model training. Notably, to
alleviate memory demands per compute node, MP-SL supports multihop Parallel
SL-based training. This involves splitting the model into multiple parts and
utilizing multiple compute nodes in a pipelined manner. Extensive
experimentation validates MP-SL's capability to handle system heterogeneity,
demonstrating that the multihop configuration proves more efficient than
horizontally scaled one-hop Parallel SL setups, especially in scenarios
involving more cost-effective compute nodes.