TENPLEX: Changing Resources of Deep Learning Jobs using Parallelizable
  Tensor Collections
Marcel Wagenländer Guo Li Bo Zhao Luo Mai Peter Pietzuch
Deep learning (DL) jobs use multi-dimensional parallelism, i.e they combine
data, model, and pipeline parallelism, to use large GPU clusters efficiently.
This couples jobs tightly to a set of GPU devices, but jobs may experience
changes to the device allocation: (i) resource elasticity during training adds
or removes devices; (ii) hardware maintenance may require redeployment on
different devices; and (iii) device failures force jobs to run with fewer
devices. Current DL frameworks lack support for these scenarios, as they cannot
change the multi-dimensional parallelism of an already-running job in an
efficient and model-independent way.
  We describe Tenplex, a state management library for DL frameworks that
enables jobs to change the GPU allocation and job parallelism at runtime.
Tenplex achieves this by externalizing the DL job state during training as a
parallelizable tensor collection (PTC). When the GPU allocation for the DL job
changes, Tenplex uses the PTC to transform the DL job state: for the dataset
state, Tenplex repartitions it under data parallelism and exposes it to workers
through a virtual file system; for the model state, Tenplex obtains it as
partitioned checkpoints and transforms them to reflect the new parallelization
configuration. For efficiency, these PTC transformations are executed in
parallel with a minimum amount of data movement between devices and workers.
Our experiments show that Tenplex enables DL jobs to support dynamic
parallelization with low overhead.