DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Xuanlei Zhao Shenggan Cheng Zangwei Zheng Zheming Yang Ziming Liu Yang You
Scaling large models with long sequences across applications like language
generation, video generation and multimodal tasks requires efficient sequence
parallelism. However, existing sequence parallelism methods all assume a single
sequence dimension and fail to adapt to multi-dimensional transformer
architectures that perform attention calculations across different dimensions.
This paper introduces Dynamic Sequence Parallelism (DSP), a novel approach to
enable efficient sequence parallelism for multi-dimensional transformer models.
The key idea is to dynamically switch the parallelism dimension according to
the current computation stage, leveraging the potential characteristics of
multi-dimensional attention. This dynamic dimension switching allows sequence
parallelism with minimal communication overhead compared to applying
traditional single-dimension parallelism to multi-dimensional models.
Experiments show DSP improves end-to-end throughput by 42.0% to 216.8% over
prior sequence parallelism methods.