ForestColl: Efficient Collective Communications on Heterogeneous Network
  Fabrics
Liangyu Zhao Saeed Maleki Ziyue Yang Hossein Pourreza Aashaka Shah Changho Hwang Arvind Krishnamurthy
As modern DNN models grow ever larger, collective communications between the
accelerators (allreduce, etc.) emerge as a significant performance bottleneck.
Designing efficient communication schedules is challenging given today's highly
diverse and heterogeneous network fabrics. In this paper, we present
ForestColl, a tool that generates efficient schedules for any network topology.
ForestColl constructs broadcast/aggregation spanning trees as the communication
schedule, achieving theoretically minimum network congestion. Its schedule
generation runs in strongly polynomial time and is highly scalable. ForestColl
supports any network fabrics, including both switching fabrics and direct
connections, as well as any network graph structure. We evaluated ForestColl on
multi-cluster AMD MI250 and NVIDIA A100 platforms. ForestColl's schedules
achieved up to 52\% higher performance compared to the vendors' own optimized
communication libraries, RCCL and NCCL. ForestColl also outperforms other
state-of-the-art schedule generation techniques with both up to 61\% more
efficient generated schedules and orders of magnitude faster schedule
generation speed.