I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey
Noah Lewis Jean Luca Bez Suren Byna
High-Performance Computing (HPC) systems excel in managing distributed
workloads, and the growing interest in Artificial Intelligence (AI) has
resulted in a surge in demand for faster methods of Machine Learning (ML) model
training and inference. In the past, research on HPC I/O focused on optimizing
the underlying storage system for modeling and simulation applications and
checkpointing the results, causing writes to be the dominant I/O operation.
These applications typically access large portions of the data written by
simulations or experiments. ML workloads, in contrast, perform small I/O reads
spread across a large number of random files. This shift of I/O access patterns
poses several challenges to HPC storage systems. In this paper, we survey I/O
in ML applications on HPC systems, and target literature within a 6-year time
window from 2019 to 2024. We provide an overview of the common phases of ML,
review available profilers and benchmarks, examine the I/O patterns encountered
during ML training, explore I/O optimizations utilized in modern ML frameworks
and proposed in recent literature, and lastly, present gaps requiring further
R&D. We seek to summarize the common practices used in accessing data by ML
applications and expose research gaps that could spawn further R&D.