FFCV: Accelerating Training by Removing Data Bottlenecks
Guillaume Leclerc Andrew Ilyas Logan Engstrom Sung Min Park Hadi Salman Aleksander Madry
We present FFCV, a library for easy and fast machine learning model training.
FFCV speeds up model training by eliminating (often subtle) data bottlenecks
from the training process. In particular, we combine techniques such as an
efficient file storage format, caching, data pre-loading, asynchronous data
transfer, and just-in-time compilation to (a) make data loading and transfer
significantly more efficient, ensuring that GPUs can reach full utilization;
and (b) offload as much data processing as possible to the CPU asynchronously,
freeing GPU cycles for training. Using FFCV, we train ResNet-18 and ResNet-50
on the ImageNet dataset with competitive tradeoff between accuracy and training
time. For example, we are able to train an ImageNet ResNet-50 model to 75\% in
only 20 mins on a single machine. We demonstrate FFCV's performance,
ease-of-use, extensibility, and ability to adapt to resource constraints
through several case studies. Detailed installation instructions,
documentation, and Slack support channel are available at https://ffcv.io/ .