Reducing the Barriers to Entry for Foundation Model Training
Paolo Faraboschi Ellis Giles Justin Hotard Konstanty Owczarek Andrew Wheeler
The world has recently witnessed an unprecedented acceleration in demands for
Machine Learning and Artificial Intelligence applications. This spike in demand
has imposed tremendous strain on the underlying technology stack in supply
chain, GPU-accelerated hardware, software, datacenter power density, and energy
consumption. If left on the current technological trajectory, future demands
show insurmountable spending trends, further limiting market players, stifling
innovation, and widening the technology gap. To address these challenges, we
propose a fundamental change in the AI training infrastructure throughout the
technology ecosystem. The changes require advancements in supercomputing and
novel AI training approaches, from high-end software to low-level hardware,
microprocessor, and chip design, while advancing the energy efficiency required
by a sustainable infrastructure. This paper presents the analytical framework
that quantitatively highlights the challenges and points to the opportunities
to reduce the barriers to entry for training large language models.