DEAP: Design Space Exploration for DNN Accelerator Parallelism
Ekansh Agrawal Xiangyu Sam Xu
The boom in Large Language Models (LLMs) like GPT-4 and ChatGPT has marked a
significant advancement in artificial intelligence. These models are becoming
increasingly complex and powerful to train and serve. This growth in
capabilities comes with a substantial increase in computational requirements,
both in terms of hardware resources and energy consumption. The goal of this
paper is to showcase how hardware and software co-design can come together and
allow us to create customized hardware systems for specific LLM workloads. We
propose a simulation workflow that allows us to combine model parallelism
techniques with a multi-accelerator simulation framework for efficiency
metrics. We focus on inference workloads and report power, cycle, and latency
metrics upon performing a design space exploration search over multiple
software and hardware configurations.