Merino: Entropy-driven Design for Generative Language Models on IoT
  Devices
Youpeng Zhao Ming Lin Huadong Tang Qiang Wu Jun Wang
Generative Large Language Models (LLMs) stand as a revolutionary advancement
in the modern era of artificial intelligence (AI). However, directly deploying
LLMs in resource-constrained hardware, such as Internet-of-Things (IoT)
devices, is difficult due to their high computational cost. In this paper, we
propose a novel information-entropy framework for designing mobile-friendly
generative language models. Our key design paradigm is to maximize the entropy
of transformer decoders within the given computational budgets. The whole
design procedure involves solving a mathematical programming (MP) problem,
which can be done on the CPU within minutes, making it nearly zero-cost. We
evaluate our designed models, termed MeRino, across nine NLP downstream tasks,
showing their competitive performance against the state-of-the-art
autoregressive transformer models under the mobile setting. Notably, MeRino
achieves similar or better zero performance compared to the 350M parameter OPT
while being 4.9x faster on NVIDIA Jetson Nano with 5.5x reduction in model
size. Code will be made available soon.