Transformer-Lite: High-efficiency Deployment of Large Language Models on
  Mobile Phone GPUs
Luchang Li Sheng Qian Jie Lu Lunxi Yuan Rui Wang Qin Xie
The Large Language Model (LLM) is widely employed for tasks such as
intelligent assistants, text summarization, translation, and multi-modality on
mobile phones. However, the current methods for on-device LLM deployment
maintain slow inference speed, which causes poor user experience. To facilitate
high-efficiency LLM deployment on device GPUs, we propose four optimization
techniques: (a) a symbolic expression-based approach to support dynamic shape
model inference; (b) operator optimizations and execution priority setting to
enhance inference speed and reduce phone lagging; (c) an FP4 quantization
method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based
technique to eliminate the need for copying KV cache after LLM inference.
Furthermore, we implement these methods in our mobile inference engine,
Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We
evaluated Transformer-Lite's performance using LLMs with varied architectures
and parameters ranging from 2B to 14B. Specifically, we achieved prefill and
decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s
and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based
FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the
prefill speed and 2~3x speedup for the decoding speed.