ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel
  Decoding
Shuzhang Zhong Zebin Yang Meng Li Ruihao Gong Runsheng Wang Ru Huang
Recent advancements in generative large language models (LLMs) have
significantly boosted the performance in natural language processing tasks.
However, their efficiency is hampered by the inherent limitations in
autoregressive token generation. While parallel decoding with token tree
verification, e.g., Medusa, has been proposed to improve decoding parallelism
and efficiency, it often struggles with maintaining contextual relationships
due to its independent token prediction approach and incurs significant
verification overhead, especially with large tree sizes and batch processing.
In this paper, we propose ProPD, an efficient LLM parallel decoding framework
based on dynamic token tree pruning and generation. ProPD features an advanced
early pruning mechanism to efficiently eliminate unpromising token sequences to
improve verification efficiency. Additionally, it introduces a dynamic token
tree generation algorithm to balance the computation and parallelism of the
verification phase in real-time and maximize the overall efficiency across
different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across
a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD
consistently outperforms existing decoding algorithms by 1.1-3.2x.