Penalty Decoding: Well Suppress the Self-Reinforcement Effect in
  Open-Ended Text Generation
Wenhong Zhu Hongkun Hao Rui Wang
The decoding algorithm is critical for open-ended text generation,
transforming latent representations into coherent and meaningful outputs. This
paper investigates the self-reinforcement effect in text generation and the
effectiveness of a repetition penalty to mitigate it. However, determining the
optimal repetition penalty value is challenging. To tackle this, we propose a
forgetting mechanism that disregards distant tokens, reducing the burden of
penalty selection. In addition, we introduce a length penalty to address overly
short sentences caused by excessive penalties. Our penalty decoding approach
incorporating three strategies helps resolve issues with sampling methods
deviating from factual information. Experimental results demonstrate the
efficacy of our approach in generating high-quality sentences resembling human
output.