Is It a Free Lunch for Removing Outliers during Pretraining?
Baohao Liao Christof Monz
With the growing size of large language models, the role of quantization
becomes increasingly significant. However, outliers present in weights or
activations notably influence the performance of quantized models. Recently,
\citet{qtransformer} introduced a novel softmax function aimed at pretraining
models in an outlier-free manner, thereby enhancing their suitability for
quantization. Interestingly, we observed that such an approach leads to
performance degradation in full precision. Building on this insight, we enhance
the method by ensuring its normalization is invariant to sequence length, a
crucial factor for bridging the gap between pretraining and fine-tuning.
Moreover, this improved method also facilitates successful pretraining of
causal language models.