Self-Distilled Quantization: Achieving High Compression Rates in
  Transformer-Based Language Models
James O' Neill Sourav Dutta
We investigate the effects of post-training quantization and
quantization-aware training on the generalization of Transformer language
models. We present a new method called self-distilled quantization (SDQ) that
minimizes accumulative quantization errors and outperforms baselines. We apply
SDQ to multilingual models XLM-R-Base and InfoXLM-Base and demonstrate that
both models can be reduced from 32-bit floating point weights to 8-bit integer
weights while maintaining a high level of performance on the XGLUE benchmark.
Our results also highlight the challenges of quantizing multilingual models,
which must generalize to languages they were not fine-tuned on.