Optimizing transformer-based machine translation model for single GPU
  training: a hyperparameter ablation study
Luv Verma Ketaki N. Kolhatkar
In machine translation tasks, the relationship between model complexity and
performance is often presumed to be linear, driving an increase in the number
of parameters and consequent demands for computational resources like multiple
GPUs. To explore this assumption, this study systematically investigates the
effects of hyperparameters through ablation on a sequence-to-sequence machine
translation pipeline, utilizing a single NVIDIA A100 GPU. Contrary to
expectations, our experiments reveal that combinations with the most parameters
were not necessarily the most effective. This unexpected insight prompted a
careful reduction in parameter sizes, uncovering "sweet spots" that enable
training sophisticated models on a single GPU without compromising translation
quality. The findings demonstrate an intricate relationship between
hyperparameter selection, model size, and computational resource needs. The
insights from this study contribute to the ongoing efforts to make machine
translation more accessible and cost-effective, emphasizing the importance of
precise hyperparameter tuning over mere scaling.