Evaluating and Enhancing Robustness of Deep Recommendation Systems
  Against Hardware Errors
Dongning Ma Xun Jiao Fred Lin Mengshi Zhang Alban Desmaison Thomas Sellinger Daniel Moore Sriram Sankar
Deep recommendation systems (DRS) heavily depend on specialized HPC hardware
and accelerators to optimize energy, efficiency, and recommendation quality.
Despite the growing number of hardware errors observed in large-scale fleet
systems where DRS are deployed, the robustness of DRS has been largely
overlooked. This paper presents the first systematic study of DRS robustness
against hardware errors. We develop Terrorch, a user-friendly, efficient and
flexible error injection framework on top of the widely-used PyTorch. We
evaluate a wide range of models and datasets and observe that the DRS
robustness against hardware errors is influenced by various factors from model
parameters to input characteristics. We also explore 3 error mitigation methods
including algorithm based fault tolerance (ABFT), activation clipping and
selective bit protection (SBP). We find that applying activation clipping can
recover up to 30% of the degraded AUC-ROC score, making it a promising
mitigation method.