Zero-Space Cost Fault Tolerance for Transformer-based Language Models on
  ReRAM
Bingbing Li Geng Yuan Zigeng Wang Shaoyi Huang Hongwu Peng Payman Behnam Wujie Wen Hang Liu Caiwen Ding
Resistive Random Access Memory (ReRAM) has emerged as a promising platform
for deep neural networks (DNNs) due to its support for parallel in-situ
matrix-vector multiplication. However, hardware failures, such as
stuck-at-fault defects, can result in significant prediction errors during
model inference. While additional crossbars can be used to address these
failures, they come with storage overhead and are not efficient in terms of
space, energy, and cost. In this paper, we propose a fault protection mechanism
that incurs zero space cost. Our approach includes: 1) differentiable structure
pruning of rows and columns to reduce model redundancy, 2) weight duplication
and voting for robust output, and 3) embedding duplicated most significant bits
(MSBs) into the model weight. We evaluate our method on nine tasks of the GLUE
benchmark with the BERT model, and experimental results prove its
effectiveness.