Outlier-Efficient Hopfield Layers for Large Transformer-Based Models
Jerry Yao-Chieh Hu Pei-Hsuan Chang Robin Luo Hong-Yu Chen Weijian Li Wei-Po Wang Han Liu
We introduce an Outlier-Efficient Modern Hopfield Model (termed
$\mathtt{OutEffHop}$) and use it to address the outlier-induced challenge of
quantizing gigantic transformer-based models. Our main contribution is a novel
associative memory model facilitating \textit{outlier-efficient} associative
memory retrievals. Interestingly, this memory model manifests a model-based
interpretation of an outlier-efficient attention mechanism
($\text{Softmax}_1$): it is an approximation of the memory retrieval process of
$\mathtt{OutEffHop}$. Methodologically, this allows us to debut novel
outlier-efficient Hopfield layers a powerful attention alternative with
superior post-quantization performance. Theoretically, the Outlier-Efficient
Modern Hopfield Model retains and improves the desirable properties of the
standard modern Hopfield models, including fixed point convergence and
exponential storage capacity. Empirically, we demonstrate the proposed model's
efficacy across large-scale transformer-based and Hopfield-based models
(including BERT, OPT, ViT and STanHop-Net), benchmarking against
state-of-the-art methods including $\mathtt{Clipped\_Softmax}$ and
$\mathtt{Gated\_Attention}$. Notably, $\mathtt{OutEffHop}$ achieves on average
$\sim$22+\% reductions in both average kurtosis and maximum infinity norm of
model outputs accross 4 models.