Bitformer: An efficient Transformer with bitwise operation-based
  attention for Big Data Analytics at low-cost low-precision devices
Gaoxiang Duan Junkai Zhang Xiaoying Zheng Yongxin Zhu
In the current landscape of large models, the Transformer stands as a
cornerstone, playing a pivotal role in shaping the trajectory of modern models.
However, its application encounters challenges attributed to the substantial
computational intricacies intrinsic to its attention mechanism. Moreover, its
reliance on high-precision floating-point operations presents specific hurdles,
particularly evident in computation-intensive scenarios such as edge computing
environments. These environments, characterized by resource-constrained devices
and a preference for lower precision, necessitate innovative solutions.
  To tackle the exacting data processing demands posed by edge devices, we
introduce the Bitformer model, an inventive extension of the Transformer
paradigm. Central to this innovation is a novel attention mechanism that
adeptly replaces conventional floating-point matrix multiplication with bitwise
operations. This strategic substitution yields dual advantages. Not only does
it maintain the attention mechanism's prowess in capturing intricate long-range
information dependencies, but it also orchestrates a profound reduction in the
computational complexity inherent in the attention operation. The transition
from an $O(n^2d)$ complexity, typical of floating-point operations, to an
$O(n^2T)$ complexity characterizing bitwise operations, substantiates this
advantage. Notably, in this context, the parameter $T$ remains markedly smaller
than the conventional dimensionality parameter $d$.
  The Bitformer model in essence endeavors to reconcile the indomitable
requirements of modern computing landscapes with the constraints posed by edge
computing scenarios. By forging this innovative path, we bridge the gap between
high-performing models and resource-scarce environments, thus unveiling a
promising trajectory for further advancements in the field.