Converting Transformers to Polynomial Form for Secure Inference Over
  Homomorphic Encryption
Itamar Zimerman Moran Baruch Nir Drucker Gilad Ezov Omri Soceanu Lior Wolf
Designing privacy-preserving deep learning models is a major challenge within
the deep learning community. Homomorphic Encryption (HE) has emerged as one of
the most promising approaches in this realm, enabling the decoupling of
knowledge between the model owner and the data owner. Despite extensive
research and application of this technology, primarily in convolutional neural
networks, incorporating HE into transformer models has been challenging because
of the difficulties in converting these models into a polynomial form. We break
new ground by introducing the first polynomial transformer, providing the first
demonstration of secure inference over HE with transformers. This includes a
transformer architecture tailored for HE, alongside a novel method for
converting operators to their polynomial equivalent. This innovation enables us
to perform secure inference on LMs with WikiText-103. It also allows us to
perform image classification with CIFAR-100 and Tiny-ImageNet. Our models yield
results comparable to traditional methods, bridging the performance gap with
transformers of similar scale and underscoring the viability of HE for
state-of-the-art applications. Finally, we assess the stability of our models
and conduct a series of ablations to quantify the contribution of each model
component.