I can't see it but I can Fine-tune it: On Encrypted Fine-tuning of
  Transformers using Fully Homomorphic Encryption
Prajwal Panzade Daniel Takabi Zhipeng Cai
In today's machine learning landscape, fine-tuning pretrained transformer
models has emerged as an essential technique, particularly in scenarios where
access to task-aligned training data is limited. However, challenges surface
when data sharing encounters obstacles due to stringent privacy regulations or
user apprehension regarding personal information disclosure. Earlier works
based on secure multiparty computation (SMC) and fully homomorphic encryption
(FHE) for privacy-preserving machine learning (PPML) focused more on
privacy-preserving inference than privacy-preserving training. In response, we
introduce BlindTuner, a privacy-preserving fine-tuning system that enables
transformer training exclusively on homomorphically encrypted data for image
classification. Our extensive experimentation validates BlindTuner's
effectiveness by demonstrating comparable accuracy to non-encrypted models.
Notably, our findings highlight a substantial speed enhancement of 1.5x to 600x
over previous work in this domain.