HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic
  Encryption
Seewoo Lee Garam Lee Jung Woo Kim Junbum Shin Mun-Kyu Lee
Transfer learning is a de facto standard method for efficiently training
machine learning models for data-scarce problems by adding and fine-tuning new
classification layers to a model pre-trained on large datasets. Although
numerous previous studies proposed to use homomorphic encryption to resolve the
data privacy issue in transfer learning in the machine learning as a service
setting, most of them only focused on encrypted inference. In this study, we
present HETAL, an efficient Homomorphic Encryption based Transfer Learning
algorithm, that protects the client's privacy in training tasks by encrypting
the client data using the CKKS homomorphic encryption scheme. HETAL is the
first practical scheme that strictly provides encrypted training, adopting
validation-based early stopping and achieving the accuracy of nonencrypted
training. We propose an efficient encrypted matrix multiplication algorithm,
which is 1.8 to 323 times faster than prior methods, and a highly precise
softmax approximation algorithm with increased coverage. The experimental
results for five well-known benchmark datasets show total training times of
567-3442 seconds, which is less than an hour.