Efficient Skip Connections Realization for Secure Inference on Encrypted
  Data
Nir Drucker Itamar Zimerman
Homomorphic Encryption (HE) is a cryptographic tool that allows performing
computation under encryption, which is used by many privacy-preserving machine
learning solutions, for example, to perform secure classification. Modern deep
learning applications yield good performance for example in image processing
tasks benchmarks by including many skip connections. The latter appears to be
very costly when attempting to execute model inference under HE. In this paper,
we show that by replacing (mid-term) skip connections with (short-term) Dirac
parameterization and (long-term) shared-source skip connection we were able to
reduce the skip connections burden for HE-based solutions, achieving x1.3
computing power improvement for the same accuracy.