Leveraging Continuously Differentiable Activation Functions for Learning
  in Quantized Noisy Environments
Vivswan Shah Nathan Youngblood
Real-world analog systems intrinsically suffer from noise that can impede
model convergence and accuracy on a variety of deep learning models. We
demonstrate that differentiable activations like GELU and SiLU enable robust
propagation of gradients which help to mitigate analog quantization error that
is ubiquitous to all analog systems. We perform analysis and training of
convolutional, linear, and transformer networks in the presence of quantized
noise. Here, we are able to demonstrate that continuously differentiable
activation functions are significantly more noise resilient over conventional
rectified activations. As in the case of ReLU, the error in gradients are 100x
higher than those in GELU near zero. Our findings provide guidance for
selecting appropriate activations to realize performant and reliable hardware
implementations across several machine learning domains such as computer
vision, signal processing, and beyond.