Stable and Robust Deep Learning By Hyperbolic Tangent Exponential Linear
  Unit (TeLU)
Alfredo Fernandez Ankur Mali
In this paper, we introduce the Hyperbolic Tangent Exponential Linear Unit
(TeLU), a novel neural network activation function, represented as $f(x) =
x{\cdot}tanh(e^x)$. TeLU is designed to overcome the limitations of
conventional activation functions like ReLU, GELU, and Mish by addressing the
vanishing and, to an extent, the exploding gradient problems. Our theoretical
analysis and empirical assessments reveal that TeLU outperforms existing
activation functions in stability and robustness, effectively adjusting
activation outputs' mean towards zero for enhanced training stability and
convergence. Extensive evaluations against popular activation functions (ReLU,
GELU, SiLU, Mish, Logish, Smish) across advanced architectures, including
Resnet-50, demonstrate TeLU's lower variance and superior performance, even
under hyperparameter conditions optimized for other functions. In large-scale
tests with challenging datasets like CIFAR-10, CIFAR-100, and TinyImageNet,
encompassing 860 scenarios, TeLU consistently showcased its effectiveness,
positioning itself as a potential new standard for neural network activation
functions, boosting stability and performance in diverse deep learning
applications.