Three Decades of Activations: A Comprehensive Survey of 400 Activation
  Functions for Neural Networks
Vladimír Kunc Jiří Kléma
Neural networks have proven to be a highly effective tool for solving complex
problems in many areas of life. Recently, their importance and practical
usability have further been reinforced with the advent of deep learning. One of
the important conditions for the success of neural networks is the choice of an
appropriate activation function introducing non-linearity into the model. Many
types of these functions have been proposed in the literature in the past, but
there is no single comprehensive source containing their exhaustive overview.
The absence of this overview, even in our experience, leads to redundancy and
the unintentional rediscovery of already existing activation functions. To
bridge this gap, our paper presents an extensive survey involving 400
activation functions, which is several times larger in scale than previous
surveys. Our comprehensive compilation also references these surveys; however,
its main goal is to provide the most comprehensive overview and systematization
of previously published activation functions with links to their original
sources. The secondary aim is to update the current understanding of this
family of functions.