Adaptive Activation Functions for Predictive Modeling with Sparse
  Experimental Data
Farhad Pourkamali-Anaraki Tahamina Nasrin Robert E. Jensen Amy M. Peterson Christopher J. Hansen
A pivotal aspect in the design of neural networks lies in selecting
activation functions, crucial for introducing nonlinear structures that capture
intricate input-output patterns. While the effectiveness of adaptive or
trainable activation functions has been studied in domains with ample data,
like image classification problems, significant gaps persist in understanding
their influence on classification accuracy and predictive uncertainty in
settings characterized by limited data availability. This research aims to
address these gaps by investigating the use of two types of adaptive activation
functions. These functions incorporate shared and individual trainable
parameters per hidden layer and are examined in three testbeds derived from
additive manufacturing problems containing fewer than one hundred training
instances. Our investigation reveals that adaptive activation functions, such
as Exponential Linear Unit (ELU) and Softplus, with individual trainable
parameters, result in accurate and confident prediction models that outperform
fixed-shape activation functions and the less flexible method of using
identical trainable activation functions in a hidden layer. Therefore, this
work presents an elegant way of facilitating the design of adaptive neural
networks in scientific and engineering problems.