Improving the Expressive Power of Deep Neural Networks through Integral
  Activation Transform
Zezhong Zhang Feng Bao Guannan Zhang
The impressive expressive power of deep neural networks (DNNs) underlies
their widespread applicability. However, while the theoretical capacity of deep
architectures is high, the practical expressive power achieved through
successful training often falls short. Building on the insights gained from
Neural ODEs, which explore the depth of DNNs as a continuous variable, in this
work, we generalize the traditional fully connected DNN through the concept of
continuous width. In the Generalized Deep Neural Network (GDNN), the
traditional notion of neurons in each layer is replaced by a continuous state
function. Using the finite rank parameterization of the weight integral kernel,
we establish that GDNN can be obtained by employing the Integral Activation
Transform (IAT) as activation layers within the traditional DNN framework. The
IAT maps the input vector to a function space using some basis functions,
followed by nonlinear activation in the function space, and then extracts
information through the integration with another collection of basis functions.
A specific variant, IAT-ReLU, featuring the ReLU nonlinearity, serves as a
smooth generalization of the scalar ReLU activation. Notably, IAT-ReLU exhibits
a continuous activation pattern when continuous basis functions are employed,
making it smooth and enhancing the trainability of the DNN. Our numerical
experiments demonstrate that IAT-ReLU outperforms regular ReLU in terms of
trainability and better smoothness.