Optimal Nonlinearities Improve Generalization Performance of Random
  Features
Samet Demir Zafer Doğan
Random feature model with a nonlinear activation function has been shown to
perform asymptotically equivalent to a Gaussian model in terms of training and
generalization errors. Analysis of the equivalent model reveals an important
yet not fully understood role played by the activation function. To address
this issue, we study the "parameters" of the equivalent model to achieve
improved generalization performance for a given supervised learning problem. We
show that acquired parameters from the Gaussian model enable us to define a set
of optimal nonlinearities. We provide two example classes from this set, e.g.,
second-order polynomial and piecewise linear functions. These functions are
optimized to improve generalization performance regardless of the actual form.
We experiment with regression and classification problems, including synthetic
and real (e.g., CIFAR10) data. Our numerical results validate that the
optimized nonlinearities achieve better generalization performance than
widely-used nonlinear functions such as ReLU. Furthermore, we illustrate that
the proposed nonlinearities also mitigate the so-called double descent
phenomenon, which is known as the non-monotonic generalization performance
regarding the sample size and the model size.