Comparing Spectral Bias and Robustness For Two-Layer Neural Networks:
  SGD vs Adaptive Random Fourier Features
Aku Kammonen Lisi Liang Anamika Pandey Raúl Tempone
We present experimental results highlighting two key differences resulting
from the choice of training algorithm for two-layer neural networks. The
spectral bias of neural networks is well known, while the spectral bias
dependence on the choice of training algorithm is less studied. Our experiments
demonstrate that an adaptive random Fourier features algorithm (ARFF) can yield
a spectral bias closer to zero compared to the stochastic gradient descent
optimizer (SGD). Additionally, we train two identically structured classifiers,
employing SGD and ARFF, to the same accuracy levels and empirically assess
their robustness against adversarial noise attacks.