Constructing Extreme Learning Machines with zero Spectral Bias
Kaumudi Joshi Vukka Snigdha Arya Kumar Bhattacharya
The phenomena of Spectral Bias, where the higher frequency components of a
function being learnt in a feedforward Artificial Neural Network (ANN) are seen
to converge more slowly than the lower frequencies, is observed ubiquitously
across ANNs. This has created technology challenges in fields where resolution
of higher frequencies is crucial, like in Physics Informed Neural Networks
(PINNs). Extreme Learning Machines (ELMs) that obviate an iterative solution
process which provides the theoretical basis of Spectral Bias (SB), should in
principle be free of the same. This work verifies the reliability of this
assumption, and shows that it is incorrect. However, the structure of ELMs
makes them naturally amenable to implementation of variants of Fourier Feature
Embeddings, which have been shown to mitigate SB in ANNs. This approach is
implemented and verified to completely eliminate SB, thus bringing into
feasibility the application of ELMs for practical problems like PINNs where
resolution of higher frequencies is essential.