A Novel Explanation Against Linear Neural Networks
Anish Lakkapragada
Linear Regression and neural networks are widely used to model data. Neural
networks distinguish themselves from linear regression with their use of
activation functions that enable modeling nonlinear functions. The standard
argument for these activation functions is that without them, neural networks
only can model a line. However, a novel explanation we propose in this paper
for the impracticality of neural networks without activation functions, or
linear neural networks, is that they actually reduce both training and testing
performance. Having more parameters makes LNNs harder to optimize, and thus
they require more training iterations than linear regression to even
potentially converge to the optimal solution. We prove this hypothesis through
an analysis of the optimization of an LNN and rigorous testing comparing the
performance between both LNNs and linear regression on synthethic, noisy
datasets.