Machine Learning vs Deep Learning: The Generalization Problem
Yong Yi Bay Kathleen A. Yearick
The capacity to generalize beyond the range of training data is a pivotal
challenge, often synonymous with a model's utility and robustness. This study
investigates the comparative abilities of traditional machine learning (ML)
models and deep learning (DL) algorithms in terms of extrapolation -- a more
challenging aspect of generalization because it requires the model to make
inferences about data points that lie outside the domain it has been trained
on. We present an empirical analysis where both ML and DL models are trained on
an exponentially growing function and then tested on values outside the
training domain. The choice of this function allows us to distinctly showcase
the divergence in performance when models are required to predict beyond the
scope of their training data. Our findings suggest that deep learning models
possess inherent capabilities to generalize beyond the training scope, an
essential feature for real-world applications where data is often incomplete or
extends beyond the observed range. This paper argues for a nuanced
understanding of the structural differences between ML and DL models, with an
emphasis on the implications for both theoretical research and practical
deployment.