Breaking the Curse of Dimensionality in Deep Neural Networks by Learning
  Invariant Representations
Leonardo Petrini
Artificial intelligence, particularly the subfield of machine learning, has
seen a paradigm shift towards data-driven models that learn from and adapt to
data. This has resulted in unprecedented advancements in various domains such
as natural language processing and computer vision, largely attributed to deep
learning, a special class of machine learning models. Deep learning arguably
surpasses traditional approaches by learning the relevant features from raw
data through a series of computational layers.
  This thesis explores the theoretical foundations of deep learning by studying
the relationship between the architecture of these models and the inherent
structures found within the data they process. In particular, we ask What
drives the efficacy of deep learning algorithms and allows them to beat the
so-called curse of dimensionality-i.e. the difficulty of generally learning
functions in high dimensions due to the exponentially increasing need for data
points with increased dimensionality? Is it their ability to learn relevant
representations of the data by exploiting their structure? How do different
architectures exploit different data structures? In order to address these
questions, we push forward the idea that the structure of the data can be
effectively characterized by its invariances-i.e. aspects that are irrelevant
for the task at hand.
  Our methodology takes an empirical approach to deep learning, combining
experimental studies with physics-inspired toy models. These simplified models
allow us to investigate and interpret the complex behaviors we observe in deep
learning systems, offering insights into their inner workings, with the
far-reaching goal of bridging the gap between theory and practice.