Black holes and the loss landscape in machine learning
Pranav Kumar Taniya Mandal Swapnamay Mondal
Understanding the loss landscape is an important problem in machine learning.
One key feature of the loss function, common to many neural network
architectures, is the presence of exponentially many low lying local minima.
Physical systems with similar energy landscapes may provide useful insights. In
this work, we point out that black holes naturally give rise to such
landscapes, owing to the existence of black hole entropy. For definiteness, we
consider 1/8 BPS black holes in $\mathcal{N} = 8$ string theory. These provide
an infinite family of potential landscapes arising in the microscopic
descriptions of corresponding black holes. The counting of minima amounts to
black hole microstate counting. Moreover, the exact numbers of the minima for
these landscapes are a priori known from dualities in string theory. Some of
the minima are connected by paths of low loss values, resembling mode
connectivity. We estimate the number of runs needed to find all the solutions.
Initial explorations suggest that Stochastic Gradient Descent can find a
significant fraction of the minima.