Exact capacity of the \emph{wide} hidden layer treelike neural networks
  with generic activations
Mihailo Stojnic
Recent progress in studying \emph{treelike committee machines} (TCM) neural
networks (NN) in
\cite{Stojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23,Stojnictcmspnncapdiffactrdt23}
showed that the Random Duality Theory (RDT) and its a \emph{partially
lifted}(pl RDT) variant are powerful tools that can be used for very precise
networks capacity analysis. Here, we consider \emph{wide} hidden layer networks
and uncover that certain aspects of numerical difficulties faced in
\cite{Stojnictcmspnncapdiffactrdt23} miraculously disappear. In particular, we
employ recently developed \emph{fully lifted} (fl) RDT to characterize the
\emph{wide} ($d\rightarrow \infty$) TCM nets capacity. We obtain explicit,
closed form, capacity characterizations for a very generic class of the hidden
layer activations. While the utilized approach significantly lowers the amount
of the needed numerical evaluations, the ultimate fl RDT usefulness and success
still require a solid portion of the residual numerical work. To get the
concrete capacity values, we take four very famous activations examples:
\emph{\textbf{ReLU}}, \textbf{\emph{quadratic}}, \textbf{\emph{erf}}, and
\textbf{\emph{tanh}}. After successfully conducting all the residual numerical
work for all of them, we uncover that the whole lifting mechanism exhibits a
remarkably rapid convergence with the relative improvements no better than
$\sim 0.1\%$ happening already on the 3-rd level of lifting. As a convenient
bonus, we also uncover that the capacity characterizations obtained on the
first and second level of lifting precisely match those obtained through the
statistical physics replica theory methods in \cite{ZavPeh21} for the generic
and in \cite{BalMalZech19} for the ReLU activations.