Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth
  Soft-Thresholding
Shaik Basheeruddin Shah Pradyumna Pradhan Wei Pu Ramunaidu Randhi Miguel R. D. Rodrigues Yonina C. Eldar
Solving linear inverse problems plays a crucial role in numerous
applications. Algorithm unfolding based, model-aware data-driven approaches
have gained significant attention for effectively addressing these problems.
Learned iterative soft-thresholding algorithm (LISTA) and alternating direction
method of multipliers compressive sensing network (ADMM-CSNet) are two widely
used such approaches, based on ISTA and ADMM algorithms, respectively. In this
work, we study optimization guarantees, i.e., achieving near-zero training loss
with the increase in the number of learning epochs, for finite-layer unfolded
networks such as LISTA and ADMM-CSNet with smooth soft-thresholding in an
over-parameterized (OP) regime. We achieve this by leveraging a modified
version of the Polyak-Lojasiewicz, denoted PL$^*$, condition. Satisfying the
PL$^*$ condition within a specific region of the loss landscape ensures the
existence of a global minimum and exponential convergence from initialization
using gradient descent based methods. Hence, we provide conditions, in terms of
the network width and the number of training samples, on these unfolded
networks for the PL$^*$ condition to hold. We achieve this by deriving the
Hessian spectral norm of these networks. Additionally, we show that the
threshold on the number of training samples increases with the increase in the
network width. Furthermore, we compare the threshold on training samples of
unfolded networks with that of a standard fully-connected feed-forward network
(FFNN) with smooth soft-thresholding non-linearity. We prove that unfolded
networks have a higher threshold value than FFNN. Consequently, one can expect
a better expected error for unfolded networks than FFNN.