Spectral Norm of Convolutional Layers with Circular and Zero Paddings
Blaise Delattre Quentin Barthélemy Alexandre Allauzen
This paper leverages the use of \emph{Gram iteration} an efficient,
deterministic, and differentiable method for computing spectral norm with an
upper bound guarantee. Designed for circular convolutional layers, we
generalize the use of the Gram iteration to zero padding convolutional layers
and prove its quadratic convergence. We also provide theorems for bridging the
gap between circular and zero padding convolution's spectral norm. We design a
\emph{spectral rescaling} that can be used as a competitive $1$-Lipschitz layer
that enhances network robustness. Demonstrated through experiments, our method
outperforms state-of-the-art techniques in precision, computational cost, and
scalability. The code of experiments is available at
https://github.com/blaisedelattre/lip4conv.