Spectrum Extraction and Clipping for Implicitly Linear Layers
Ali Ebrahimpour Boroojeny Matus Telgarsky Hari Sundaram
We show the effectiveness of automatic differentiation in efficiently and
correctly computing and controlling the spectrum of implicitly linear
operators, a rich family of layer types including all standard convolutional
and dense layers. We provide the first clipping method which is correct for
general convolution layers, and illuminate the representational limitation that
caused correctness issues in prior work. We study the effect of the batch
normalization layers when concatenated with convolutional layers and show how
our clipping method can be applied to their composition. By comparing the
accuracy and performance of our algorithms to the state-of-the-art methods,
using various experiments, we show they are more precise and efficient and lead
to better generalization and adversarial robustness. We provide the code for
using our methods at https://github.com/Ali-E/FastClip.