A Recipe for Improved Certifiable Robustness: Capacity and Data
Kai Hu Klas Leino Zifan Wang Matt Fredrikson
A key challenge, supported both theoretically and empirically, is that
robustness demands greater network capacity and more data than standard
training. However, effectively adding capacity under stringent Lipschitz
constraints has proven more difficult than it may seem, evident by the fact
that state-of-the-art approach tend more towards \emph{underfitting} than
overfitting. Moreover, we posit that a lack of careful exploration of the
design space for Lipshitz-based approaches has left potential performance gains
on the table. In this work, we provide a more comprehensive evaluation to
better uncover the potential of Lipschitz-based certification methods. Using a
combination of novel techniques, design optimizations, and synthesis of prior
work, we are able to significantly improve the state-of-the-art \emph{verified
robust accuracy} (VRA) for deterministic certification on a variety of
benchmark datasets, and over a range of perturbation sizes. Of particular note,
we discover that the addition of large "Cholesky-orthogonalized residual dense"
layers to the end of existing state-of-the-art Lipschitz-controlled ResNet
architectures is especially effective for increasing network capacity and
performance. Combined with filtered generative data augmentation, our final
results further the state of the art deterministic VRA by up to 8.5 percentage
points. Code is available at \url{https://github.com/hukkai/liresnet}.