A High Dimensional Model for Adversarial Training: Geometry and
  Trade-Offs
Kasimir Tanner Matteo Vilucchio Bruno Loureiro Florent Krzakala
This work investigates adversarial training in the context of margin-based
linear classifiers in the high-dimensional regime where the dimension $d$ and
the number of data points $n$ diverge with a fixed ratio $\alpha = n / d$. We
introduce a tractable mathematical model where the interplay between the data
and adversarial attacker geometries can be studied, while capturing the core
phenomenology observed in the adversarial robustness literature. Our main
theoretical contribution is an exact asymptotic description of the sufficient
statistics for the adversarial empirical risk minimiser, under generic convex
and non-increasing losses. Our result allow us to precisely characterise which
directions in the data are associated with a higher generalisation/robustness
trade-off, as defined by a robustness and a usefulness metric. In particular,
we unveil the existence of directions which can be defended without penalising
accuracy. Finally, we show the advantage of defending non-robust features
during training, identifying a uniform protection as an inherently effective
defence mechanism.