BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor
  Learning
Baoyuan Wu Hongrui Chen Mingda Zhang Zihao Zhu Shaokui Wei Danni Yuan Mingli Zhu Ruotong Wang Li Liu Chao Shen
As an emerging and vital topic for studying deep neural networks'
vulnerability (DNNs), backdoor learning has attracted increasing interest in
recent years, and many seminal backdoor attack and defense algorithms are being
developed successively or concurrently, in the status of a rapid arms race.
However, mainly due to the diverse settings, and the difficulties of
implementation and reproducibility of existing works, there is a lack of a
unified and standardized benchmark of backdoor learning, causing unfair
comparisons, and unreliable conclusions (e.g., misleading, biased or even false
conclusions). Consequently, it is difficult to evaluate the current progress
and design the future development roadmap of this literature. To alleviate this
dilemma, we build a comprehensive benchmark of backdoor learning called
BackdoorBench. Our benchmark makes three valuable contributions to the research
community. 1) We provide an integrated implementation of state-of-the-art
(SOTA) backdoor learning algorithms (currently including 16 attack and 27
defense algorithms), based on an extensible modular-based codebase. 2) We
conduct comprehensive evaluations of 12 attacks against 16 defenses, with 5
poisoning ratios, based on 4 models and 4 datasets, thus 11,492 pairs of
evaluations in total. 3) Based on above evaluations, we present abundant
analysis from 8 perspectives via 18 useful analysis tools, and provide several
inspiring insights about backdoor learning. We hope that our efforts could
build a solid foundation of backdoor learning to facilitate researchers to
investigate existing algorithms, develop more innovative algorithms, and
explore the intrinsic mechanism of backdoor learning. Finally, we have created
a user-friendly website at http://backdoorbench.com, which collects all
important information of BackdoorBench, including codebase, docs, leaderboard,
and model Zoo.