Architectural Neural Backdoors from First Principles
Harry Langford Ilia Shumailov Yiren Zhao Robert Mullins Nicolas Papernot
While previous research backdoored neural networks by changing their
parameters, recent work uncovered a more insidious threat: backdoors embedded
within the definition of the network's architecture. This involves injecting
common architectural components, such as activation functions and pooling
layers, to subtly introduce a backdoor behavior that persists even after (full
re-)training. However, the full scope and implications of architectural
backdoors have remained largely unexplored. Bober-Irizar et al. [2023]
introduced the first architectural backdoor; they showed how to create a
backdoor for a checkerboard pattern, but never explained how to target an
arbitrary trigger pattern of choice. In this work we construct an arbitrary
trigger detector which can be used to backdoor an architecture with no human
supervision. This leads us to revisit the concept of architecture backdoors and
taxonomise them, describing 12 distinct types. To gauge the difficulty of
detecting such backdoors, we conducted a user study, revealing that ML
developers can only identify suspicious components in common model definitions
as backdoors in 37% of cases, while they surprisingly preferred backdoored
models in 33% of cases. To contextualize these results, we find that language
models outperform humans at the detection of backdoors. Finally, we discuss
defenses against architectural backdoors, emphasizing the need for robust and
comprehensive strategies to safeguard the integrity of ML systems.