Dynamic Neural Network is All You Need: Understanding the Robustness of
  Dynamic Mechanisms in Neural Networks
Mirazul Haque Wei Yang
Deep Neural Networks (DNNs) have been used to solve different day-to-day
problems. Recently, DNNs have been deployed in real-time systems, and lowering
the energy consumption and response time has become the need of the hour. To
address this scenario, researchers have proposed incorporating dynamic
mechanism to static DNNs (SDNN) to create Dynamic Neural Networks (DyNNs)
performing dynamic amounts of computation based on the input complexity.
Although incorporating dynamic mechanism into SDNNs would be preferable in
real-time systems, it also becomes important to evaluate how the introduction
of dynamic mechanism impacts the robustness of the models. However, there has
not been a significant number of works focusing on the robustness trade-off
between SDNNs and DyNNs. To address this issue, we propose to investigate the
robustness of dynamic mechanism in DyNNs and how dynamic mechanism design
impacts the robustness of DyNNs. For that purpose, we evaluate three research
questions. These evaluations are performed on three models and two datasets.
Through the studies, we find that attack transferability from DyNNs to SDNNs is
higher than attack transferability from SDNNs to DyNNs. Also, we find that
DyNNs can be used to generate adversarial samples more efficiently than SDNNs.
Then, through research studies, we provide insight into the design choices that
can increase robustness of DyNNs against the attack generated using static
model. Finally, we propose a novel attack to understand the additional attack
surface introduced by the dynamic mechanism and provide design choices to
improve robustness against the attack.