Dual Student Networks for Data-Free Model Stealing
James Beetham Navid Kardan Ajmal Mian Mubarak Shah
Existing data-free model stealing methods use a generator to produce samples
in order to train a student model to match the target model outputs. To this
end, the two main challenges are estimating gradients of the target model
without access to its parameters, and generating a diverse set of training
samples that thoroughly explores the input space. We propose a Dual Student
method where two students are symmetrically trained in order to provide the
generator a criterion to generate samples that the two students disagree on. On
one hand, disagreement on a sample implies at least one student has classified
the sample incorrectly when compared to the target model. This incentive
towards disagreement implicitly encourages the generator to explore more
diverse regions of the input space. On the other hand, our method utilizes
gradients of student models to indirectly estimate gradients of the target
model. We show that this novel training objective for the generator network is
equivalent to optimizing a lower bound on the generator's loss if we had access
to the target model gradients. We show that our new optimization framework
provides more accurate gradient estimation of the target model and better
accuracies on benchmark classification datasets. Additionally, our approach
balances improved query efficiency with training computation cost. Finally, we
demonstrate that our method serves as a better proxy model for transfer-based
adversarial attacks than existing data-free model stealing methods.