A Unified Membership Inference Method for Visual Self-supervised Encoder
  via Part-aware Capability
Jie Zhu Jirong Zha Ding Li Leye Wang
Self-supervised learning shows promise in harnessing extensive unlabeled
data, but it also confronts significant privacy concerns, especially in vision.
In this paper, we aim to perform membership inference on visual self-supervised
models in a more realistic setting: self-supervised training method and details
are unknown for an adversary when attacking as he usually faces a black-box
system in practice. In this setting, considering that self-supervised model
could be trained by completely different self-supervised paradigms, e.g.,
masked image modeling and contrastive learning, with complex training details,
we propose a unified membership inference method called PartCrop. It is
motivated by the shared part-aware capability among models and stronger part
response on the training data. Specifically, PartCrop crops parts of objects in
an image to query responses with the image in representation space. We conduct
extensive attacks on self-supervised models with different training protocols
and structures using three widely used image datasets. The results verify the
effectiveness and generalization of PartCrop. Moreover, to defend against
PartCrop, we evaluate two common approaches, i.e., early stop and differential
privacy, and propose a tailored method called shrinking crop scale range. The
defense experiments indicate that all of them are effective. Our code is
available at https://github.com/JiePKU/PartCrop