Cross-Domain Transfer Learning with CoRTe: Consistent and Reliable
  Transfer from Black-Box to Lightweight Segmentation Model
Claudia Cuttano Antonio Tavera Fabio Cermelli Giuseppe Averta Barbara Caputo
Many practical applications require training of semantic segmentation models
on unlabelled datasets and their execution on low-resource hardware.
Distillation from a trained source model may represent a solution for the first
but does not account for the different distribution of the training data.
Unsupervised domain adaptation (UDA) techniques claim to solve the domain
shift, but in most cases assume the availability of the source data or an
accessible white-box source model, which in practical applications are often
unavailable for commercial and/or safety reasons. In this paper, we investigate
a more challenging setting in which a lightweight model has to be trained on a
target unlabelled dataset for semantic segmentation, under the assumption that
we have access only to black-box source model predictions. Our method, named
CoRTe, consists of (i) a pseudo-labelling function that extracts reliable
knowledge from the black-box source model using its relative confidence, (ii) a
pseudo label refinement method to retain and enhance the novel information
learned by the student model on the target data, and (iii) a consistent
training of the model using the extracted pseudo labels. We benchmark CoRTe on
two synthetic-to-real settings, demonstrating remarkable results when using
black-box models to transfer knowledge on lightweight models for a target data
distribution.