Leveraging Normalization Layer in Adapters With Progressive Learning and
  Adaptive Distillation for Cross-Domain Few-Shot Learning
Yongjin Yang Taehyeon Kim Se-Young Yun
Cross-domain few-shot learning presents a formidable challenge, as models
must be trained on base classes and then tested on novel classes from various
domains with only a few samples at hand. While prior approaches have primarily
focused on parameter-efficient methods of using adapters, they often overlook
two critical issues: shifts in batch statistics and noisy sample statistics
arising from domain discrepancy variations. In this paper, we introduce a novel
generic framework that leverages normalization layer in adapters with
Progressive Learning and Adaptive Distillation (ProLAD), marking two principal
contributions. First, our methodology utilizes two separate adapters: one
devoid of a normalization layer, which is more effective for similar domains,
and another embedded with a normalization layer, designed to leverage the batch
statistics of the target domain, thus proving effective for dissimilar domains.
Second, to address the pitfalls of noisy statistics, we deploy two strategies:
a progressive training of the two adapters and an adaptive distillation
technique derived from features determined by the model solely with the adapter
devoid of a normalization layer. Through this adaptive distillation, our
approach functions as a modulator, controlling the primary adapter for
adaptation, based on each domain. Evaluations on standard cross-domain few-shot
learning benchmarks confirm that our technique outperforms existing
state-of-the-art methodologies.