Domain Generalisation with Bidirectional Encoder Representations from
  Vision Transformers
Hamza Riaz Alan F. Smeaton
Domain generalisation involves pooling knowledge from source domain(s) into a
single model that can generalise to unseen target domain(s). Recent research in
domain generalisation has faced challenges when using deep learning models as
they interact with data distributions which differ from those they are trained
on. Here we perform domain generalisation on out-of-distribution (OOD) vision
benchmarks using vision transformers. Initially we examine four vision
transformer architectures namely ViT, LeViT, DeiT, and BEIT on
out-of-distribution data. As the bidirectional encoder representation from
image transformers (BEIT) architecture performs best, we use it in further
experiments on three benchmarks PACS, Home-Office and DomainNet. Our results
show significant improvements in validation and test accuracy and our
implementation significantly overcomes gaps between within-distribution and OOD
data.