Sub-token ViT Embedding via Stochastic Resonance Transformers
Dong Lao Yangchao Wu Tian Yu Liu Alex Wong Stefano Soatto
We discover the presence of quantization artifacts in Vision Transformers
(ViTs), which arise due to the image tokenization step inherent in these
architectures. These artifacts result in coarsely quantized features, which
negatively impact performance, especially on downstream dense prediction tasks.
We present a zero-shot method to improve how pre-trained ViTs handle spatial
quantization. In particular, we propose to ensemble the features obtained from
perturbing input images via sub-token spatial translations, inspired by
Stochastic Resonance, a method traditionally applied to climate dynamics and
signal processing. We term our method ``Stochastic Resonance Transformer"
(SRT), which we show can effectively super-resolve features of pre-trained
ViTs, capturing more of the local fine-grained structures that might otherwise
be neglected as a result of tokenization. SRT can be applied at any layer, on
any task, and does not require any fine-tuning. The advantage of the former is
evident when applied to monocular depth prediction, where we show that
ensembling model outputs are detrimental while applying SRT on intermediate ViT
features outperforms the baseline models by an average of 4.7% and 14.9% on the
RMSE and RMSE-log metrics across three different architectures. When applied to
semi-supervised video object segmentation, SRT also improves over the baseline
models uniformly across all metrics, and by an average of 2.4% in F&J score. We
further show that these quantization artifacts can be attenuated to some extent
via self-distillation. On the unsupervised salient region segmentation, SRT
improves upon the base model by an average of 2.1% on the maxF metric. Finally,
despite operating purely on pixel-level features, SRT generalizes to non-dense
prediction tasks such as image retrieval and object discovery, yielding
consistent improvements of up to 2.6% and 1.0% respectively.