HSViT: Horizontally Scalable Vision Transformer
Chenhao Xu Chang-Tsun Li Chee Peng Lim Douglas Creighton
While the Vision Transformer (ViT) architecture gains prominence in computer
vision and attracts significant attention from multimedia communities, its
deficiency in prior knowledge (inductive bias) regarding shift, scale, and
rotational invariance necessitates pre-training on large-scale datasets.
Furthermore, the growing layers and parameters in both ViT and convolutional
neural networks (CNNs) impede their applicability to mobile multimedia
services, primarily owing to the constrained computational resources on edge
devices. To mitigate the aforementioned challenges, this paper introduces a
novel horizontally scalable vision transformer (HSViT). Specifically, a novel
image-level feature embedding allows ViT to better leverage the inductive bias
inherent in the convolutional layers. Based on this, an innovative horizontally
scalable architecture is designed, which reduces the number of layers and
parameters of the models while facilitating collaborative training and
inference of ViT models across multiple nodes. The experimental results depict
that, without pre-training on large-scale datasets, HSViT achieves up to 10%
higher top-1 accuracy than state-of-the-art schemes, ascertaining its superior
preservation of inductive bias. The code is available at
https://github.com/xuchenhao001/HSViT.