Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
Adam Pardyl Grzegorz Kurzejamski Jan Olszewski Tomasz Trzciński Bartosz Zieliński
Vision transformers have excelled in various computer vision tasks but mostly
rely on rigid input sampling using a fixed-size grid of patches. This limits
their applicability in real-world problems, such as in the field of robotics
and UAVs, where one can utilize higher input elasticity to boost model
performance and efficiency. Our paper addresses this limitation by formalizing
the concept of input elasticity for vision transformers and introducing an
evaluation protocol, including dedicated metrics for measuring input
elasticity. Moreover, we propose modifications to the transformer architecture
and training regime, which increase its elasticity. Through extensive
experimentation, we spotlight opportunities and challenges associated with
input sampling strategies.