PanoSwin: a Pano-style Swin Transformer for Panorama Understanding
Zhixin Ling Zhen Xing Xiangdong Zhou Manliang Cao Guichun Zhou
In panorama understanding, the widely used equirectangular projection (ERP)
entails boundary discontinuity and spatial distortion. It severely deteriorates
the conventional CNNs and vision Transformers on panoramas. In this paper, we
propose a simple yet effective architecture named PanoSwin to learn panorama
representations with ERP. To deal with the challenges brought by
equirectangular projection, we explore a pano-style shift windowing scheme and
novel pitch attention to address the boundary discontinuity and the spatial
distortion, respectively. Besides, based on spherical distance and Cartesian
coordinates, we adapt absolute positional embeddings and relative positional
biases for panoramas to enhance panoramic geometry information. Realizing that
planar image understanding might share some common knowledge with panorama
understanding, we devise a novel two-stage learning framework to facilitate
knowledge transfer from the planar images to panoramas. We conduct experiments
against the state-of-the-art on various panoramic tasks, i.e., panoramic object
detection, panoramic classification, and panoramic layout estimation. The
experimental results demonstrate the effectiveness of PanoSwin in panorama
understanding.