Minimalist and High-Quality Panoramic Imaging with PSF-aware
  Transformers
Qi Jiang Shaohua Gao Yao Gao Kailun Yang Zhonghua Yi Hao Shi Lei Sun Kaiwei Wang
High-quality panoramic images with a Field of View (FoV) of 360-degree are
essential for contemporary panoramic computer vision tasks. However,
conventional imaging systems come with sophisticated lens designs and heavy
optical components. This disqualifies their usage in many mobile and wearable
applications where thin and portable, minimalist imaging systems are desired.
In this paper, we propose a Panoramic Computational Imaging Engine (PCIE) to
address minimalist and high-quality panoramic imaging. With less than three
spherical lenses, a Minimalist Panoramic Imaging Prototype (MPIP) is
constructed based on the design of the Panoramic Annular Lens (PAL), but with
low-quality imaging results due to aberrations and small image plane size. We
propose two pipelines, i.e. Aberration Correction (AC) and Super-Resolution and
Aberration Correction (SR&AC), to solve the image quality problems of MPIP,
with imaging sensors of small and large pixel size, respectively. To provide a
universal network for the two pipelines, we leverage the information from the
Point Spread Function (PSF) of the optical system and design a PSF-aware
Aberration-image Recovery Transformer (PART), in which the self-attention
calculation and feature extraction are guided via PSF-aware mechanisms. We
train PART on synthetic image pairs from simulation and put forward the PALHQ
dataset to fill the gap of real-world high-quality PAL images for low-level
vision. A comprehensive variety of experiments on synthetic and real-world
benchmarks demonstrates the impressive imaging results of PCIE and the
effectiveness of plug-and-play PSF-aware mechanisms. We further deliver
heuristic experimental findings for minimalist and high-quality panoramic
imaging. Our dataset and code will be available at
https://github.com/zju-jiangqi/PCIE-PART.