DIAR: Deep Image Alignment and Reconstruction using Swin Transformers
Monika Kwiatkowski Simon Matern Olaf Hellwich
When taking images of some occluded content, one is often faced with the
problem that every individual image frame contains unwanted artifacts, but a
collection of images contains all relevant information if properly aligned and
aggregated. In this paper, we attempt to build a deep learning pipeline that
simultaneously aligns a sequence of distorted images and reconstructs them. We
create a dataset that contains images with image distortions, such as lighting,
specularities, shadows, and occlusion. We create perspective distortions with
corresponding ground-truth homographies as labels. We use our dataset to train
Swin transformer models to analyze sequential image data. The attention maps
enable the model to detect relevant image content and differentiate it from
outliers and artifacts. We further explore using neural feature maps as
alternatives to classical key point detectors. The feature maps of trained
convolutional layers provide dense image descriptors that can be used to find
point correspondences between images. We utilize this to compute coarse image
alignments and explore its limitations.