Decomposer: Semi-supervised Learning of Image Restoration and Image
  Decomposition
Boris Meinardus Mariusz Trzeciakiewicz Tim Herzig Monika Kwiatkowski Simon Matern Olaf Hellwich
We present Decomposer, a semi-supervised reconstruction model that decomposes
distorted image sequences into their fundamental building blocks - the original
image and the applied augmentations, i.e., shadow, light, and occlusions. To
solve this problem, we use the SIDAR dataset that provides a large number of
distorted image sequences: each sequence contains images with shadows,
lighting, and occlusions applied to an undistorted version. Each distortion
changes the original signal in different ways, e.g., additive or multiplicative
noise. We propose a transformer-based model to explicitly learn this
decomposition. The sequential model uses 3D Swin-Transformers for
spatio-temporal encoding and 3D U-Nets as prediction heads for individual parts
of the decomposition. We demonstrate that by separately pre-training our model
on weakly supervised pseudo labels, we can steer our model to optimize for our
ambiguous problem definition and learn to differentiate between the different
image distortions.