Serpent: Scalable and Efficient Image Restoration via Multi-scale
  Structured State Space Models
Mohammad Shahab Sepehri Zalan Fabian Mahdi Soltanolkotabi
The landscape of computational building blocks of efficient image restoration
architectures is dominated by a combination of convolutional processing and
various attention mechanisms. However, convolutional filters are inherently
local and therefore struggle at modeling long-range dependencies in images. On
the other hand, attention excels at capturing global interactions between
arbitrary image regions, however at a quadratic cost in image dimension. In
this work, we propose Serpent, an architecture that leverages recent advances
in state space models (SSMs) in its core computational block. SSMs, originally
introduced for sequence modeling, can maintain a global receptive field with a
favorable linear scaling in input size. Our preliminary results demonstrate
that Serpent can achieve reconstruction quality on par with state-of-the-art
techniques, while requiring orders of magnitude less compute (up to $150$ fold
reduction in FLOPS) and a factor of up to $5\times$ less GPU memory while
maintaining a compact model size.