HartleyMHA: Self-Attention in Frequency Domain for Resolution-Robust and
  Parameter-Efficient 3D Image Segmentation
Ken C. L. Wong Hongzhi Wang Tanveer Syeda-Mahmood
With the introduction of Transformers, different attention-based models have
been proposed for image segmentation with promising results. Although
self-attention allows capturing of long-range dependencies, it suffers from a
quadratic complexity in the image size especially in 3D. To avoid the
out-of-memory error during training, input size reduction is usually required
for 3D segmentation, but the accuracy can be suboptimal when the trained models
are applied on the original image size. To address this limitation, inspired by
the Fourier neural operator (FNO), we introduce the HartleyMHA model which is
robust to training image resolution with efficient self-attention. FNO is a
deep learning framework for learning mappings between functions in partial
differential equations, which has the appealing properties of zero-shot
super-resolution and global receptive field. We modify the FNO by using the
Hartley transform with shared parameters to reduce the model size by orders of
magnitude, and this allows us to further apply self-attention in the frequency
domain for more expressive high-order feature combination with improved
efficiency. When tested on the BraTS'19 dataset, it achieved superior
robustness to training image resolution than other tested models with less than
1% of their model parameters.