GLIMPSE: Generalized Local Imaging with MLPs
AmirEhsan Khorashadizadeh Valentin Debarnot Tianlin Liu Ivan Dokmanić
Deep learning is the current de facto state of the art in tomographic
imaging. A common approach is to feed the result of a simple inversion, for
example the backprojection, to a convolutional neural network (CNN) which then
computes the reconstruction. Despite strong results on 'in-distribution' test
data similar to the training data, backprojection from sparse-view data
delocalizes singularities, so these approaches require a large receptive field
to perform well. As a consequence, they overfit to certain global structures
which leads to poor generalization on out-of-distribution (OOD) samples.
Moreover, their memory complexity and training time scale unfavorably with
image resolution, making them impractical for application at realistic clinical
resolutions, especially in 3D: a standard U-Net requires a substantial 140GB of
memory and 2600 seconds per epoch on a research-grade GPU when training on
1024x1024 images. In this paper, we introduce GLIMPSE, a local processing
neural network for computed tomography which reconstructs a pixel value by
feeding only the measurements associated with the neighborhood of the pixel to
a simple MLP. While achieving comparable or better performance with successful
CNNs like the U-Net on in-distribution test data, GLIMPSE significantly
outperforms them on OOD samples while maintaining a memory footprint almost
independent of image resolution; 5GB memory suffices to train on 1024x1024
images. Further, we built GLIMPSE to be fully differentiable, which enables
feats such as recovery of accurate projection angles if they are out of
calibration.