Transformers for CT Reconstruction From Monoplanar and Biplanar
  Radiographs
Firas Khader Gustav Müller-Franzes Tianyu Han Sven Nebelung Christiane Kuhl Johannes Stegmaier Daniel Truhn
Computed Tomography (CT) scans provide detailed and accurate information of
internal structures in the body. They are constructed by sending x-rays through
the body from different directions and combining this information into a
three-dimensional volume. Such volumes can then be used to diagnose a wide
range of conditions and allow for volumetric measurements of organs. In this
work, we tackle the problem of reconstructing CT images from biplanar x-rays
only. X-rays are widely available and even if the CT reconstructed from these
radiographs is not a replacement of a complete CT in the diagnostic setting, it
might serve to spare the patients from radiation where a CT is only acquired
for rough measurements such as determining organ size. We propose a novel
method based on the transformer architecture, by framing the underlying task as
a language translation problem. Radiographs and CT images are first embedded
into latent quantized codebook vectors using two different autoencoder
networks. We then train a GPT model, to reconstruct the codebook vectors of the
CT image, conditioned on the codebook vectors of the x-rays and show that this
approach leads to realistic looking images. To encourage further research in
this direction, we make our code publicly available on GitHub: XXX.