Self-Supervised Masked Digital Elevation Models Encoding for
  Low-Resource Downstream Tasks
Priyam Mazumdar Aiman Soliman Volodymyr Kindratenko Luigi Marini Kenton McHenry
The lack of quality labeled data is one of the main bottlenecks for training
Deep Learning models. As the task increases in complexity, there is a higher
penalty for overfitting and unstable learning. The typical paradigm employed
today is Self-Supervised learning, where the model attempts to learn from a
large corpus of unstructured and unlabeled data and then transfer that
knowledge to the required task. Some notable examples of self-supervision in
other modalities are BERT for Large Language Models, Wav2Vec for Speech
Recognition, and the Masked AutoEncoder for Vision, which all utilize
Transformers to solve a masked prediction task. GeoAI is uniquely poised to
take advantage of the self-supervised methodology due to the decades of data
collected, little of which is precisely and dependably annotated. Our goal is
to extract building and road segmentations from Digital Elevation Models (DEM)
that provide a detailed topography of the earths surface. The proposed
architecture is the Masked Autoencoder pre-trained on ImageNet (with the
limitation that there is a large domain discrepancy between ImageNet and DEM)
with an UperNet Head for decoding segmentations. We tested this model with 450
and 50 training images only, utilizing roughly 5% and 0.5% of the original data
respectively. On the building segmentation task, this model obtains an 82.1%
Intersection over Union (IoU) with 450 Images and 69.1% IoU with only 50
images. On the more challenging road detection task the model obtains an 82.7%
IoU with 450 images and 73.2% IoU with only 50 images. Any hand-labeled dataset
made today about the earths surface will be immediately obsolete due to the
constantly changing nature of the landscape. This motivates the clear necessity
for data-efficient learners that can be used for a wide variety of downstream
tasks.