SeMAnD: Self-Supervised Anomaly Detection in Multimodal Geospatial
  Datasets
Daria Reshetova Swetava Ganguli C. V. Krishnakumar Iyer Vipul Pandey
We propose a Self-supervised Anomaly Detection technique, called SeMAnD, to
detect geometric anomalies in Multimodal geospatial datasets. Geospatial data
comprises of acquired and derived heterogeneous data modalities that we
transform to semantically meaningful, image-like tensors to address the
challenges of representation, alignment, and fusion of multimodal data. SeMAnD
is comprised of (i) a simple data augmentation strategy, called
RandPolyAugment, capable of generating diverse augmentations of vector
geometries, and (ii) a self-supervised training objective with three components
that incentivize learning representations of multimodal data that are
discriminative to local changes in one modality which are not corroborated by
the other modalities. Detecting local defects is crucial for geospatial anomaly
detection where even small anomalies (e.g., shifted, incorrectly connected,
malformed, or missing polygonal vector geometries like roads, buildings,
landcover, etc.) are detrimental to the experience and safety of users of
geospatial applications like mapping, routing, search, and recommendation
systems. Our empirical study on test sets of different types of real-world
geometric geospatial anomalies across 3 diverse geographical regions
demonstrates that SeMAnD is able to detect real-world defects and outperforms
domain-agnostic anomaly detection strategies by 4.8-19.7% as measured using
anomaly classification AUC. We also show that model performance increases (i)
up to 20.4% as the number of input modalities increase and (ii) up to 22.9% as
the diversity and strength of training data augmentations increase.