Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing
  Trimodal Data
Christian Stippel Thomas Heitzinger Rafael Sterzinger Martin Kampel
In pervasive machine learning, especially in Human Behavior Analysis (HBA),
RGB has been the primary modality due to its accessibility and richness of
information. However, linked with its benefits are challenges, including
sensitivity to lighting conditions and privacy concerns. One possibility to
overcome these vulnerabilities is to resort to different modalities. For
instance, thermal is particularly adept at accentuating human forms, while
depth adds crucial contextual layers. Despite their known benefits, only a few
HBA-specific datasets that integrate these modalities exist. To address this
shortage, our research introduces a novel generative technique for creating
trimodal, i.e., RGB, thermal, and depth, human-focused datasets. This technique
capitalizes on human segmentation masks derived from RGB images, combined with
thermal and depth backgrounds that are sourced automatically. With these two
ingredients, we synthesize depth and thermal counterparts from existing RGB
data utilizing conditional image-to-image translation. By employing this
approach, we generate trimodal data that can be leveraged to train models for
settings with limited data, bad lightning conditions, or privacy-sensitive
areas.