FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals
  in Factorized Orthogonal Latent Space
Shengzhong Liu Tomoyoshi Kimura Dongxin Liu Ruijie Wang Jinyang Li Suhas Diggavi Mani Srivastava Tarek Abdelzaher
This paper proposes a novel contrastive learning framework, called FOCAL, for
extracting comprehensive features from multimodal time-series sensing signals
through self-supervised training. Existing multimodal contrastive frameworks
mostly rely on the shared information between sensory modalities, but do not
explicitly consider the exclusive modality information that could be critical
to understanding the underlying sensing physics. Besides, contrastive
frameworks for time series have not handled the temporal information locality
appropriately. FOCAL solves these challenges by making the following
contributions: First, given multimodal time series, it encodes each modality
into a factorized latent space consisting of shared features and private
features that are orthogonal to each other. The shared space emphasizes feature
patterns consistent across sensory modalities through a modal-matching
objective. In contrast, the private space extracts modality-exclusive
information through a transformation-invariant objective. Second, we propose a
temporal structural constraint for modality features, such that the average
distance between temporally neighboring samples is no larger than that of
temporally distant samples. Extensive evaluations are performed on four
multimodal sensing datasets with two backbone encoders and two classifiers to
demonstrate the superiority of FOCAL. It consistently outperforms the
state-of-the-art baselines in downstream tasks with a clear margin, under
different ratios of available labels. The code and self-collected dataset are
available at https://github.com/tomoyoshki/focal.