Guided Masked Self-Distillation Modeling for Distributed Multimedia
  Sensor Event Analysis
Masahiro Yasuda Noboru Harada Yasunori Ohishi Shoichiro Saito Akira Nakayama Nobutaka Ono
Observations with distributed sensors are essential in analyzing a series of
human and machine activities (referred to as 'events' in this paper) in complex
and extensive real-world environments. This is because the information obtained
from a single sensor is often missing or fragmented in such an environment;
observations from multiple locations and modalities should be integrated to
analyze events comprehensively. However, a learning method has yet to be
established to extract joint representations that effectively combine such
distributed observations. Therefore, we propose Guided Masked sELf-Distillation
modeling (Guided-MELD) for inter-sensor relationship modeling. The basic idea
of Guided-MELD is to learn to supplement the information from the masked sensor
with information from other sensors needed to detect the event. Guided-MELD is
expected to enable the system to effectively distill the fragmented or
redundant target event information obtained by the sensors without being overly
dependent on any specific sensors. To validate the effectiveness of the
proposed method in novel tasks of distributed multimedia sensor event analysis,
we recorded two new datasets that fit the problem setting: MM-Store and
MM-Office. These datasets consist of human activities in a convenience store
and an office, recorded using distributed cameras and microphones. Experimental
results on these datasets show that the proposed Guided-MELD improves event
tagging and detection performance and outperforms conventional inter-sensor
relationship modeling methods. Furthermore, the proposed method performed
robustly even when sensors were reduced.