Layout Sequence Prediction From Noisy Mobile Modality
Haichao Zhang Yi Xu Hongsheng Lu Takayuki Shimizu Yun Fu
Trajectory prediction plays a vital role in understanding pedestrian movement
for applications such as autonomous driving and robotics. Current trajectory
prediction models depend on long, complete, and accurately observed sequences
from visual modalities. Nevertheless, real-world situations often involve
obstructed cameras, missed objects, or objects out of sight due to
environmental factors, leading to incomplete or noisy trajectories. To overcome
these limitations, we propose LTrajDiff, a novel approach that treats objects
obstructed or out of sight as equally important as those with fully visible
trajectories. LTrajDiff utilizes sensor data from mobile phones to surmount
out-of-sight constraints, albeit introducing new challenges such as modality
fusion, noisy data, and the absence of spatial layout and object size
information. We employ a denoising diffusion model to predict precise layout
sequences from noisy mobile data using a coarse-to-fine diffusion strategy,
incorporating the RMS, Siamese Masked Encoding Module, and MFM. Our model
predicts layout sequences by implicitly inferring object size and projection
status from a single reference timestamp or significantly obstructed sequences.
Achieving SOTA results in randomly obstructed experiments and extremely short
input experiments, our model illustrates the effectiveness of leveraging noisy
mobile data. In summary, our approach offers a promising solution to the
challenges faced by layout sequence and trajectory prediction models in
real-world settings, paving the way for utilizing sensor data from mobile
phones to accurately predict pedestrian bounding box trajectories. To the best
of our knowledge, this is the first work that addresses severely obstructed and
extremely short layout sequences by combining vision with noisy mobile
modality, making it the pioneering work in the field of layout sequence
trajectory prediction.