TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint
  Prediction in Aerial Scenes
Liangyu Xu Wanxuan Lu Hongfeng Yu Yongqiang Mao Hanbo Bi Chenglong Liu Xian Sun Kun Fu
As drone technology advances, using unmanned aerial vehicles for aerial
surveys has become the dominant trend in modern low-altitude remote sensing.
The surge in aerial video data necessitates accurate prediction for future
scenarios and motion states of the interested target, particularly in
applications like traffic management and disaster response. Existing video
prediction methods focus solely on predicting future scenes (video frames),
suffering from the neglect of explicitly modeling target's motion states, which
is crucial for aerial video interpretation. To address this issue, we introduce
a novel task called Target-Aware Aerial Video Prediction, aiming to
simultaneously predict future scenes and motion states of the target. Further,
we design a model specifically for this task, named TAFormer, which provides a
unified modeling approach for both video and target motion states.
Specifically, we introduce Spatiotemporal Attention (STA), which decouples the
learning of video dynamics into spatial static attention and temporal dynamic
attention, effectively modeling the scene appearance and motion. Additionally,
we design an Information Sharing Mechanism (ISM), which elegantly unifies the
modeling of video and target motion by facilitating information interaction
through two sets of messenger tokens. Moreover, to alleviate the difficulty of
distinguishing targets in blurry predictions, we introduce Target-Sensitive
Gaussian Loss (TSGL), enhancing the model's sensitivity to both target's
position and content. Extensive experiments on UAV123VP and VisDroneVP (derived
from single-object tracking datasets) demonstrate the exceptional performance
of TAFormer in target-aware video prediction, showcasing its adaptability to
the additional requirements of aerial video interpretation for target
awareness.