Viewport Prediction for Volumetric Video Streaming by Exploring Video
  Saliency and Trajectory Information
Jie Li Zhixin Li Zhi Liu Pengyuan Zhou Richang Hong Qiyue Li Han Hu
Volumetric video, also known as hologram video, is a novel medium that
portrays natural content in Virtual Reality (VR), Augmented Reality (AR), and
Mixed Reality (MR). It is expected to be the next-gen video technology and a
prevalent use case for 5G and beyond wireless communication. Considering that
each user typically only watches a section of the volumetric video, known as
the viewport, it is essential to have precise viewport prediction for optimal
performance. However, research on this topic is still in its infancy. In the
end, this paper presents and proposes a novel approach, named Saliency and
Trajectory Viewport Prediction (STVP), which aims to improve the precision of
viewport prediction in volumetric video streaming. The STVP extensively
utilizes video saliency information and viewport trajectory. To our knowledge,
this is the first comprehensive study of viewport prediction in volumetric
video streaming. In particular, we introduce a novel sampling method, Uniform
Random Sampling (URS), to reduce computational complexity while still
preserving video features in an efficient manner. Then we present a saliency
detection technique that incorporates both spatial and temporal information for
detecting static, dynamic geometric, and color salient regions. Finally, we
intelligently fuse saliency and trajectory information to achieve more accurate
viewport prediction. We conduct extensive simulations to evaluate the
effectiveness of our proposed viewport prediction methods using
state-of-the-art volumetric video sequences. The experimental results show the
superiority of the proposed method over existing schemes. The dataset and
source code will be publicly accessible after acceptance.