Remembering What Is Important: A Factorised Multi-Head Retrieval and
  Auxiliary Memory Stabilisation Scheme for Human Motion Prediction
Tharindu Fernando Harshala Gammulle Sridha Sridharan Simon Denman Clinton Fookes
Humans exhibit complex motions that vary depending on the task that they are
performing, the interactions they engage in, as well as subject-specific
preferences. Therefore, forecasting future poses based on the history of the
previous motions is a challenging task. This paper presents an innovative
auxiliary-memory-powered deep neural network framework for the improved
modelling of historical knowledge. Specifically, we disentangle
subject-specific, task-specific, and other auxiliary information from the
observed pose sequences and utilise these factorised features to query the
memory. A novel Multi-Head knowledge retrieval scheme leverages these
factorised feature embeddings to perform multiple querying operations over the
historical observations captured within the auxiliary memory. Moreover, our
proposed dynamic masking strategy makes this feature disentanglement process
dynamic. Two novel loss functions are introduced to encourage diversity within
the auxiliary memory while ensuring the stability of the memory contents, such
that it can locate and store salient information that can aid the long-term
prediction of future motion, irrespective of data imbalances or the diversity
of the input data distribution. With extensive experiments conducted on two
public benchmarks, Human3.6M and CMU-Mocap, we demonstrate that these design
choices collectively allow the proposed approach to outperform the current
state-of-the-art methods by significant margins: $>$ 17\% on the Human3.6M
dataset and $>$ 9\% on the CMU-Mocap dataset.