Streaming egocentric action anticipation: An evaluation scheme and
  approach
Antonino Furnari Giovanni Maria Farinella
Egocentric action anticipation aims to predict the future actions the camera
wearer will perform from the observation of the past. While predictions about
the future should be available before the predicted events take place, most
approaches do not pay attention to the computational time required to make such
predictions. As a result, current evaluation schemes assume that predictions
are available right after the input video is observed, i.e., presuming a
negligible runtime, which may lead to overly optimistic evaluations. We propose
a streaming egocentric action evaluation scheme which assumes that predictions
are performed online and made available only after the model has processed the
current input segment, which depends on its runtime. To evaluate all models
considering the same prediction horizon, we hence propose that slower models
should base their predictions on temporal segments sampled ahead of time. Based
on the observation that model runtime can affect performance in the considered
streaming evaluation scenario, we further propose a lightweight action
anticipation model based on feed-forward 3D CNNs which is optimized using
knowledge distillation techniques with a novel past-to-future distillation
loss. Experiments on the three popular datasets EPIC-KITCHENS-55,
EPIC-KITCHENS-100 and EGTEA Gaze+ show that (i) the proposed evaluation scheme
induces a different ranking on state-of-the-art methods as compared to classic
evaluations, (ii) lightweight approaches tend to outmatch more computationally
expensive ones, and (iii) the proposed model based on feed-forward 3D CNNs and
knowledge distillation outperforms current art in the streaming egocentric
action anticipation scenario.