EgoAdapt: A multi-stream evaluation study of adaptation to real-world
  egocentric user video
Matthias De Lange Hamid Eghbalzadeh Reuben Tan Michael Iuzzolino Franziska Meier Karl Ridgeway
In egocentric action recognition a single population model is typically
trained and subsequently embodied on a head-mounted device, such as an
augmented reality headset. While this model remains static for new users and
environments, we introduce an adaptive paradigm of two phases, where after
pretraining a population model, the model adapts on-device and online to the
user's experience. This setting is highly challenging due to the change from
population to user domain and the distribution shifts in the user's data
stream. Coping with the latter in-stream distribution shifts is the focus of
continual learning, where progress has been rooted in controlled benchmarks but
challenges faced in real-world applications often remain unaddressed. We
introduce EgoAdapt, a benchmark for real-world egocentric action recognition
that facilitates our two-phased adaptive paradigm, and real-world challenges
naturally occur in the egocentric video streams from Ego4d, such as long-tailed
action distributions and large-scale classification over 2740 actions. We
introduce an evaluation framework that directly exploits the user's data stream
with new metrics to measure the adaptation gain over the population model,
online generalization, and hindsight performance. In contrast to single-stream
evaluation in existing works, our framework proposes a meta-evaluation that
aggregates the results from 50 independent user streams. We provide an
extensive empirical study for finetuning and experience replay.