X-MIC: Cross-Modal Instance Conditioning for Egocentric Action
  Generalization
Anna Kukleva Fadime Sener Edoardo Remelli Bugra Tekin Eric Sauser Bernt Schiele Shugao Ma
Lately, there has been growing interest in adapting vision-language models
(VLMs) to image and third-person video classification due to their success in
zero-shot recognition. However, the adaptation of these models to egocentric
videos has been largely unexplored. To address this gap, we propose a simple
yet effective cross-modal adaptation framework, which we call X-MIC. Using a
video adapter, our pipeline learns to align frozen text embeddings to each
egocentric video directly in the shared embedding space. Our novel adapter
architecture retains and improves generalization of the pre-trained VLMs by
disentangling learnable temporal modeling and frozen visual encoder. This
results in an enhanced alignment of text embeddings to each egocentric video,
leading to a significant improvement in cross-dataset generalization. We
evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for
fine-grained cross-dataset action generalization, demonstrating the
effectiveness of our method. Code is available at
https://github.com/annusha/xmic