A Study on Differentiable Logic and LLMs for EPIC-KITCHENS-100
  Unsupervised Domain Adaptation Challenge for Action Recognition 2023
Yi Cheng Ziwei Xu Fen Fang Dongyun Lin Hehe Fan Yongkang Wong Ying Sun Mohan Kankanhalli
In this technical report, we present our findings from a study conducted on
the EPIC-KITCHENS-100 Unsupervised Domain Adaptation task for Action
Recognition. Our research focuses on the innovative application of a
differentiable logic loss in the training to leverage the co-occurrence
relations between verb and noun, as well as the pre-trained Large Language
Models (LLMs) to generate the logic rules for the adaptation to unseen action
labels. Specifically, the model's predictions are treated as the truth
assignment of a co-occurrence logic formula to compute the logic loss, which
measures the consistency between the predictions and the logic constraints. By
using the verb-noun co-occurrence matrix generated from the dataset, we observe
a moderate improvement in model performance compared to our baseline framework.
To further enhance the model's adaptability to novel action labels, we
experiment with rules generated using GPT-3.5, which leads to a slight decrease
in performance. These findings shed light on the potential and challenges of
incorporating differentiable logic and LLMs for knowledge extraction in
unsupervised domain adaptation for action recognition. Our final submission
(entitled `NS-LLM') achieved the first place in terms of top-1 action
recognition accuracy.