LEAP: LLM-Generation of Egocentric Action Programs
Eadom Dessalene Michael Maynord Cornelia Fermüller Yiannis Aloimonos
We introduce LEAP (illustrated in Figure 1), a novel method for generating
video-grounded action programs through use of a Large Language Model (LLM).
These action programs represent the motoric, perceptual, and structural aspects
of action, and consist of sub-actions, pre- and post-conditions, and control
flows. LEAP's action programs are centered on egocentric video and employ
recent developments in LLMs both as a source for program knowledge and as an
aggregator and assessor of multimodal video information. We apply LEAP over a
majority (87\%) of the training set of the EPIC Kitchens dataset, and release
the resulting action programs as a publicly available dataset here
(https://drive.google.com/drive/folders/1Cpkw_TI1IIxXdzor0pOXG3rWJWuKU5Ex?usp=drive_link).
We employ LEAP as a secondary source of supervision, using its action programs
in a loss term applied to action recognition and anticipation networks. We
demonstrate sizable improvements in performance in both tasks due to training
with the LEAP dataset. Our method achieves 1st place on the EPIC Kitchens
Action Recognition leaderboard as of November 17 among the networks restricted
to RGB-input (see Supplementary Materials).