Action-conditioned video data improves predictability
Meenakshi Sarkar Debasish Ghose
Long-term video generation and prediction remain challenging tasks in
computer vision, particularly in partially observable scenarios where cameras
are mounted on moving platforms. The interaction between observed image frames
and the motion of the recording agent introduces additional complexities. To
address these issues, we introduce the Action-Conditioned Video Generation
(ACVG) framework, a novel approach that investigates the relationship between
actions and generated image frames through a deep dual Generator-Actor
architecture. ACVG generates video sequences conditioned on the actions of
robots, enabling exploration and analysis of how vision and action mutually
influence one another in dynamic environments. We evaluate the framework's
effectiveness on an indoor robot motion dataset which consists of sequences of
image frames along with the sequences of actions taken by the robotic agent,
conducting a comprehensive empirical study comparing ACVG to other
state-of-the-art frameworks along with a detailed ablation study.