AD3: Implicit Action is the Key for World Models to Distinguish the
  Diverse Visual Distractors
Yucen Wang Shenghua Wan Le Gan Shuai Feng De-Chuan Zhan
Model-based methods have significantly contributed to distinguishing
task-irrelevant distractors for visual control. However, prior research has
primarily focused on heterogeneous distractors like noisy background videos,
leaving homogeneous distractors that closely resemble controllable agents
largely unexplored, which poses significant challenges to existing methods. To
tackle this problem, we propose Implicit Action Generator (IAG) to learn the
implicit actions of visual distractors, and present a new algorithm named
implicit Action-informed Diverse visual Distractors Distinguisher (AD3), that
leverages the action inferred by IAG to train separated world models. Implicit
actions effectively capture the behavior of background distractors, aiding in
distinguishing the task-irrelevant components, and the agent can optimize the
policy within the task-relevant state space. Our method achieves superior
performance on various visual control tasks featuring both heterogeneous and
homogeneous distractors. The indispensable role of implicit actions learned by
IAG is also empirically validated.