Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent’s action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels. Project website: https://daniilidis-group.github.io/learned_action_spaces Agents behaving in real-world environments rely on perception to judge what actions they can take and what effect these actions will have. Purely perceptual learning may play an important role in how these action representations are acquired and used. In this work, we focus on the problem of learning an agent's action space from unlabeled visual observations. To see the usefulness of this strategy, consider an infant that is first learning to walk. From around 10 months of age, infants rapidly progress from crawling, to irregular gaits with frequent falling, and finally to reliable locomotion (Adolph et al. (2012) ). But before they first attempt to walk, infants have extensive sensory exposure to adults walking. Unsupervised learning from sensory experience of this type appears to play a critical role in how humans acquire representations of actions before they can reliably reproduce the corresponding behaviour BID33 ). Infants need to relate the set of motor primitives they can generate to the action spaces exploited by adults (Dominici et al. (2011)) , and a representation acquired by observation may allow an infant to more efficiently learn to produce natural, goal-directed walking behavior.Reinforcement learning (RL) provides an alternative to the (passive) unsupervised learning approach as it implicitly discovers an agent's action space and the consequences of its actions. Recent breakthroughs in model-free and model-based RL suggest that end-to-end training can be used to learn mappings between sensory input and actions BID14 ; BID12 ; BID11 ; Finn & Levine (2017) ; BID25 ). However, these methods require active observations and the sensorimotor mappings learned in this way cannot be easily generalized to new agents with different control interfaces. Methods for sensorimotor learning from purely visual Using latent composition to recover actions from passive data. a) Two sequences starting from different initial states but changing according to the same actions. Without requiring labels, our model learns to represent the action in sequences like these identically. We train a representation z to capture the dynamics of the scene and its compositional structure: applying (z 1 and z 2 ) should have the same effect as applying the composed representation g(z 1 , z 2 ). These properties capture the fact that effector systems, such as a robot arm, use the same composable action space in many different states. b) The learned action space z recovered by our method (PCA visualization). Points are colored by the true action u: true actions can be easily decoded from z, validating that the structure of the action space has been captured.data may facilitate learning where action information is not available, such as when using video data collected from the Internet. Such methods may also be useful for imitation learning, where ground truth actions are often hard or impossible to collect other than by visual observation (Finn et al. (2017) ; BID18 ). More generally, learning from passive observations may make it easier to reuse action representations between systems with different effectors and goals. The representations learned by unsupervised methods are invariant to these choices because the model does not have access to motor commands or goals during training.In this work, we evaluate the proposal that learning what you can do before doing anything can lead to action space representations that make subsequent learning more efficient. To this end, we develop a model that learns to represent an agent's action space given only unlabeled videos of the agent. The resulting representation enables direct planning in the latent space. Given a small number of action-labeled sequences we can execute the plan by learning a simple mapping from latent action representations to the agent's controls. This representation may be analogous to those in the parietal and premotor areas of cortex, which include populations of neurons that represent the structure of actions produced both by the self and by others BID20 ; BID21 ) and that are critical for reliably producing flexible, voluntary motor control (see BID6 , Chapter 38). In the brain, representations of this kind could plausibly be learned using specialized loss functions BID13 ) whose effect is to induce the prior needed to determine the structure of actions in observation data.In contrast to most approaches to unsupervised learning of dynamics, which focus on learning the statistical structure of the environment, we focus on disentangling action information from the instantaneous state of the environment FIG0 . We base our work on recent stochastic video prediction methods (Babaeizadeh et al. (2018); Denton & Fergus (2018) ; BID10 ) and impose two properties on the latent representation. First, we train the representation to be minimal, i.e. containing minimal information about the current world state. This forces the representation to focus on dynamic properties of the sensory input. A similar objective has been used in previous work to constrain the capacity of video prediction models (Denton & Fergus (2018) ). Second, we train the representation to be composable by introducing a novel loss term that enforces that the cumulative effect of a sequence of actions can be computed from the individual actions' representations ( FIG0 . Composability encourages disentangling: as a composed representation does not have access to the static content of the intermediate frames, a representation is composable only if the individual action representations are disentangled from the static content. Taken together, these two properties lead to a representation of sensory dynamics that captures the structure of the agent's actions.We make the following three contributions. First, we introduce a method for unsupervised learning of an agent's action space by training the latent representation of a stochastic video prediction model for the desiderata of minimality and composability. Second, we show that our method learns a representation of actions that is independent of scene content and visual characteristics on (i) a simulated robot with one degree of freedom and (ii) the BAIR robot pushing dataset (Ebert et al. (2017) ). Finally, we demonstrate that the learned representation can be used for action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action-labeled videos than extant supervised methods. We have shown a way of learning the structure of an agent's action space from visual observations alone by imposing the properties of minimality and composability on a latent variable for stochastic video prediction. This strategy offers a data-efficient alternative to approaches that rely on fully supervised action-conditioned methods. The resulting representation can be used for a range of tasks, such as action-conditioned video prediction and planning in the learned latent action space. The representation is insensitive to the static scene content and visual characteristics of the environment. It captures meaningful structure in synthetic settings and achieves promising results in realistic visual settings.
We learn a representation of an agent's action space from pure visual observations. We use a recurrent latent variable approach with a novel composability loss. Proposes a compositional latent-variable model to learn models that predict what will happen next in scenarios where action-labels are not available in abundance. A variational IB based approach to learn action representations directly from videos of actions being taken, achieving better efficiency of subsequent learning methods while requiring lesser amount of action label videos. This paper proposes an approach to video prediction which autonomously finds an action space encoding differences between subsequent frames