The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry. We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts. If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain. SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner. Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts. State of the art deep learning approaches to machine learning have achieved impressive results in many problem domains, including classification BID7 BID33 , density modelling BID6 Oord et al., 2016a; , and reinforcement learning BID17 BID10 BID28 . They are still, however, far from possessing many traits characteristic of human intelligence. Such deep learning techniques tend to be overly data hungry, often rely on significant human supervision and tend to overfit to the training data distribution BID14 BID5 ). An important step towards bridging the gap between human and artificial intelligence is endowing algorithms with compositional concepts BID14 BID5 . Compositionality allows for reuse of a finite set of primitives (addressing the data efficiency and human supervision issues) across many scenarios by recombining them to produce an exponentially large number of novel yet coherent and potentially useful concepts (addressing the overfitting problem). Compositionality is at the core of such human abilities as creativity, imagination and language-based communication.We propose that concepts are abstractions over a set of primitives. For example, consider a toy hierarchy of visual concepts shown in Fig. 1 . Each node in this hierarchy is defined as a subset of visual primitives that make up the scene in the input image. These visual primitives might include factors like object identity, object colour, floor colour and wall colour. As one traverses the hierarchy from the subordinate over basic to superordinate levels of abstraction BID27 (i.e. from the more specific to the more general concepts corresponding to the same visual scene), the number of concept-defining visual primitives decreases. Hence, each parent concept in such a hierarchy is an Figure 1 : Schematic of an implicit concept hierarchy built upon a subset of four visual primitives: object identity (I), object colour (O), floor colour (F ) and wall colour (W ) (other visual primitives necessary to generate the scene are ignored in this example). Concepts form an implicit hierarchy, where each parent is an abstraction over its children and over the original set of visual primitives (the values of the concept-defining sets of visual primitives are indicated by the bold capital letters). In order to generate an image that corresponds to a concept, one has to fill in values for the factors that got abstracted away (represented as "_"), e.g. by sampling from their respective priors. Given certain nodes in the concept hierarchy, one can traverse the other nodes using logical operations. See Sec.3 for our formal definition of concepts. abstraction (i.e. a subset) over its children and over the original set of visual primitives. A more formal definition of concepts is provided in Sec. 3.Intelligent agents are able to discover and learn abstract compositional concepts using little supervision BID1 BID31 BID2 BID29 . Think of human word learning -we acquire the meaning of words through a combination of a continual stream of unsupervised visual data occasionally paired with a corresponding word label. This paper describes SCAN (Symbol-Concept Association Network, see Fig. 2A ), a neural network model capable of learning grounded visual concepts in a largely unsupervised manner through fast symbol association. First, we use the β-VAE BID8 to learn a set of independent representational primitives through unsupervised exposure to the visual data. This is equivalent to learning a disentangled (factorised and interpretable) representation of the independent ground truth "generative factors" of the data BID3 . Next, we allow SCAN to discover meaningful abstractions over these disentangled primitives by exposing it to a small number of symbol-image pairs that apply to a particular concept (e.g. a few example images of an apple paired with the symbol "apple"). SCAN learns the meaning of the concept by identifying the set of visual primitives that all the visual examples have in common (e.g. all observed apples are small, round and red). The corresponding symbol ("apple") then becomes a "pointer" to the newly acquired concept {small, round, red} -a way to access and manipulate the concept without having to know its exact representational form. Our approach does not make any assumptions about how these symbols are encoded, which also allows SCAN to learn multiple referents to the same concept, i.e. synonyms.Once a concept is acquired, it should be possible to use it for bi-directional inference: the model should be able to generate diverse visual samples that correspond to a particular concept (sym2img) and vice versa (img2sym). Since the projection from the space of visual primitives to the space of concepts (img2sym, red dash arrow in Fig. 1 ) involves abstraction and hence a loss of information, one then needs to add compatible information back in when moving from the space of concepts to that of visual primitives (sym2img, blue dot arrow in Fig. 1 ). In our setup, concepts are defined in terms of a set of relevant visual primitives (e.g. colour, shape and size for "apple"). This leaves a set of irrelevant visual attributes (e.g. lighting, position, background) to be "filled in". We do so by defaulting them to their respective priors, which ensures high diversity of samples (in both image or symbol space) for each concept during img2sym and sym2img inferences.The structured nature of learnt concepts acquired by SCAN allows for sample efficient learning of logical recombination operators: AND (corresponding to a set union of relevant primitives), IN COMMON (corresponding to set intersection) and IGNORE (corresponding to set difference), by pairing a small number of valid visual examples of recombined concepts with the respective operator names. Once the meaning of the operators has been successfully learned, SCAN can exploit the compositionality of the acquired concepts, and traverse previously unexplored parts of the implicit underlying concept hierarchy by manipulating and recombining existing concepts in novel ways. For example, a new node corresponding to the concept {blue, small} can be reached through the following instructions: "blue" AND "small" (going down the hierarchy from more general to more specific), "blueberry" IN COMMON "bluebell" (going up the hierarchy from more specific to more general) or "blueberry" IGNORE "round" (also going up the hierarchy). DISPLAYFORM0 To summarise, our paper 1) presents SCAN, a neural network model capable of learning compositional and hierarchical representations of visual concepts; 2) demonstrates that SCAN can be successfully trained with very little supervised data; 3) shows that after training, SCAN can perform multimodal (visual and symbolic) bi-directional inference and generation with high accuracy and diversity, outperforming all baselines; 4) shows that the addition of logical recombination operations allows SCAN to break out of its limited training data distribution and reach new nodes within the implicit hierarchy of concepts. This paper introduced a new approach to learning grounded visual concepts. We defined concepts as abstractions over independent (and often interpretable) visual primitives, where each concept is given by learned distributions over a set of relevant visual factors. We proposed that all other (irrelevant) visual factors should default to their prior in order to produce a diverse set of samples corresponding to a concept. We then proposed SCAN, a neural network implementation of such an approach, which was able to discover and learn an implicit hierarchy of abstract concepts from as few as five symbolimage pairs per concept and no assumptions on the nature of symbolic representations. SCAN was then capable of bi-directional inference, generating diverse and accurate image samples from symbolic instructions, and vice versa, qualitatively and quantitatively outperforming all baselines, including on a realistic CelebA dataset with noisy attribute labels. The structure of the learnt concepts allowed us to train an extension to SCAN that could perform logical recombination operators. We demonstrated how such operators could be used to traverse the implicit concept hierarchy, including imagining completely new concepts. Due to the sample efficiency and the limited number of assumptions in our approach, the representations learnt by SCAN should be immediately applicable within a large set of broader problem domains, including reinforcement learning, classification, control and planning.
We present a neural variational model for learning language-guided compositional visual concepts. Proposes a novel neural net architecture that learns object concepts by combining a beta-VAE and SCAN. This paper introduces a VAE-based model for translating between images and text, with their latent representation well-suited to applying symbolic operations, giving them a more expressive language for sampling images from text.  This paper proposes a new model called SCAN (Symbol-Concept Association Network) for hierarchical concept learning and allows for generalization to new concepts composed from existing concepts using logical operators.