The robust and efficient recognition of visual relations in images is a hallmark of biological vision. Here, we argue that, despite recent progress in visual recognition, modern machine vision algorithms are severely limited in their ability to learn visual relations. Through controlled experiments, we demonstrate that visual-relation problems strain convolutional neural networks (CNNs). The networks eventually break altogether when rote memorization becomes impossible such as when the intra-class variability exceeds their capacity. We further show that another type of feedforward network, called a relational network (RN), which was shown to successfully solve seemingly difficult visual question answering (VQA) problems on the CLEVR datasets, suffers similar limitations. Motivated by the comparable success of biological vision, we argue that feedback mechanisms including working memory and attention are the key computational components underlying abstract visual reasoning. Consider the two images in Fig. 1 . The image on the left was correctly classified as a flute by a deep convolutional neural network (CNN) BID13 . This is quite a remarkable feat for such a complicated image, which includes distractors that partially occlude the object of interest. After the network was trained on millions of photographs, this and many other images were accurately categorized into one thousand natural object categories, surpassing, for the first time, the accuracy of a human observer on the ImageNet classification challenge. Now, consider the image on the right. On its face, it is quite simple compared to the image on the left. It is just a binary image containing two curves. Further, it has a rather distinguishing property, at least to the human eye: both curves are the same. The relation between the two items in this simple scene is rather intuitive and immediately obvious to a human observer. Yet, the CNN failed to learn this relation even after seeing millions of training examples.Why is it that a CNN can accurately detect the flute in Fig. 1a while struggling to recognize the simple relation depicted in Fig. 1b? (a) (b) Figure 1: Two images: The image in panel (a) can be classified with high confidence as containing a flute by contemporary computer vision algorithms. However, these same algorithms struggle to learn the concept of "sameness" as exemplified by the image with the two curves shown in panel (b) . The image in panel (b) is sampled from the SVRT challenge BID6 .That such task is difficult, and even sometimes impossible for contemporary computer vision algorithms including CNNs, is known BID6 BID12 BID4 BID26 but has, so far, been overlooked. To make matters worse, the issue has been overshadowed by the recent success of a novel class of neural networks called relational networks (RNs) on seemingly challenging visual question answering (VQA) benchmarks. However , RNs have so far only been tested using toy datasets like the sort-of-CLEVR dataset which depicts combinations of items of only a handful of colors and shapes BID22 . As we will show, RNs suffer the same limitations as CNNs for a same-different task such as the one shown in Fig. 1b .This failure of modern computer vision algorithms is all the more striking given the widespread ability to recognize visual relations across the animal kingdom, from human and non-human primates BID3 BID15 to rodents BID28 , birds BID2 BID18 and even insects BID10 . Examining the failures of existing models is a critical step on the path to understanding the computational principles underlying visual reasoning. Yet, to our knowledge , there has not been any systematic exploration of the limits of contemporary machine learning algorithms on relational reasoning problems.Previous work by BID6 showed that black-box classifiers fail on most tasks from the synthetic visual reasoning test (SVRT), a battery of twenty-three visual-relation problems, despite massive amounts of training data. More recent work by BID4 and BID26 each showed how two different CNN architectures could only solve a handful of the twentythree SVRT problems. Similarly, BID12 , after showing how CNNs fail to learn a same-different task with simple binary "sprite" items, only managed to train a multi-layer perceptron on this task by providing carefully engineered training schedules. However, these results of BID4 , BID12 and BID26 were inconclusive: does the inability of feedforward neural networks to solve various visual-relation problems reflect a poor choice of hyperparameters for their particular implementation or rather a systematic failure of the entire class of feedforward models?Here, we propose to systematically probe the limits of CNNs and other state-of-the-art visual reasoning networks (RNs) on visual-relation tasks. Through a series of controlled experiments , we demonstrate that visual-relation tasks strain CNNs and that these limitations are not alleviated in RNs, which were specifically designed to tackle visual-relation problems. A brief review of the biological vision literature suggests that two key brain mechanisms, working memory and attention, underlie primates' ability to reason about visual relations. We argue that these mechanisms and possibly other feedback mechanisms are needed to extend current computer vision models to efficiently learn to solve complex visual reasoning tasks.Our contributions are threefold: (i) We perform the first systematic performance analysis of CNN architectures on each of the twenty-three SVRT problems. This yields a dichotomy of visual-relation problems, hard same-different problems vs. easy spatial-relation problems.(ii) We describe a novel, controlled, visual-relation challenge which convincingly shows that CNNs solve same-different tasks via rote memorization. (iii) We show that a simple modification of the sort-of-CLEVR challenge similarly breaks state-of-the-art relational network architectures.Overall, we wish to motivate the computer vision community to reconsider existing visual question answering challenges and turn to neuroscience and cognitive science for inspiration to help with the design of visual reasoning architectures. Our results indicate that visual-relation problems can quickly exceed the representational capacity of CNNs. While learning templates for individual objects appears to be quite tractable for today's deep networks, learning templates for arrangements of objects become rapidly intractable because of the combinatorial explosion in the number of templates needed. That stimuli with a combinatorial structure are difficult to represent with feedforward networks has been long acknowledged by cognitive scientists at least as early as BID7 . However, this limitation seems to have been somehow overlooked by current computer vision scientists.Compared to the feedforward networks in this study, biological visual systems excel at detecting relations. BID6 found that humans are capable of learning rather complicated visual rules and generalizing them to new instances from just a few SVRT training examples. For instance, their participants could learn the rule underlying the hardest SVRT problem for CNNs in our Experiment 1, problem 20, from an average of about 6 examples. Moreover, problem 20 is rather complicated, involving two shapes such that "one shape can be obtained from the other by reflection around the perpendicular bisector of the line joining their centers." In contrast, the best performing network for this problem in our high-throughput search could not get significantly above chance after a million training examples.Visual reasoning ability is not just found in humans. For example, birds and primates can be trained to recognize same-different relations and then transfer this knowledge to novel objects BID29 . A recent, striking example of same-different learning in animals comes from BID18 who essentially showed that ducklings can perform a one-shot version of our Experiment 3 from birth. During a training phase, newly hatched ducklings were exposed to a single pair of simple 3D objects that were either the same or different. Later, they demonstrated a preference for novel objects obeying the relationship observed in the training phase. This result suggests that these animals can either rapidly learn the abstract concepts of same and different from a single example or they simply possess these concepts innately. Contrast the behavior of these ducklings with the CNN+RN of Experiment 3, which demonstrated no ability to transfer the concept of same-different to novel objects FIG2 even after hundreds of thousands of training examples. For a recent review of similar literature (including additional evidence for abstract relational reasoning in pigeons and nutcrackers), see BID30 .There is substantial evidence that the neural substrate of visual-relation detection may depend on reentrant/feedback signals beyond feedforward, pre-attentive processes. It is relatively well accepted that, despite the widespread presence of feedback connections in our visual cortex, certain visual recognition tasks, including the detection of natural object categories, are possible in the near absence of cortical feedback -based primarily on a single feedforward sweep of activity through our visual cortex BID23 . However , psychophysical evidence suggests that this feedforward sweep is too spatially coarse to localize objects even when they can be recognized BID5 . The implication is that object localization in clutter requires attention BID31 .It is difficult to imagine how one could recognize the spatial relation between two objects without spatial information. Indeed, converging neuroscience evidence BID17 BID19 BID21 BID14 BID8 BID27 suggests that the processing of spatial relations between pairs of objects in a cluttered scene requires attention, even when participants are able to detect the presence of the individual objects pre-attentively, presumably in a single feedforward sweep.Another brain mechanism that has been implicated in our ability to process visual relations is working memory BID16 BID11 BID1 BID0 . In particular, imaging studies BID16 BID11 have highlighted the role of working memory in prefrontal and premotor cortices when participants solve Raven's progressive matrices which require both spatial and same-different reasoning.What is the computational role of attention and working memory in the detection of visual relations? One assumption BID8 is that these two mechanisms allow flexible representations of relations to be constructed dynamically at run-time via a sequence of attention shifts rather than statically by storing visual-relation templates in synaptic weights (as done in feedforward neural networks). Such representations built "on-the-fly" circumvent the combinatorial explosion associated with the storage of templates for all possible relations, helping to prevent the capacity overload associated with feedforward neural networks.Humans can easily detect when two objects are the same up to some transformation BID24 or when objects exist in a given spatial relation BID6 BID8 . More generally, humans can effortlessly construct an unbounded set of structured descriptions about the visual world around them BID9 . Given the vast superiority of humans over modern computers in their ability to detect visual relations, we see the exploration of attentional and mnemonic mechanisms as an important step in our computational understanding of visual reasoning.
Using a novel, controlled, visual-relation challenge, we show that same-different tasks critically strain the capacity of CNNs; we argue that visual relations can be better solved using attention-mnemonic strategies. Demonstrates that convolutional and relational neural networks fail to solve visual relation problems by training networks on artificially generated visual relation data.  This paper explores how current CNN's and Relational Networks fail to recognize visual relations in images.