Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'Stylized-ImageNet', a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation. How are Convolutional Neural Networks (CNNs) able to reach impressive performance on complex perceptual tasks such as object recognition (Krizhevsky et al., 2012) and semantic segmentation (Long et al., 2015) ? One widely accepted intuition is that CNNs combine low-level features (e.g. edges) to increasingly complex shapes (such as wheels, car windows) until the object (e.g. car) can be readily classified. As Kriegeskorte (2015) puts it, "the network acquires complex knowledge about the kinds of shapes associated with each category. [...] High-level units appear to learn representations of shapes occurring in natural images" (p. 429). This notion also appears in other explanations, such as in LeCun et al. (2015) : Intermediate CNN layers recognise "parts of familiar objects, and subsequent layers [...] detect objects as combinations of these parts" (p. 436). We term this explanation the shape hypothesis.This hypothesis is supported by a number of empirical findings. Visualisation techniques like Deconvolutional Networks (Zeiler & Fergus, 2014) often highlight object parts in high-level CNN features. 1 Moreover, CNNs have been proposed as computational models of human shape perception by Kubilius et al. (2016) , who conducted an impressive number of experiments comparing human and CNN shape representations and concluded that CNNs "implicitly learn representations of shape that reflect human shape perception" (p. 15). Ritter et al. (2017) discovered that CNNs develop a so-called "shape bias" just like children, i.e. that object shape is more important than colour for object classification (although see BID15 for contrary evidence). Furthermore, CNNs are currently the most predictive models for human ventral stream object recognition (e.g. BID2 BID2 ; and it is well-known that object shape is the single most important cue for human object recognition (Landau et al., 1988) , much more than other cues like size or texture (which may explain the ease at which humans recognise line drawings or millennia-old cave paintings).On the other hand, some rather disconnected findings point to an important role of object textures for CNN object recognition. CNNs can still classify texturised images perfectly well, even if the global shape structure is completely destroyed BID1 . Conversely , standard CNNs are bad at recognising object sketches where object shapes are preserved yet all texture cues are missing BID0 . Additionally , two studies suggest that local information such as textures may actually be sufficient to "solve" ImageNet object recognition: BID6 discovered that a linear classifier on top of a CNN's texture representation (Gram matrix) achieves hardly any classification performance loss compared to original network performance. More recently , BID1 demonstrated that CNNs with explicitly constrained receptive field sizes throughout all layers are able to reach surprisingly high accuracies on ImageNet, even though this effectively limits a model to recognising small local patches rather than integrating object parts for shape recognition. Taken together , it seems that local textures indeed provide sufficient information about object classes-ImageNet object recognition could, in principle, be achieved through texture recognition alone. In the light of these findings, we believe that it is time to consider a second explanation, which we term the texture hypothesis: in contrast to the common assumption, object textures are more important than global object shapes for CNN object recognition.Resolving these two contradictory hypotheses is important both for the deep learning community (to increase our understanding of neural network decisions) as well as for the human vision and neuroscience communities (where CNNs are being used as computational models of human object recognition and shape perception). In this work we aim to shed light on this debate with a number of carefully designed yet relatively straightforward experiments. Utilising style transfer BID7 , we created images with a texture-shape cue conflict such as the cat shape with elephant texture depicted in Figure 1c . This enables us to quantify texture and shape biases in both humans and CNNs. To this end, we perform nine comprehensive and careful psychophysical experiments comparing humans against CNNs on exactly the same images, totalling 48,560 psychophysical trials across 97 observers. These experiments provide behavioural evidence in favour of the texture hypothesis: A cat with an elephant texture is an elephant to CNNs, and still a cat to humans. Beyond quantifying existing biases, we subsequently present results for our two other main contributions: changing biases, and discovering emergent benefits of changed biases. We show that the texture bias in standard CNNs can be overcome and changed towards a shape bias if trained on a suitable data set. Remarkably, networks with a higher shape bias are inherently more robust to many different image distortions (for some even reaching or surpassing human performance, despite never being trained on any of them) and reach higher performance on classification and object recognition tasks. DISPLAYFORM0 As noted in the Introduction, there seems to be a large discrepancy between the common assumption that CNNs use increasingly complex shape features to recognise objects and recent empirical findings which suggest a crucial role of object textures instead. In order to explicitly probe this question, we utilised style transfer BID7 to generate images with conflicting shape and texture information. On the basis of extensive experiments on both CNNs and human observers in a controlled psychophysical lab setting, we provide evidence that unlike humans, ImageNet-trained CNNs tend to classify objects according to local textures instead of global object shapes. In combination with previous work which showed that changing other major object dimensions such as colour BID10 and object size relative to the context (Eckstein et al., 2017) do not have a strong detrimental impact on CNN recognition performance, this highlights the special role that local cues such as textures seem to play in CNN object recognition.Intriguingly, this offers an explanation for a number of rather disconnected findings: CNNs match texture appearance for humans (Wallis et al., 2017) , and their predictive power for neural responses along the human ventral stream appears to be largely due to human-like texture representations, but not human-like contour representations (Laskar et al., 2018; Long & Konkle, 2018) . Furthermore, texture-based generative modelling approaches such as style transfer BID7 , single image super-resolution BID12 as well as static and dynamic texture synthesis BID6 BID5 all produce excellent results using standard CNNs, while CNNbased shape transfer seems to be very difficult BID11 . CNNs can still recognise images with scrambled shapes BID1 ), but they have much more difficulties recognising objects with missing texture information BID0 Yu et al., 2017) . Our hypothesis might also explain why an image segmentation model trained on a database of synthetic texture images transfers to natural images and videos (Ustyuzhaninov et al., 2018) . Beyond that, our results show marked behavioural differences between ImageNet-trained CNNs and human observers. While both human and machine vision systems achieve similarly high accuracies on standard images BID10 , our findings suggest that the underlying classification strategies might actually be very different. This is problematic, since CNNs are being used as computational models for human object recognition (e.g. BID2 BID2 .In order to reduce the texture bias of CNNs we introduced Stylized-ImageNet (SIN), a data set that removes local cues through style transfer and thereby forces networks to go beyond texture recognition. Using this data set, we demonstrated that a ResNet-50 architecture can indeed learn to recognise objects based on object shape, revealing that the texture bias in current CNNs is not by design but induced by ImageNet training data. This indicates that standard ImageNet-trained models may be taking a "shortcut" by focusing on local textures, which could be seen as a version of Occam's razor: If textures are sufficient, why should a CNN learn much else? While texture classification may be easier than shape recognition, we found that shape-based features trained on SIN generalise well to natural images.Our results indicate that a more shape-based representation can be beneficial for recognition tasks that rely on pre-trained ImageNet CNNs. Furthermore , while ImageNet-trained CNNs generalise poorly towards a wide range of image distortions (e.g. BID3 BID9 , our ResNet-50 trained on Stylized-ImageNet often reaches or even surpasses human-level robustness (without ever being trained on the specific image degradations). This is exciting because BID10 showed that networks trained on specific distortions in general do not acquire robustness against other unseen image manipulations. This emergent behaviour highlights the usefulness of a shape-based representation: While local textures are easily distorted by all sorts of noise (including those in the real world, such as rain and snow), the object shape remains relatively stable. Furthermore, this finding offers a compellingly simple explanation for the incredible robustness of humans when coping with distortions: a shape-based representation. In summary, we provided evidence that machine recognition today overly relies on object textures rather than global object shapes as commonly assumed. We demonstrated the advantages of a shapebased representation for robust inference (using our Stylized-ImageNet data set 5 to induce such a representation in neural networks). We envision our findings as well as our openly available model weights, code and behavioural data set (49K trials across 97 observers) 6 to achieve three goals: Firstly, an improved understanding of CNN representations and biases. Secondly, a step towards more plausible models of human visual object recognition. Thirdly, a useful starting point for future undertakings where domain knowledge suggests that a shape-based representation may be more beneficial than a texture-based one. We would like to thank Dan Hendrycks for providing the results of We followed the paradigm of Geirhos et al. (2018) for maximal comparability. A trial consisted of 300 ms presentation of a fixation square and a 200 ms presentation of the stimulus image, which was followed by a full-contrast pink noise mask (1/f spectral shape) of the same size lasting for 200 ms. Participants had to choose one of 16 entry-level categories by clicking on a response screen shown for 1500 ms. On this screen, icons of all 16 categories were arranged in a 4 × 4 grid. The experiments were not self-paced and therefore one trial always lasted 2200 ms (300 ms + 200 ms + 200 ms + 1500 ms = 2200 ms). The necessary time to complete an experiment with 1280 stimuli was 47 minutes, for 160 stimuli six minutes, and for 48 stimuli two minutes. In the experiments with 1280 trials, observers were given the possibility of taking a brief break after every block of 256 trials (five blocks in total).As preparation, participants were shown the response screen prior to an experiment and were asked to name all 16 categories in order to get an overview over the possible stimuli categories and to make sure that all categories were clear from the beginning. They were instructed to click on the category they believed was presented. Responses through clicking on a response screen could be changed within the 1500 ms response interval, only the last entered response was counted as the answer. Prior to the real experiment a practice session was performed for the participants to get used to the time course of the experiment and the position of category items on the response screen. This screen was shown for an additional 300 ms in order to provide feedback and indicate whether the entered answer was incorrect. In that case , a short low beep sound occurred and the correct category was highlighted by setting its background to white. The practice session consisted of 320 trials. After 160 trials the participants had the chance to take a short break. In the break, their performance of the first block was shown on the screen along the percentage of trials where no answer was entered. After the practice blocks, observers were shown an example image of the manipulation (not used in the experiment) to minimise surprise. Images used in the practice session were natural images from 16-class-ImageNet BID10 , hence there was no overlap with images or manipulations used in the experiments.
ImageNet-trained CNNs are biased towards object texture (instead of shape like humans). Overcoming this major difference between human and machine vision yields improved detection performance and previously unseen robustness to image distortions. Using image stylizaton to augment training data for ImageNet-trained CNNs to make resulting networks appear more aligned with human judgements This paper studies CNNs like AlexNet, VGG, GoogleNet, and ResNet50, shows these models are biased towards texture when trained on ImageNet, and proposes a new ImageNet dataset.