Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers. Attention has become the subject of intensive research within the deep learning community. While biology is sometimes mentioned as a source of inspiration BID34 BID23 BID2 You et al., 2016; BID3 BID41 BID1 , the attentional mechanisms that have been considered remain limited in comparison to the rich and diverse array of processes used by the human visual system (see BID15 , for a review). In addition, whereas human attention is controlled by varying task demands, attention networks used in computer vision are solely optimized for object recognition. This means that, unlike infants who can rely on a myriad of visual cues and supervision to learn to focus their attention BID15 , DCNs must solve this challenging problem with weak supervisory signals derived from statistical associations between image pixels and class labels. Here, we investigate how explicit human supervision -teaching DCNs what and where to attend -affects their performance and interpretability. We have described the ClickMe dataset, which is aimed at supplementing ImageNet with nearly a half-million human-derived attention maps. The approach was validated with human psychophysics, which indicated the sufficiency of ClickMe features for rapid visual categorization. When participants viewed images that were masked to reveal commonly selected ClickMe map locations, they reached ceiling recognition accuracy when only 6% of image pixels were visible. By comparison, participants viewing images masked according to bottom-up saliency map locations did not reach ceiling performance until the full image was visible. These results indicate that ClickMe.ai may also provide novel insights into human vision with a measure of feature diagnosticity that goes beyond classic bottom-up saliency measures. While a detailed analysis of the ClickMe features falls outside the scope of the present study, we expect a more systematic analysis of this data, including the timecourse of feature selection BID4 BID11 , will aid our understanding of the different attention mechanisms responsible for the selection of diagnostic image features.We also extended the squeeze-and-excitation (SE) module which constituted the building block of the winning architecture in the ILSVRC17 challenge. We trained an SE-ResNet-50 on a reduced amount of data (∼ 300K samples) and found that the architecture overfits compared to a standard ResNet-50. We described a novel global-and-local attention (GALA) module and found that the proposed GALA-ResNet-50, however, significantly increases accuracy in this regime and cuts down top-5 error by ∼ 25% over both . In addition, we described an approach to co-train GALA using ClickMe supervision and cue the network to attend to image regions that are diagnostic to humans for object recognition. The routine casts ClickMe map prediction as an auxiliary task that can be combined with a primary visual categorization task. We found a trade-off between learning visual representations that are more similar to those used by human observers vs. learning visual representations that are more optimal for ILSVRC. The proper trade-off resulted in a model with better classification accuracy and more interpretable visual representations (both qualitatively and according to quantitative experiments on the ClickMe dataset and Microsoft COCO images).While recent advancements in DCNs have led to models that perform on par with human observers in basic visual recognition tasks, there is also growing evidence of qualitative differences in the visual strategies that they employ BID30 BID38 BID8 BID22 . It is not known whether these discrepancies arise because of differences in mechanisms for visual inference or fundamentally different training routines. However , our success in encouraging DCNs to learn more human-like representations with ClickMe map supervision suggests that improved training regimens can help close this gap. In particular , DCNs lack explicit mechanisms for perceptual grouping and figure-ground segmentation which are known to play a key role in the development of our visual system BID18 BID27 ) by simplifying the process of discarding background clutter. In the absence of figure-ground mechanisms, DCNs are compelled to associate foreground objects and their context as single perceptual units. This leads to DCN representations that are significantly more distributed compared to those used by humans BID22 . We hope that this work will help catalyze interest in the development of novel training paradigms that leverage combinations of visual cues (depth, motion, etc) for figure-ground segregation in order to substitute for the human supervision used here for co-training GALA.
A large-scale dataset for training attention models for object recognition leads to more accurate, interpretable, and human-like object recognition. Argues recent gains in visual recognition stem from using visual attention mechanisms in deep convolutional networks, which learn where to focus through a weak form of supervision based on image class labels. Presents a new take on attention in which a large attention dataset is collected and used to train a NN in a supervised manner to exploit self-reported human attention. This paper proposes a new approach to use more informative signals, specifically, regions humans deem important on images, to improve deep convolutional neural networks.