In this paper, we propose deep convolutional generative adversarial networks (DCGAN) that learn to produce a 'mental image' of the input image as internal representation of a certain category of input data distribution.   This mental image is what the DCGAN 'imagines' that the input image might look like under ideal conditions.   The mental image contains a version of the input that is iconic, without any peculiarities that do not contribute to the ideal representation of the input data distribution within a category. A DCGAN learns this association by training an encoder to capture salient features from the original image and a decoder to convert salient features into its associated mental image representation.   Our new approach, which we refer to as a Mental Image DCGAN (MIDCGAN), learns features that are useful for recognizing entire classes of objects, and that this in turn has the benefit of helping single and zero shot recognition.   We demonstrate our approach on object instance recognition and handwritten digit recognition tasks. Deep convolutional neural networks have had a revolutionary impact on machine learning and computer vision, yet we still fall dramatically short when it comes to learning in a manner that is most similar to people. Consider the way that children interact with objects when they are very young Yu et al., 2009 ): during their interaction, children look at objects from many different perspectives. Eventually they build up a preference for certain viewpoints after examining objects for a long period of time. In this paper, we consider the question of what would happen if we were to train a deep convolutional generative adversarial network in the same manner. For this, we provide the mental image (i.e., an ideal representation), and then provide samples of many different variations of the input image. The Mental Image DCGAN (MIDCGAN) is trained to associate each of these samples in a specific input distribution back to the mental image. This association is learned using a GAN architecture (see FIG1 ) with a generator composed of an encoder and decoder. MID-CGAN trains the encoder to learn salient bottleneck features for each class while the decoder learns to generate a mental image from bottleneck features. We will show that MIDCGAN bottleneck features are better suited for learning than those features that are generated without the benefit of using a mental image.Stated more formally, a typical learning task seeks to learn the data distribution, p(x) mapping to a class or category label y or p(y|x). The diversity of samples in the training data are limiting in the way the learner represents the category class internally. The MIDCGAN approach on the other hand provides a mental imagex as target to be learned and stored as representation of a category target distribution. The learner maps the input data distribution, p(x), to this canonical representation of the category,x, i.e p(x|x). During this mapping process, the MIDCGAN creates an internal bottleneck feature vector that is best representative of the input distribution mapping to the mental image.We demonstrate the effectiveness of mental image DCGANs (MIDCGAN) on two different problems. First, we demonstrate this for handwritten digits as proof of concept. In this case, we assume that a helpful tutor has provided an ideal representation of the digit, so the mental image is a stencil FIG2 . When the MIDCGAN sees a digit, it is trained to think of how it might look if it were looking at the stencil. We mainly demonstrate the performance of MIDCGAN on instance based object recognition. In this case, the helpful tutor provides the system with an iconic view of the object. The MIDCGAN observes the objects from different viewpoints, but at each time it is trained to think of how the object might look like if it were looking at the object from an ideal or iconic perspective. BID22 .We evaluate this in three different ways. First , we evaluate quantitatively the usefulness of the bottleneck features from MIDCGAN on learning tasks (comparing to DCGAN features trained without a mental image). Second , we evaluate qualitatively MIDCGAN's ability to generate mental images. Finally , we evaluate MIDCGAN's ability to perform few shot recognition on objects whose mental image was not learned or transfer learning. More precisely how does MIDCGAN perform when asked to imagine what an object in its frontal view, when it has never seen the object before. In this section, we will present and discuss the four major experiments. In the first two, MNIST and SVHN are evaluated using stencils targets as mental images. In the final two, we evaluate object instance recognition using the Big Berkeley Instance Recognition Database (BigBIRD) and the University of Washington Kinects Objects Dataset. In the first experiment, we learn mental images for each of the objects. In the second experiment, we use the features learned previously to recognize an entirely new database. In all experiments, we show that learning in this manner outperforms features learned from a typical DCGAN architecture. We have demonstrated the use of mental model autoencoder-based DCGAN (MIDCGAN) on digit classification (MNIST), and object recognition on both the Kinects Objects Dataset (RGB-D) and the Bigbird Dataset. Although MIDCGAN was demonstrated on representative viewpoints, the selection of the mental target image could be arbitrary. In cases when MIDCGAN was not sure about the object, the generated image did not resemble the actual representative image, or important details about the representative image were omitted. In essence, it could potentially provide another way to determine the confidence in the prediction of the object class.MIDCGAN can be used on robotics platform with real images in the wild. Given the growing interest in active object manipulation and recognition in the robotics community BID2 . Incorporating real manipulation of objects as separate modality with MIDCGAN could permit robots to learn about objects in their environment with minimal supervision. The mental images described in this paper map very well to the concept of prototype learning in the cognitive science community. Several improvements could be made to MIDCGAN in the future. First, we have generally assumed the availability of a tutor to select a mental image. Autonomous selection of the mental images is an inherent extension to allow semi-supervised and fully unsupervised MIDCGAN training. Alternatively, prototype images can be selected in some systematic manner using some heuristics related to the objects (i.e., a cup holds liquid so prototype is face up, the handle is to be grabbed so that should be visible).
Object instance recognition with adversarial autoencoders was performed with a novel 'mental image' target that is canonical representation of the input image. The paper proposes a method to learn features for object recognition that is invariant to various transformations of the object, most notably object pose. This paper investigated the task of few shot recognition via a generated “mental image” as intermediate representation given the input image.