Computational Models of Object Recognition in Cortex: A Review
---
paper_title: View-based Models of 3D Object Recognition: Invariance to Imaging Transformations

paper_content:

This report describes the main features of a view-based model of object recognition. The model does not attempt to account for specific cortical structures; it tries to capture general properties to be expected in a biological architecture for object recognition. The basic module is a regularization network (RBF-like; see Poggio and Girosi, 1989; Poggio, 1990) in which each of the hidden units is broadly tuned to a specific view of the object to be recognized. The network output, which may be largely view independent, is first described in terms of some simple simulations. The following refinements and details of the basic module are then discussed: (1) some of the units may represent only components of views of the object--the optimal stimulus for the unit, its "center," is effectively a complex feature; (2) the units' properties are consistent with the usual description of cortical neurons as tuned to multidimensional optimal stimuli and may be realized in terms of plausible biophysical mechanisms; (3) in learning to recognize new objects, preexisting centers may be used and modified, but also new centers may be created incrementally so as to provide maximal view invariance; (4) modules are part of a hierarchical structure--the output of a network may be used as one of the inputs to another, in this way synthesizing increasingly complex features and templates; (5) in several recognition tasks, in particular at the basic level, a single center using view-invariant features may be sufficient.
---
paper_title: Representation And Recognition In Vision

paper_content:

Researchers have long sought to understand what the brain does when we see an object, what two people have in common when they see the same object, and what a "seeing" machine would need to have in common with a human visual system. Recent neurobiological and computational advances in the study of vision have now brought us close to answering these and other questions about representation. In Representation and Recognition in Vision, Shimon Edelman bases a comprehensive approach to visual representation on the notion of correspondence between proximal (internal) and distal similarities in objects. This leads to a computationally feasible and formally veridical representation of distal objects that addresses the needs of shape categorization and can be used to derive models of perceived similarity. Edelman first discusses the representational needs of various visual recognition tasks, and surveys current theories of representation in this context. He then develops a theory of representation that is related to Shepard's notion of second-order isomorphism between representations and their targets. Edelman goes beyond Shepard by specifying the conditions under which the representations can be made formally veridical. Edelman assesses his theory's performance in identification and categorization of 3D shapes and examines it in light of psychological and neurobiological data concerning the object-processing stream in primate vision. He also discusses the connections between his theory and other efforts to understand representation in the brain.
---
paper_title: High-Level Vision: Object Recognition and Visual Cognition

paper_content:

Object recognition: shape-based recognition what is recognition? why object recognition is difficult. Approaches to object recognition: invariant properties and feature spaces parts and structural descriptions the alignment approach which is the correct approach?. The alignment of pictorial descriptions: using corresponding features the use of multiple models for 3-D objects aligning pictorial descriptions transforming the image or the models? before and after alignment. The alignment of smooth bounding contours: the curvate method accuracy of the curvature method empirical testing. Recognition by the combination of views: modelling objects by view combinations objects with sharp edges using two views only using a single view the use of depth values summary of the basic scheme objects with smooth boundaries recognition by image combinations extensions to the view-combination scheme psychophysical and physiological evidence interim conclusions: recognition by multiple views. Classifications: classification and identification the role of object classification class-based processing using class prototypes pictorial classification evidence from psychology and biology are classes in the world or in our head? the organization of recognition memory. Image and model correspondence: feature correspondence contour matching correspondence-less methods correspondence processes in human vision model construction compensating for illumination changes. Segmentation and saliency: is segmentation feasible? bottom-up and top-down segmentation extracting globally salient structures saliency, selection, and completion what can bottom-up segmentation achieve? Visual cognition and visual routines: perceiving "inside" and "outside" spatial analysis by visual routines conclusions and open problems the elemental operations the assembly and storage of routines routines and recognition. Sequence seeking and counter streams - a model for visual cortex: the sequence-seeking scheme biological embodiment summary. Appendices: alignment by feature the curvature method errors of the curvature method locally affine matching definitions.
---
paper_title: Visual Object Recognition

paper_content:

Technical solutions are described for training an object-recognition neural network that identifies an object in a computer-readable image. An example method includes assigning a first neural network for determining a visual alignment model of the images for determining a normalized alignment of the object. The method further includes assigning a second neural network for determining a visual representation model of the images for recognizing the object. The method further includes determining the visual alignment model by training the first neural network and determining the visual representation model by training the second neural network independent of the first. The method further includes determining a combined object recognition model by training a combination of the first neural network and the second neural network. The method further includes recognizing the object in the image based on the combined object recognition model by passing the image through each of the neural networks.
---
paper_title: High-Level Vision: Object Recognition and Visual Cognition

paper_content:

Object recognition: shape-based recognition what is recognition? why object recognition is difficult. Approaches to object recognition: invariant properties and feature spaces parts and structural descriptions the alignment approach which is the correct approach?. The alignment of pictorial descriptions: using corresponding features the use of multiple models for 3-D objects aligning pictorial descriptions transforming the image or the models? before and after alignment. The alignment of smooth bounding contours: the curvate method accuracy of the curvature method empirical testing. Recognition by the combination of views: modelling objects by view combinations objects with sharp edges using two views only using a single view the use of depth values summary of the basic scheme objects with smooth boundaries recognition by image combinations extensions to the view-combination scheme psychophysical and physiological evidence interim conclusions: recognition by multiple views. Classifications: classification and identification the role of object classification class-based processing using class prototypes pictorial classification evidence from psychology and biology are classes in the world or in our head? the organization of recognition memory. Image and model correspondence: feature correspondence contour matching correspondence-less methods correspondence processes in human vision model construction compensating for illumination changes. Segmentation and saliency: is segmentation feasible? bottom-up and top-down segmentation extracting globally salient structures saliency, selection, and completion what can bottom-up segmentation achieve? Visual cognition and visual routines: perceiving "inside" and "outside" spatial analysis by visual routines conclusions and open problems the elemental operations the assembly and storage of routines routines and recognition. Sequence seeking and counter streams - a model for visual cortex: the sequence-seeking scheme biological embodiment summary. Appendices: alignment by feature the curvature method errors of the curvature method locally affine matching definitions.
---
paper_title: A Computational Model for Visual Selection

paper_content:

We propose a computational model for detecting and localizing instances from an object class in static gray-level images. We divide detection into visual selection and final classification, concentrating on the former: drastically reducing the number of candidate regions that require further, usually more intensive, processing, but with a minimum of computation and missed detections. Bottom-up processing is based on local groupings of edge fragments constrained by loose geometrical relationships. They have no a priori semantic or geometric interpretation. The role of training is to select special groupings that are moderately likely at certain places on the object but rate in the background. We show that the statistics in both populations are stable. The candidate regions are those that contain global arrangements of several local groupings. Whereas our model was not conceived to explain brain functions, it does cohere with evidence about the functions of neurons in V1 and V2, such as responses to coarse or incomplete patterns (e.g., illusory contours) and to scale and translation invariance in IT. Finally, the algorithm is applied to face and symbol detection.
---
paper_title: Dynamic binding in a neural network for shape recognition.

paper_content:

Given a single view of an object, humans can readily recognize that object from other views that preserve the parts in the original view. Empirical evidence suggests that this capacity reflects the activation of a viewpoint-invariant structural description specifying the object's parts and the relations among them. This article presents a neural network that generates such a description. Structural description is made possible through a solution to the dynamic binding problem: Temporary conjunctions of attributes (parts and relations) are represented by synchronized oscillatory activity among independent units representing those attributes
---
paper_title: Recognition-by-components: A theory of human image understanding

paper_content:

The perceptual recognition of objects is conceptualized to be a process in which the image of the input is segmented at regions of deep concavity into an arrangement of simple geometric components, such as blocks, cylinders, wedges, and cones. The fundamental assumption of the proposed theory, recognition-by-components (RBC), is that a modest set of generalized-cone components, called geons (N £ 36), can be derived from contrasts of five readily detectable properties of edges in a two-dimensiona l image: curvature, collinearity, symmetry, parallelism, and cotermination. The detection of these properties is generally invariant over viewing position an$ image quality and consequently allows robust object perception when the image is projected from a novel viewpoint or is degraded. RBC thus provides a principled account of the heretofore undecided relation between the classic principles of perceptual organization and pattern recognition: The constraints toward regularization (Pragnanz) characterize not the complete object but the object's components. Representational power derives from an allowance of free combinations of the geons. A Principle of Componential Recovery can account for the major phenomena of object recognition: If an arrangement of two or three geons can be recovered from the input, objects can be quickly recognized even when they are occluded, novel, rotated in depth, or extensively degraded. The results from experiments on the perception of briefly presented pictures by human observers provide empirical support for the theory. Any single object can project an infinity of image configurations to the retina. The orientation of the object to the viewer can vary continuously, each giving rise to a different two-dimensional projection. The object can be occluded by other objects or texture fields, as when viewed behind foliage. The object need not be presented as a full-colored textured image but instead can be a simplified line drawing. Moreover, the object can even be missing some of its parts or be a novel exemplar of its particular category. But it is only with rare exceptions that an image fails to be rapidly and readily classified, either as an instance of a familiar object category or as an instance that cannot be so classified (itself a form of classification).
---
paper_title: A Computational Model for Visual Selection

paper_content:

We propose a computational model for detecting and localizing instances from an object class in static gray-level images. We divide detection into visual selection and final classification, concentrating on the former: drastically reducing the number of candidate regions that require further, usually more intensive, processing, but with a minimum of computation and missed detections. Bottom-up processing is based on local groupings of edge fragments constrained by loose geometrical relationships. They have no a priori semantic or geometric interpretation. The role of training is to select special groupings that are moderately likely at certain places on the object but rate in the background. We show that the statistics in both populations are stable. The candidate regions are those that contain global arrangements of several local groupings. Whereas our model was not conceived to explain brain functions, it does cohere with evidence about the functions of neurons in V1 and V2, such as responses to coarse or incomplete patterns (e.g., illusory contours) and to scale and translation invariance in IT. Finally, the algorithm is applied to face and symbol detection.
---
paper_title: High-Level Vision: Object Recognition and Visual Cognition

paper_content:

Object recognition: shape-based recognition what is recognition? why object recognition is difficult. Approaches to object recognition: invariant properties and feature spaces parts and structural descriptions the alignment approach which is the correct approach?. The alignment of pictorial descriptions: using corresponding features the use of multiple models for 3-D objects aligning pictorial descriptions transforming the image or the models? before and after alignment. The alignment of smooth bounding contours: the curvate method accuracy of the curvature method empirical testing. Recognition by the combination of views: modelling objects by view combinations objects with sharp edges using two views only using a single view the use of depth values summary of the basic scheme objects with smooth boundaries recognition by image combinations extensions to the view-combination scheme psychophysical and physiological evidence interim conclusions: recognition by multiple views. Classifications: classification and identification the role of object classification class-based processing using class prototypes pictorial classification evidence from psychology and biology are classes in the world or in our head? the organization of recognition memory. Image and model correspondence: feature correspondence contour matching correspondence-less methods correspondence processes in human vision model construction compensating for illumination changes. Segmentation and saliency: is segmentation feasible? bottom-up and top-down segmentation extracting globally salient structures saliency, selection, and completion what can bottom-up segmentation achieve? Visual cognition and visual routines: perceiving "inside" and "outside" spatial analysis by visual routines conclusions and open problems the elemental operations the assembly and storage of routines routines and recognition. Sequence seeking and counter streams - a model for visual cortex: the sequence-seeking scheme biological embodiment summary. Appendices: alignment by feature the curvature method errors of the curvature method locally affine matching definitions.
---
paper_title: Representation and recognition of the spatial organization of three-dimensional shapes

paper_content:

The human visual process can be studied by examining the computational problems associated with deriving useful information from retinal images. In this paper, we apply this approach to the problem of representing three-dimensional shapes for the purpose of recognition. 1. Three criteria, accessibility, scope and uniqueness, and stability and sensitivity, are presented for judging the usefulness of a representation for shape recognition. 2. Three aspects of a representation9s design are considered, (i) the representation9s coordinate system, (ii) its primitives, which are the primary units of shape information used in the representation, and (iii) the organization the representation imposes on the information in its descriptions. 3. In terms of these design issues and the criteria presented, a shape representation for recognition should: (i) use an object-centred coordinate system, (ii) include volumetric primitives of varied sizes, and (iii) have a modular organization. A representation based on a shape9s natural axes (for example the axes identified by a stick figure) follows directly from these choices. 4. The basic process for deriving a shape description in this representation must involve: (i) a means for identifying the natural axes of a shape in its image and (ii) a mechanism for transforming viewer-centred axis specifications to specifications in an object-centred coordinate system. 5. Shape recognition involves: (i) a collection of stored shape descriptions, and (ii) various indexes into the collection that allow a newly derived description to be associated with an appropriate stored description. The most important of these indexes allows shape recognition to proceed conservatively from the general to the specific based on the specificity of the information available from the image. 6. New constraints supplied by a conservative recognition process can be used to extract more information from the image. A relaxation process for carrying out this constraint analysis is described.
---
paper_title: Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex

paper_content:

Recent neurophysiological experiments appear to indicate that the responses of visual cortical neurons in a monkey freely viewing a natural scene can sometimes differ substantially from those obtained when the same image subregions are flashed during a conventional fixation task. These new findings attain significance from the fact that neurophysiological research in the past has been based predominantly on cell recordings obtained during fixation tasks, under the assumption that these data would be useful in predicting responses in more general situations. We describe a hierarchical model of visual memory that reconciles the two differing experimental results mentioned above by predicting neural responses in both fixating and free-viewing conditions. The model dynamically combines input-driven bottom-up signals with expectation-driven top-down signals to achieve optimal estimation of current state using a Kalman filter based framework. The architecture of the model posits a role for the reciprocal connections between adjoining visual cortical areas in determining neural response properties.
---
paper_title: Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

paper_content:

A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.
---
paper_title: Properties of Simulated Neurons from a Model of Primate Inferior Temporal Cortex

paper_content:

The physiological properties of neurons in inferior temporal (IT) cortex of the macaque monkey suggest that this cortical area plays a major role in visual pattern recognition. Based on the properties of IT, and one of its major sources of input, V4, a model is proposed that can account for some of the shape recognition properties of IT neurons including selectivity for complex visual stimuli and tolerance to the size and location of the stimuli. The model is composed of three components. First, stimulus location tolerance is modeled after the complex-cell-like properties observed in some V4 neurons. The second component of the model is an attentionally controlled scaling mechanism that facilitates size-invariant shape recognition. The transition from edge orientation-selective neurons in V4 to neurons with more complicated stimulus preference in IT is explained by the third component of the model, a competitive learning mechanism. Single-unit analysis of receptive field properties, stimulus selectivity, and stimulus size and position tolerance was performed on "neurons" from the simulation. Comparison of results from the simulation and a study of actual IT neurons shows that the set of mechanisms incorporated into the simulation is sufficient to emulate the physiological data.
---
paper_title: SEEMORE: Combining Color, Shape, and Texture Histogramming in a Neurally Inspired Approach to Visual Object Recognition

paper_content:

Severe architectural and timing constraints within the primate visual system support the conjecture that the early phase of object recognition in the brain is based on a feedforward feature-extraction hierarchy. To assess the plausibility of this conjecture in an engineering context, a difficult three-dimensional object recognition domain was developed to challenge a pure feedforward, receptive-field based recognition model called SEEMORE. SEEMORE is based on 102 viewpoint-invariant nonlinear filters that as a group are sensitive to contour, texture, and color cues. The visual domain consists of 100 real objects of many different types, including rigid (shovel), nonrigid (telephone cord), and statistical (maple leaf cluster) objects and photographs of complex scenes. Objects were in dividually presented in color video images under normal room lighting conditions. Based on 12 to 36 training views, SEEMORE was required to recognize unnormalized test views of objects that could vary in position, orientation ...
---
paper_title: News On Views: Pandemonium Revisited

paper_content:

How do we recognize objects from different viewpoints? A new model, based on the known properties of cortical neurons, may help resolve this long-standing debate.
---
paper_title: A Neurobiological Model of Visual Attention and Invariant Pattern Recognition Based on Dynamic Routing of Information

paper_content:

We present a biologically plausible model of an attentional mechanism for forming position- and scale-invariant representations of objects in the visual world. The model relies on a set of control neurons to dynamically modify the synaptic strengths of intracortical connections so that information from a windowed region of primary visual cortex (V1) is selectively routed to higher cortical areas. Local spatial relationships (i.e., topography) within the attentional window are preserved as information is routed through the cortex. This enables attended objects to be represented in higher cortical areas within an object-centered reference frame that is position and scale invariant. We hypothesize that the pulvinar may provide the control signals for routing information through the cortex. The dynamics of the control neurons are governed by simple differential equations that could be realized by neurobiologically plausible circuits. In preattentive mode, the control neurons receive their input from a low-level “saliency map” representing potentially interesting regions of a scene. During the pattern recognition phase, control neurons are driven by the interaction between top-down (memory) and bottom-up (retinal input) sources. The model respects key neurophysiological, neuroanatomical, and psychophysical data relating to attention, and it makes a variety of experimentally testable predictions.
---
paper_title: Neurophysiology of shape processing

paper_content:

Recent physiological findings are reviewed and synthesized into a model of shape processing and object recognition. Gestalt laws (e.g. good continuation, closure) and ‘non-accidental’ image properties (e.g. colinear terminating lines) are resolved in prestriate visual cortex, (areas V2 and V3) to support the extraction of 2D shape boundaries. Processing of shape continues along a ventral route through inferior temporal (IT) cortex where a vast catalogue of 2D shape primitives is established. Each catalogue entry is size-specific (±0.5 log scale unit) and orientation-specific (±45°), but can generalize over position (±150 degree2). Several shape components are used to activate representations of the approximate appearance of one object type at one view, orientation and size. Subsequent generalization, first over orientation and size, then over view, and finally over object sub-component, is achieved in the anterior temporal cortex by combining descriptions of the same object from different orientations and views, through associative learning. This scheme provides a route to 3D object recognition through 2D shape description and reduces the problem of perceptual invariance to a series of independent analyses with an associative link established between the outputs. The system relies on parallel processing with computations performed in a series of hierarchical steps with relatively simple operations at each stage.
---
paper_title: Shifter circuits: a computational strategy for dynamic aspects of visual processing

paper_content:

Abstract ::: We propose a general strategy for dynamic control of information flow between arrays of neurons at different levels of the visual pathway, starting in the lateral geniculate nucleus and the geniculorecipient layers of cortical area V1. This strategy can be used for resolving computational problems arising in the domains of stereopsis, directed visual attention, and the perception of moving images. In each of these situations, some means of dynamically controlling how retinal outputs map onto higher-level targets is desirable--in order to achieve binocular fusion, to allow shifts of the focus of attention, and to prevent blurring of moving images. The proposed solution involves what we term "shifter circuits," which allow for dynamic shifts in the relative alignment of input and output arrays without loss of local spatial relationships. The shifts are produced in increments along a succession of relay stages that are linked by diverging excitatory inputs. The direction of shift is controlled at each stage by inhibitory neurons that selectively suppress appropriate sets of ascending inputs. The shifter hypothesis is consistent with available anatomical and physiological evidence on the organization of the primate visual pathway, and it offers a sensible explanation for a variety of otherwise puzzling facts, such as the plethora of cells in the geniculorecipient layers of V1.
---
paper_title: Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque.

paper_content:

1. We recorded from single neurons in the dorsal bank and fundus of the anterior portion of the superior temporal sulcus, an area we term the superior temporal polysensory area (STP). Five macaques were studied under anesthesia ( N20) and immobilization in repeated recording sessions. 2. Almost all of the neurons were visually responsive, and over half responded to more than one sensory modality; 21% responded to visual and auditory stimuli, 17% responded to visual and somesthetic stimuli, 17% were trimodal, and 41% were exclusively visual. 3. Almost all the visual receptive fields extended into both visual half-fields, and the majority approached the size of the visual field of the monkey, including both monocular crescents. Somesthetic receptive fields were also bilateral and usually included most of the body surface. 4. Virtually all neurons responded better to moving visual stimuli than to stationary visual stimuli, and almost half were sensitive to the direction of movement. Several classes of directional neurons were found, including a) neurons selective for a single direction of movement throughout their receptive field, b) neurons selective for directions of movement radially symmetric about the center of gaze, and c) neurons selective for movement in depth. 5. The majority of neurons (70%) had little or no preference for stimulus size, shape, orientation, or contrast. The minority (30%) responded best to particular stimuli. Some of these appeared to be selective for faces. 6. The properties of most STP neurons, such as large receptive fields, sensitivity to movement, insensitivity to form, and polymodal responsiveness, suggest that STP is more involved in orientation and spatial functions than in pattern recognition.
---
paper_title: Rotating objects to recognize them: A case study on the role of viewpoint dependency in the recognition of three-dimensional objects.

paper_content:

Successful object recognition is essential for finding food, identifying kin, and avoiding danger, as well as many other adaptive behaviors. To accomplish this feat, the visual system must reconstruct 3-D interpretations from 2-D “snapshots” falling on the retina. Theories of recognition address this process by focusing on the question of how object representations are encoded with respect to viewpoint. Although empirical evidence has been equivocal on this question, a growing body of surprising results, including those obtained in the experiments presented in this case study, indicates that recognition is often viewpoint dependent. Such findings reveal a prominent role for viewpointdependent mechanisms and provide support for themultiple-views approach, in which objects are encoded as a set of view-specific representations that are matched to percepts using normalization procedures.
---
paper_title: Activity of neurones in the inferotemporal cortex of the alert monkey

paper_content:

The activity of neurones in the inferotemporal cortex of the alert rhesus monkey was recorded while the monkey was shown visual stimuli, which included both food and non-food objects for comparison with the activity of neurones in the lateral hypothalamus and substantia innominata. In the anteroventral part of the inferotemporal cortex, neurones were found with visual responses which were sustained while the animal looked at the appropriate visual stimuli. The latency of the responses was 100 msec or more. The majority (96/142 or 68%) of these neurones responded more strongly to some stimuli than to others. These units usually had different responses when objects were shown from different views, and physical factors such as shape, size, orientation, colour and texture appeared to account for the responses of some of these units. Association of visual stimuli with a food reward (glucose solution) or an aversive taste (5% saline solution) did not affect the magnitude of the responses of the neurones to the stimuli either during the learning or after the period of learning. Nor did feeding the monkey to satiety affect the responses of the neurones to their effective stimuli.
---
paper_title: Inferotemporal Cortex and Object Vision

paper_content:

Cells in area TE of the inferotemporal cortex of the monkey brain selectively respond to various moderately complex object features, and those that cluster in a columnar region that runs perpendicular to the cortical surface respond to similar features. Although cells within a column respond to similar features, their selectivity is not necessarily identical. The data of optical imaging in 'E have suggested that the borders between neighboring columns are not discrete; a continuous mapping of complex feature space within a larger region contains several partially overlapped columns. This continuous mapping may be used for various computations, such as production of the image of the object at different viewing angles, illumination conditions. and articulation poses.
---
paper_title: Face-Selective Cells in the Temporal Cortex of Monkeys

paper_content:

The notion of a neuron that responds selectively to the image of a particular complex object has been controversial ever since Gross and his colleagues reported neurons in the temporal cortex of monkeys that were selective for the sight of a monkey's hand (Gross, Rocha-Miranda, & Bender, 1972). Since that time, evidence has mounted for neurons in the temporal lobe that respond selectively to faces. The present paper presents a critical analysis of the evidence for face neurons and discusses the implications of these neurons for models of object recognition. The paper also presents some possible reasons for the evolution of face neurons and suggests some analogies with the development of language in humans.
---
paper_title: Psychophysical support for a two-dimensional view interpolation theory of object recognition.

paper_content:

Does the human brain represent objects for recognition by storing a series of two-dimensional snapshots, or are the object models, in some sense, three-dimensional analogs of the objects they represent? One way to address this question is to explore the ability of the human visual system to generalize recognition from familiar to unfamiliar views of three-dimensional objects. Three recently proposed theories of object recognition--viewpoint normalization or alignment of three-dimensional models [Ullman, S. (1989) Cognition 32, 193-254], linear combination of two-dimensional views [Ullman, S. & Basri, R. (1990) Recognition by Linear Combinations of Models (Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge), A. I. Memo No. 1152], and view approximation [Poggio, T. & Edelman, S. (1990) Nature (London) 343, 263-266]--predict different patterns of generalization to unfamiliar views. We have exploited the conflicting predictions to test the three theories directly in a psychophysical experiment involving computer-generated three-dimensional objects. Our results suggest that the human visual system is better described as recognizing these objects by two-dimensional view interpolation than by alignment or other methods that rely on object-centered three-dimensional models.
---
paper_title: Visual Object Recognition

paper_content:

Technical solutions are described for training an object-recognition neural network that identifies an object in a computer-readable image. An example method includes assigning a first neural network for determining a visual alignment model of the images for determining a normalized alignment of the object. The method further includes assigning a second neural network for determining a visual representation model of the images for recognizing the object. The method further includes determining the visual alignment model by training the first neural network and determining the visual representation model by training the second neural network independent of the first. The method further includes determining a combined object recognition model by training a combination of the first neural network and the second neural network. The method further includes recognizing the object in the image based on the combined object recognition model by passing the image through each of the neural networks.
---
paper_title: Shape representation in the inferior temporal cortex of monkeys

paper_content:

Background: The inferior temporal cortex (IT) of the monkey has long been known to play an essential role in visual object recognition. Damage to this area results in severe deficits in perceptual learning and object recognition, without significantly affecting basic visual capacities. Consistent with these ablation studies is the discovery of IT neurons that respond to complex two-dimensional visual patterns, or objects such as faces or body parts. What is the role of these neurons in object recognition? Is such a complex configurational selectivity specific to biologically meaningful objects, or does it develop as a result of extensive exposure to any objects whose identification relies on subtle shape differences? If so, would IT neurons respond selectively to recently learned views or features of novel objects? The present study addresses this question by using combined psychophysical and electrophysiological experiments, in which monkeys learned to classify and recognize computer-generated three-dimensional objects. Results: A population of IT neurons was found that responded selectively to views of previously unfamiliar objects. The cells discharged maximally to one view of an object, and their response declined gradually as the object was rotated away from this preferred view. No selective responses were ever encountered for views that the animal systematically failed to recognize. Most neurons also exhibited orientation-dependent responses during view-plane rotations. Some neurons were found to be tuned around two views of the same object, and a very small number of cells responded in a view-invariant manner. For the five different objects that were used extensively during the training of the animals, and for which behavioral performance became view-independent, multiple cells were found that were tuned around different views of the same object. A number of view-selective units showed response invariance for changes in the size of the object or the position of its image within the parafovea. Conclusion: Our results suggest that IT neurons can develop a complex receptive field organization as a consequence of extensive training in the discrimination and recognition of objects. None of these objects had any prior meaning for the animal, nor did they resemble anything familiar in the monkey's environment. Simple geometric features did not appear to account for the neurons' selective responses. These findings support the idea that a population of neurons - each tuned to a different object aspect, and each showing a certain degree of invariance to image transformations - may, as an ensemble, encode at least some types of complex threedimensional objects. In such a system, several neurons may be active for any given vantage point, with a single unit acting like a blurred template for a limited neighborhood of a single view.
---
paper_title: Inferotemporal units in selective visual attention and short-term memory

paper_content:

1. This research was designed to further clarify how, in the primate, the neurons of the inferotemporal (IT) cortex support the cognitive functions of visually guided behavior. Specifically, the aim was to determine the role of those neurons in 1) selective attention to behaviorally relevant features of the visual environment and 2) retention of those features in temporary memory. Monkeys were trained in a memory task in which they had to discriminate and retain individual features of compound stimuli, each stimulus consisting of a colored disk with a gray symbol in the middle. A trial began with brief presentation of one such stimulus, the sample for the trial. Depending on the symbol in it, the monkey had to memorize the symbol itself or the background color; after 10-20 s of delay (retention period), two compound stimuli appeared, and the animal had to choose the one with the symbol or with the color of the sample. Thus the test required attention to the symbol, in some trials also to the color, and short-term retention of the distinctive feature for each trial, either a symbol or a color. Single-unit activity was recorded from cortex of the IT convexity, lower and upper banks of the superior temporal sulcus (STS), and from striate cortex (V1). Firing frequency was analyzed during intertrial periods and during the entirety of every trial, except for the (match) choice period. 2. In IT cortex, as in V1, many units responded to the sample stimulus. Some responded indiscriminately to all samples, whereas others responded selectively to one of their features, i.e., to one symbol or to one color. Fifteen percent of the IT units were symbol selective and 21% color selective. These neurons appeared capable of extracting individual features from complex stimuli. Some color cells (color-attentive units) responded significantly more to their preferred color when it was relevant (i.e., had to be retained) than when it was not. 3. The latency of IT-unit response to the sample stimulus was, on the average, relatively short in unselective units (mean 159 ms), longer in symbol units (mean 203 ms), and longest in color-attentive units (mean 270 ms). This order of latencies corresponds to the presumed order of participation of those three types of units in the selective attention to the component features of the sample as required by the task. It suggests intervening steps of serial processing before color information reached color-attentive cells.(ABSTRACT TRUNCATED AT 400 WORDS)
---
paper_title: Image-based object recognition in man, monkey and machine

paper_content:

Theories of visual object recognition must solve the problem of recognizing 3D objects given that perceivers only receive 2D patterns of light on their retinae. Recent findings from human psychophysics, neurophysiology and machine vision provide converging evidence for ‘image-based’ models in which objects are represented as collections of viewpoint-specific local features. This approach is contrasted with ‘structural-description’ models in which objects are represented as configurations of 3D volumes or parts. We then review recent behavioral results that address the biological plausibility of both approaches, as well as some of their computational advantages and limitations. We conclude that, although the image-based approach holds great promise, it has potential pitfalls that may be best overcome by including structural information. Thus, the most viable model of object recognition may be one that incorporates the most appealing aspects of both image-based and structural-description theories.  1998 Elsevier Science B.V. All rights reserved
---
paper_title: View-based Models of 3D Object Recognition: Invariance to Imaging Transformations

paper_content:

This report describes the main features of a view-based model of object recognition. The model does not attempt to account for specific cortical structures; it tries to capture general properties to be expected in a biological architecture for object recognition. The basic module is a regularization network (RBF-like; see Poggio and Girosi, 1989; Poggio, 1990) in which each of the hidden units is broadly tuned to a specific view of the object to be recognized. The network output, which may be largely view independent, is first described in terms of some simple simulations. The following refinements and details of the basic module are then discussed: (1) some of the units may represent only components of views of the object--the optimal stimulus for the unit, its "center," is effectively a complex feature; (2) the units' properties are consistent with the usual description of cortical neurons as tuned to multidimensional optimal stimuli and may be realized in terms of plausible biophysical mechanisms; (3) in learning to recognize new objects, preexisting centers may be used and modified, but also new centers may be created incrementally so as to provide maximal view invariance; (4) modules are part of a hierarchical structure--the output of a network may be used as one of the inputs to another, in this way synthesizing increasingly complex features and templates; (5) in several recognition tasks, in particular at the basic level, a single center using view-invariant features may be sufficient.
---
paper_title: Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

paper_content:

A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.
---
paper_title: Psychophysical support for a two-dimensional view interpolation theory of object recognition.

paper_content:

Does the human brain represent objects for recognition by storing a series of two-dimensional snapshots, or are the object models, in some sense, three-dimensional analogs of the objects they represent? One way to address this question is to explore the ability of the human visual system to generalize recognition from familiar to unfamiliar views of three-dimensional objects. Three recently proposed theories of object recognition--viewpoint normalization or alignment of three-dimensional models [Ullman, S. (1989) Cognition 32, 193-254], linear combination of two-dimensional views [Ullman, S. & Basri, R. (1990) Recognition by Linear Combinations of Models (Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge), A. I. Memo No. 1152], and view approximation [Poggio, T. & Edelman, S. (1990) Nature (London) 343, 263-266]--predict different patterns of generalization to unfamiliar views. We have exploited the conflicting predictions to test the three theories directly in a psychophysical experiment involving computer-generated three-dimensional objects. Our results suggest that the human visual system is better described as recognizing these objects by two-dimensional view interpolation than by alignment or other methods that rely on object-centered three-dimensional models.
---
paper_title: Neurophysiology of shape processing

paper_content:

Recent physiological findings are reviewed and synthesized into a model of shape processing and object recognition. Gestalt laws (e.g. good continuation, closure) and ‘non-accidental’ image properties (e.g. colinear terminating lines) are resolved in prestriate visual cortex, (areas V2 and V3) to support the extraction of 2D shape boundaries. Processing of shape continues along a ventral route through inferior temporal (IT) cortex where a vast catalogue of 2D shape primitives is established. Each catalogue entry is size-specific (±0.5 log scale unit) and orientation-specific (±45°), but can generalize over position (±150 degree2). Several shape components are used to activate representations of the approximate appearance of one object type at one view, orientation and size. Subsequent generalization, first over orientation and size, then over view, and finally over object sub-component, is achieved in the anterior temporal cortex by combining descriptions of the same object from different orientations and views, through associative learning. This scheme provides a route to 3D object recognition through 2D shape description and reduces the problem of perceptual invariance to a series of independent analyses with an associative link established between the outputs. The system relies on parallel processing with computations performed in a series of hierarchical steps with relatively simple operations at each stage.
---
paper_title: Shape representation in the inferior temporal cortex of monkeys

paper_content:

Background: The inferior temporal cortex (IT) of the monkey has long been known to play an essential role in visual object recognition. Damage to this area results in severe deficits in perceptual learning and object recognition, without significantly affecting basic visual capacities. Consistent with these ablation studies is the discovery of IT neurons that respond to complex two-dimensional visual patterns, or objects such as faces or body parts. What is the role of these neurons in object recognition? Is such a complex configurational selectivity specific to biologically meaningful objects, or does it develop as a result of extensive exposure to any objects whose identification relies on subtle shape differences? If so, would IT neurons respond selectively to recently learned views or features of novel objects? The present study addresses this question by using combined psychophysical and electrophysiological experiments, in which monkeys learned to classify and recognize computer-generated three-dimensional objects. Results: A population of IT neurons was found that responded selectively to views of previously unfamiliar objects. The cells discharged maximally to one view of an object, and their response declined gradually as the object was rotated away from this preferred view. No selective responses were ever encountered for views that the animal systematically failed to recognize. Most neurons also exhibited orientation-dependent responses during view-plane rotations. Some neurons were found to be tuned around two views of the same object, and a very small number of cells responded in a view-invariant manner. For the five different objects that were used extensively during the training of the animals, and for which behavioral performance became view-independent, multiple cells were found that were tuned around different views of the same object. A number of view-selective units showed response invariance for changes in the size of the object or the position of its image within the parafovea. Conclusion: Our results suggest that IT neurons can develop a complex receptive field organization as a consequence of extensive training in the discrimination and recognition of objects. None of these objects had any prior meaning for the animal, nor did they resemble anything familiar in the monkey's environment. Simple geometric features did not appear to account for the neurons' selective responses. These findings support the idea that a population of neurons - each tuned to a different object aspect, and each showing a certain degree of invariance to image transformations - may, as an ensemble, encode at least some types of complex threedimensional objects. In such a system, several neurons may be active for any given vantage point, with a single unit acting like a blurred template for a limited neighborhood of a single view.
---
paper_title: Linear Object Classes and Image Synthesis From a Single Example Image

paper_content:

The need to generate new views of a 3D object from a single real image arises in several fields, including graphics and object recognition. While the traditional approach relies on the use of 3D models, simpler techniques are applicable under restricted conditions. The approach exploits image transformations that are specific to the relevant object class, and learnable from example views of other "prototypical" objects of the same class. In this paper, we introduce such a technique by extending the notion of linear class proposed by the authors (1992). For linear object classes, it is shown that linear transformations can be learned exactly from a basis set of 2D prototypical views. We demonstrate the approach on artificial objects and then show preliminary evidence that the technique can effectively "rotate" high-resolution face images from a single 2D view.
---
paper_title: View-based Models of 3D Object Recognition: Invariance to Imaging Transformations

paper_content:

This report describes the main features of a view-based model of object recognition. The model does not attempt to account for specific cortical structures; it tries to capture general properties to be expected in a biological architecture for object recognition. The basic module is a regularization network (RBF-like; see Poggio and Girosi, 1989; Poggio, 1990) in which each of the hidden units is broadly tuned to a specific view of the object to be recognized. The network output, which may be largely view independent, is first described in terms of some simple simulations. The following refinements and details of the basic module are then discussed: (1) some of the units may represent only components of views of the object--the optimal stimulus for the unit, its "center," is effectively a complex feature; (2) the units' properties are consistent with the usual description of cortical neurons as tuned to multidimensional optimal stimuli and may be realized in terms of plausible biophysical mechanisms; (3) in learning to recognize new objects, preexisting centers may be used and modified, but also new centers may be created incrementally so as to provide maximal view invariance; (4) modules are part of a hierarchical structure--the output of a network may be used as one of the inputs to another, in this way synthesizing increasingly complex features and templates; (5) in several recognition tasks, in particular at the basic level, a single center using view-invariant features may be sufficient.
---
paper_title: Becoming a “Greeble” Expert: Exploring Mechanisms for Face Recognition

paper_content:

Sensitivity to configural changes in face processing has been cited as evidence for face-exclusive mechanisms. Alternatively, general mechanisms could be fine-tuned by experience with homogeneous stimuli. We tested sensitivity to configural transformations for novices and experts with nonface stimuli (“Greebles”). Parts of transformed Greebles were identified via forced-choice recognition. Regardless of expertise level, the recognition of parts in the Studied configuration was better than in isolation, suggesting an object advantage. For experts, recognizing Greeble parts in a Transformed configuration was slower than in the Studied configuration, but only at upright. Thus, expertise with visually similar objects, not faces per se, may produce configural sensitivity. © 1997 Elsevier Science Ltd.
---
paper_title: Class similarity and viewpoint invariance in the recognition of 3D objects

paper_content:

In human vision, the processes and the representations involved in identifying specific individuals are frequently assumed to be different from those used for basic level classification, because classification is largely viewpoint-invariant, but identification is not. This assumption was tested in psychophysical experiments, in which objective similarity between stimuli (and, consequently, the level of their distinction) varied in a controlled fashion. Subjects were trained to discriminate between two classes of computer-generated three-dimensional objects, one resembling monkeys and the other, dogs. Both classes were defined by the same set of 56 parameters, which encoded sizes, shapes, and placement of the limbs, ears, snout, etc. Interpolation between parameter vectors of the class prototypes yielded shapes that changed smoothly between monkey and dog. Withinclass variation was induced in each trial by randomly perturbing all the parameters. After the subjects reached 90% correct performance on a fixed canonical view of each object, discrimination performance was tested for novel views that differed by up to 60 ° from the training view. In experiment 1 (in which the distribution of parameters in each class was unimodal) and in experiment 2 (bimodal classes), the stimuli differed only parametrically and consisted of the same geons (parts), yet were recognized virtually independently of viewpoint in the low-similarity condition. In experiment 3, the prototypes differed in their arrangement of geons, yet the subjects' performance depended significantly on viewpoint in the high-similarity condition. In all three experiments, higher interstimulus similarity was associated with an increase in the mean error rate and, for misorientation of up to 45 °, with an increase in the degree of viewpoint dependence. These results suggest that a geon-level difference between stimuli is neither strictly necessary nor always sufficient for viewpoint-invariant performance. Thus, basic and subordinate-level processes in visual recognition may be more closely related than previously thought.
---
paper_title: Representation And Recognition In Vision

paper_content:

Researchers have long sought to understand what the brain does when we see an object, what two people have in common when they see the same object, and what a "seeing" machine would need to have in common with a human visual system. Recent neurobiological and computational advances in the study of vision have now brought us close to answering these and other questions about representation. In Representation and Recognition in Vision, Shimon Edelman bases a comprehensive approach to visual representation on the notion of correspondence between proximal (internal) and distal similarities in objects. This leads to a computationally feasible and formally veridical representation of distal objects that addresses the needs of shape categorization and can be used to derive models of perceived similarity. Edelman first discusses the representational needs of various visual recognition tasks, and surveys current theories of representation in this context. He then develops a theory of representation that is related to Shepard's notion of second-order isomorphism between representations and their targets. Edelman goes beyond Shepard by specifying the conditions under which the representations can be made formally veridical. Edelman assesses his theory's performance in identification and categorization of 3D shapes and examines it in light of psychological and neurobiological data concerning the object-processing stream in primate vision. He also discusses the connections between his theory and other efforts to understand representation in the brain.
---
paper_title: Rotating objects to recognize them: A case study on the role of viewpoint dependency in the recognition of three-dimensional objects.

paper_content:

Successful object recognition is essential for finding food, identifying kin, and avoiding danger, as well as many other adaptive behaviors. To accomplish this feat, the visual system must reconstruct 3-D interpretations from 2-D “snapshots” falling on the retina. Theories of recognition address this process by focusing on the question of how object representations are encoded with respect to viewpoint. Although empirical evidence has been equivocal on this question, a growing body of surprising results, including those obtained in the experiments presented in this case study, indicates that recognition is often viewpoint dependent. Such findings reveal a prominent role for viewpointdependent mechanisms and provide support for themultiple-views approach, in which objects are encoded as a set of view-specific representations that are matched to percepts using normalization procedures.
---
paper_title: Optical imaging of functional organization in the monkey inferotemporal cortex.

paper_content:

To investigate the functional organization of object recognition, the technique of optical imaging was applied to the primate inferotemporal cortex, which is thought to be essential for object recognition. The features critical for the activation of single cells were first determined in unit recordings with electrodes. In the subsequent optical imaging, presentation of the critical features activated patchy regions around 0.5 millimeters in diameter, covering the site of the electrode penetration at which the critical feature had been determined. Because signals in optical imaging reflect average neuronal activities in the regions, the result directly indicates the regional clustering of cells responding to similar features.
---
paper_title: Generalization to Novel Images in Upright and Inverted Faces

paper_content:

An image of a face depends not only on its shape, but also on the viewing position, illumination conditions, and facial expression. Any face recognition system must overcome the changes in face appearance induced by these factors. To assess the ability of human vision to genralize across changes in illumination and pose of faces, we studied the performance of subjects in a discrimination task with eiether upright or inverted faces. Subjects first learned to discriminate among images of three faces, taken under fixed viewing position an illumination. They were then tested on images of the same faces taken under all combinations of four illuminations and five viewing positions. For upright faces, we found remarkably good generalization to novel conditions. For inverted faces, the generalization to novel views was significantly worse, although the performance on the training images was similar in both cases.
---
paper_title: Psychophysical support for a two-dimensional view interpolation theory of object recognition.

paper_content:

Does the human brain represent objects for recognition by storing a series of two-dimensional snapshots, or are the object models, in some sense, three-dimensional analogs of the objects they represent? One way to address this question is to explore the ability of the human visual system to generalize recognition from familiar to unfamiliar views of three-dimensional objects. Three recently proposed theories of object recognition--viewpoint normalization or alignment of three-dimensional models [Ullman, S. (1989) Cognition 32, 193-254], linear combination of two-dimensional views [Ullman, S. & Basri, R. (1990) Recognition by Linear Combinations of Models (Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge), A. I. Memo No. 1152], and view approximation [Poggio, T. & Edelman, S. (1990) Nature (London) 343, 263-266]--predict different patterns of generalization to unfamiliar views. We have exploited the conflicting predictions to test the three theories directly in a psychophysical experiment involving computer-generated three-dimensional objects. Our results suggest that the human visual system is better described as recognizing these objects by two-dimensional view interpolation than by alignment or other methods that rely on object-centered three-dimensional models.
---
paper_title: Shape representation in the inferior temporal cortex of monkeys

paper_content:

Background: The inferior temporal cortex (IT) of the monkey has long been known to play an essential role in visual object recognition. Damage to this area results in severe deficits in perceptual learning and object recognition, without significantly affecting basic visual capacities. Consistent with these ablation studies is the discovery of IT neurons that respond to complex two-dimensional visual patterns, or objects such as faces or body parts. What is the role of these neurons in object recognition? Is such a complex configurational selectivity specific to biologically meaningful objects, or does it develop as a result of extensive exposure to any objects whose identification relies on subtle shape differences? If so, would IT neurons respond selectively to recently learned views or features of novel objects? The present study addresses this question by using combined psychophysical and electrophysiological experiments, in which monkeys learned to classify and recognize computer-generated three-dimensional objects. Results: A population of IT neurons was found that responded selectively to views of previously unfamiliar objects. The cells discharged maximally to one view of an object, and their response declined gradually as the object was rotated away from this preferred view. No selective responses were ever encountered for views that the animal systematically failed to recognize. Most neurons also exhibited orientation-dependent responses during view-plane rotations. Some neurons were found to be tuned around two views of the same object, and a very small number of cells responded in a view-invariant manner. For the five different objects that were used extensively during the training of the animals, and for which behavioral performance became view-independent, multiple cells were found that were tuned around different views of the same object. A number of view-selective units showed response invariance for changes in the size of the object or the position of its image within the parafovea. Conclusion: Our results suggest that IT neurons can develop a complex receptive field organization as a consequence of extensive training in the discrimination and recognition of objects. None of these objects had any prior meaning for the animal, nor did they resemble anything familiar in the monkey's environment. Simple geometric features did not appear to account for the neurons' selective responses. These findings support the idea that a population of neurons - each tuned to a different object aspect, and each showing a certain degree of invariance to image transformations - may, as an ensemble, encode at least some types of complex threedimensional objects. In such a system, several neurons may be active for any given vantage point, with a single unit acting like a blurred template for a limited neighborhood of a single view.
---
paper_title: Sparse population coding of faces in the inferotemporal cortex.

paper_content:

How does the brain represent objects in the world? A proportion of cells in the temporal cortex of monkeys responds specifically to objects, such as faces, but the type of coding used by these cells is not known. Population analysis of two sets of such cells showed that information is carried at the level of the population and that this information relates, in the anterior inferotemporal cortex, to the physical properties of face stimuli and, in the superior temporal polysensory area, to other aspects of the faces, such as their familiarity. There was often sufficient information in small populations of neurons to identify particular faces. These results suggest that representations of complex stimuli in the higher visual areas may take the form of a sparse population code.
---
paper_title: Responses of Neurons in Inferior Temporal Cortex During MemoryGuided Visual Search

paper_content:

Chelazzi, Leonardo, John Duncan, Earl K. Miller, and Robert Desimone. Responses of neurons in inferior temporal cortex during memory-guided visual search. J. Neurophysiol. 80: 2918–2940, 1998. A typical scene will contain many different objects, few of which are relevant to behavior at any given moment. Thus attentional mechanisms are needed to select relevant objects for visual processing and control over behavior. We examined this role of attention in the inferior temporal cortex of macaque monkeys, using a visual search paradigm. While the monkey maintained fixation, a cue stimulus was presented at the center of gaze, followed by a blank delay period. After the delay, an array of two to five choice stimuli was presented extrafoveally, and the monkey was rewarded for detecting a target stimulus matching the cue. The behavioral response was a saccadic eye movement to the target in one version of the task and a lever release in another. The array was composed of one “good” stimulus (effective in driving the cell when presented alone) and one or more “poor” stimuli (ineffective in driving the cell when presented alone). Most cells showed higher delay activity after a good stimulus used as the cue than after a poor stimulus. The baseline activity of cells was also higher preceding a good cue, if the animal expected it to occur. This activity may depend on a top-down bias in favor of cells coding the relevant stimulus. When the choice array was presented, most cells showed suppressive interactions between the stimuli as well as strong attention effects. When the choice array was presented in the contralateral visual field, most cells initially responded the same, regardless of which stimulus was the target. However, within 150–200 ms of array onset, responses were determined by the target stimulus. If the target was the good stimulus, the response to the array became equal to the response to the good stimulus presented alone. If the target was a poor stimulus, the response approached the response to that stimulus presented alone. Thus the influence of the nontarget stimulus was eliminated. These effects occurred well in advance of the behavioral response. When the array was positioned with stimuli on opposite sides of the vertical meridian, the contralateral stimulus appeared to dominate the response, and this dominant effect could not be overcome by attention. Overall, the results support a “biased competition” model of attention, according to which 1 ) objects in the visual field compete for representation in the cortex, and 2 ) this competition is biased in favor of the behaviorally relevant object by virtue of “top-down” feedback from structures involved in working memory.
---
paper_title: High-Level Vision: Object Recognition and Visual Cognition

paper_content:

Object recognition: shape-based recognition what is recognition? why object recognition is difficult. Approaches to object recognition: invariant properties and feature spaces parts and structural descriptions the alignment approach which is the correct approach?. The alignment of pictorial descriptions: using corresponding features the use of multiple models for 3-D objects aligning pictorial descriptions transforming the image or the models? before and after alignment. The alignment of smooth bounding contours: the curvate method accuracy of the curvature method empirical testing. Recognition by the combination of views: modelling objects by view combinations objects with sharp edges using two views only using a single view the use of depth values summary of the basic scheme objects with smooth boundaries recognition by image combinations extensions to the view-combination scheme psychophysical and physiological evidence interim conclusions: recognition by multiple views. Classifications: classification and identification the role of object classification class-based processing using class prototypes pictorial classification evidence from psychology and biology are classes in the world or in our head? the organization of recognition memory. Image and model correspondence: feature correspondence contour matching correspondence-less methods correspondence processes in human vision model construction compensating for illumination changes. Segmentation and saliency: is segmentation feasible? bottom-up and top-down segmentation extracting globally salient structures saliency, selection, and completion what can bottom-up segmentation achieve? Visual cognition and visual routines: perceiving "inside" and "outside" spatial analysis by visual routines conclusions and open problems the elemental operations the assembly and storage of routines routines and recognition. Sequence seeking and counter streams - a model for visual cortex: the sequence-seeking scheme biological embodiment summary. Appendices: alignment by feature the curvature method errors of the curvature method locally affine matching definitions.
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: Neural Mechanisms of Visual Working Memory in Prefrontal Cortex of the Macaque

paper_content:

Prefrontal (PF) cells were studied in monkeys performing a delayed matching to sample task, which requires working memory. The stimuli were complex visual patterns and to solve the task, the monkeys had to discriminate among the stimuli, maintain a memory of the sample stimulus during the delay periods, and evaluate whether a test stimulus matched the sample presented earlier in the trial. PF cells have properties consistent with a role in all three of these operations. Approximately 25% of the cells responded selectively to different visual stimuli. Half of the cells showed heightened activity during the delay after the sample and, for many of these cells, the magnitude of delay activity was selective for different samples. Finally, more than half of the cells responded differently to the test stimuli depending on whether they matched the sample. Because inferior temporal (IT) cortex also is important for working memory, we compared PF cells with IT cells studied in the same task. Compared with IT cortex, PF responses were less often stimulus-selective but conveyed more information about whether a given test stimulus was a match to the sample. Furthermore, sample-selective delay activity in PF cortex was maintained throughout the trial even when other test stimuli intervened during the delay, whereas delay activity in IT cortex was disrupted by intervening stimuli. The results suggest that PF cortex plays a primary role in working memory tasks and may be a source of feedback inputs to IT cortex, biasing activity in favor of behaviorally relevant stimuli.
---
paper_title: Modeling Perceptual Grouping and Figure-Ground Segregation by Means of Active Reentrant Connections

paper_content:

Abstract ::: The segmentation of visual scenes is a fundamental process of early vision, but the underlying neural mechanisms are still largely unknown. Theoretical considerations as well as neurophysiological findings point to the importance in such processes of temporal correlations in neuronal activity. In a previous model, we showed that reentrant signaling among rhythmically active neuronal groups can correlate responses along spatially extended contours. We now have modified and extended this model to address the problems of perceptual grouping and figure-ground segregation in vision. A novel feature is that the efficacy of the connections is allowed to change on a fast time scale. This results in active reentrant connections that amplify the correlations among neuronal groups. The responses of the model are able to link the elements corresponding to a coherent figure and to segregate them from the background or from another figure in a way that is consistent with the so-called Gestalt laws.
---
paper_title: Neural correlates of feature selective memory and pop-out in extrastriate area V4

paper_content:

Neural activity in area V4 was examined to assess (1) whether the effects of attentive selection for stimulus features could be based on the memory of the feature, (2) whether dynamically changing the feature selection would cause activity associated with the newly selected stimuli to pop out, and (3) whether intrusion of more than one stimulus into the receptive field would disrupt the feature-selective activity. Rhesus monkeys were trained on several variations of a conditional orientation discrimination task. A differential activation of area V4 neurons was observed in the conditional discrimination task based on the presence of a match or a nonmatch between the conditional cue (a particular color or luminance) and the color or luminance of the receptive field stimulus. The differential activation was unchanged when the cue was removed and the animal had to remember its color (or luminance) to perform the task. When the cued feature was switched from one alternative to another in the middle of a trial the differential activation of neurons reversed over the course of 150–300 msec. If the stimulus in the receptive field contained the newly selected feature, V4 neurons became activated without a concomitant change in the stimulus in classical receptive field. Across the topographic map of V4 the activity associated with the newly selected stimuli popped out, whereas the activity of deselected stimuli faded to the background levels of other background objects. Evidence of a suppressive input from stimuli outside the classical receptive field was clear in only 3 of 24 neurons examined. Intrusion into the classical receptive field by a second stimulus resulted in a diminished difference between matching and nonmatching conditions. These physiological data suggest a major role for attentional control in the parallel processing of simple feature-selective differences.
---
paper_title: The wake-sleep algorithm for unsupervised neural networks

paper_content:

An unsupervised learning algorithm for a multilayer network of stochastic neurons is described. Bottom-up "recognition" connections convert the input into representations in successive hidden layers, and top-down "generative" connections reconstruct the representation in one layer from the representation in the layer above. In the "wake" phase, neurons are driven by recognition connections, and generative connections are adapted to increase the probability that they would reconstruct the correct activity vector in the layer below. In the "sleep" phase, neurons are driven by generative connections, and recognition connections are adapted to increase the probability that they would produce the correct activity vector in the layer above.
---
paper_title: State dependent activity in monkey visual cortex

paper_content:

Responses were recorded from isolated neurons in the visual cortex of rhesus monkeys while they performed an orientation match to sample task. In each trial the animal was first cued with randomly selected orientation, and then presented with a sequence of gratings whose orientations were randomly selected. The animal was required to release a switch when it saw a grating that matched the cued orientation. For some recordings the animal was given a tactile cue by having it feel the orientation of a grooved plate that it could not see. In other experiments the cue orientation was presented visually on the screen in front of the animal and then removed before the sequence of gratings was presented. Using this task it was possible to determine if a neuron's response to a particular orientation was affected by whether or not it was the orientation for which the animal was looking. Over half the neurons examined in V4 (110/192) responded differently to the visual stimuli when the animal was cued to look for different orientations. For some neurons responses to all stimuli were strong when the animal was cued to look for a particular orientation, but weak when the same stimuli were presented in trials where the animal had been cued to look for another orientation. This type of sensitivity was found in neurons recorded while the animal was given a tactile cue, and also in other neurons tested when a visual cue was used, suggesting that the activity was not of direct sensory origin. In support of this, neurons in V4 were not strongly affected when the animal felt the grooved plate while not performing the orientation matching task. The prevalence of behavioral effects that was found using the orientation matching task suggests that extraretinal signals represent a prominent component of the activity in V4 of the behaving monkey.
---
paper_title: Emergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces

paper_content:

Olshausen and Field (1996) applied the principle of independence maximization by sparse coding to extract features from natural images. This leads to the emergence of oriented linear filters that have simultaneous localization in space and in frequency, thus resembling Gabor functions and simple cell receptive fields. In this article, we show that the same principle of independence maximization can explain the emergence of phase- and shift-invariant features, similar to those found in complex cells. This new kind of emergence is obtained by maximizing the independence between norms of projections on linear subspaces (instead of the independence of simple linear filter outputs). The norms of the projections on such "independent feature subspaces" then indicate the values of invariant features.
---
paper_title: Emergence of simple-cell receptive field properties by learning a sparse code for natural images

paper_content:

THE receptive fields of simple cells in mammalian primary visual cortex can be characterized as being spatially localized, oriented1–4 and bandpass (selective to structure at different spatial scales), comparable to the basis functions of wavelet transforms5,6. One approach to understanding such response properties of visual neurons has been to consider their relationship to the statistical structure of natural images in terms of efficient coding7–12. Along these lines, a number of studies have attempted to train unsupervised learning algorithms on natural images in the hope of developing receptive fields with similar properties13–18, but none has succeeded in producing a full set that spans the image space and contains all three of the above properties. Here we investigate the proposal8,12 that a coding strategy that maximizes sparseness is sufficient to account for these properties. We show that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex. The resulting sparse image code provides a more efficient representation for later stages of processing because it possesses a higher degree of statistical independence among its outputs.
---
paper_title: Learning Invariance From Transformation Sequences

paper_content:

The visual system can reliably identify objects even when the retinal image is transformed considerably by commonly occurring changes in the environment. A local learning rule is proposed, which allows a network to learn to generalize across such transformations. During the learning phase, the network is exposed to temporal sequences of patterns undergoing the transformation. An application of the algorithm is presented in which the network learns invariance to shift in retinal position. Such a principle may be involved in the development of the characteristic shift invariance property of complex cells in the primary visual cortex, and also in the development of more complicated invariance properties of neurons in higher visual areas.
---
paper_title: Object classification using a fragment-based representation

paper_content:

The tasks of visual object recognition and classification are natural and effortless for biological visual systems, but exceedingly difficult to replicate in computer vision systems. This difficulty arises from the large variability in images of different objects within a class, and variability in viewing conditions. In this paper we describe a fragment-based method for object classification. In this approach objects within a class are represented in terms of common image fragments, that are used as building blocks for representing a large variety of different objects that belong to a common class, such as a face or a car. Optimal fragments are selected from a training set of images based on a criterion of maximizing the mutual information of the fragments and the class they represent. For the purpose of classification the fragments are also organized into types, where each type is a collection of alternative fragments, such as different hairline or eye regions for face classification. During classification, the algorithm detects fragments of the different types, and then combines the evidence for the detected fragments to reach a final decision. The algorithm verifies the proper arrangement of the fragments and the consistency of the viewing conditions primarily by the conjunction of overlapping fragments. The method is different from previous part-based methods in using class-specific overlapping object fragments of varying complexity, and in verifying the consistent arrangement of the fragments primarily by the conjunction of overlapping detected fragments. Experimental results on the detection of face and car views show that the fragment-based approach can generalize well to completely novel image views within a class while maintaining low mis-classification error rates. We briefly discuss relationships between the proposed method and properties of parts of the primate visual system involved in object perception.
---
paper_title: Categorical Perception: The Groundwork of Cognition

paper_content:

List of contributors Preface Introduction: psychophysical and cognitive aspects of categorical perception: a critical overview S. Harnad Part I. Psychophysical Foundations of Categorical Perception: 1. Categoric perception: some psychophysical models R. E. Pastore 2. Beyond the categorical/continuous distinction: a psychophysical approach to processing modes N. A. MacMillan Part II. Categorical Perception of Speech: 3. Phonetic category boundaries are flexible B. H. Repp and A. M. Liberman 4. Auditory, articulatory, and learning explanations of categorical perception in speech S. Rosen and P. Howell 5. On infant speech perception and the acquisition of language P. D. Eimas, J. L. Miller and P. W. Jusczyk Part III. Models for Speech Categorical Perception: 6. Neural models of speech perception: a case history R. E. Remez 7. On the categorization of speech sounds R. L. Diehl and K. R. Kluender 8. Categorical partition: a fuzzy-logical model of categorization behaviour D. W. Massaro Part IV. Categorical Perception in Other Modalities and Other Species: 9. Perceptual categories in vision and audition M. H. Bornstein 10. Categorical perception of sound signals: facts and hypotheses from animal studies G. Ehret 11. A naturalistic view of categorical perception C. T. Snowden 12. The special-mechanisms debate in speech research: categorization tests on animals and infants P. K. Kuhl 13. Brain mechanisms in categorical perception M. Wilson Part V. Psychophysiological Indices of Categorical Perception: 14. Electrophysiological indices of categorical perception for speech D. L. Molfese 15. Evoked potentials and color-defined categories D. Regan Part VI. Higher-order Categories: 16. Categorization processes and categorical perception D. L. Medin and L. W. Barsalou 17. Developmental changes in category structure F. C. Keil and M. H. Kelly 18. Spatial categories: the perception and conceptualization of spatial relations E. Bialystok and D. R. Olson Part VII. Cognitive Foundations: 19. Category induction and representation S. Harnad Author index Subject index.
---
paper_title: Global and fine information coded by single neurons in the temporal visual cortex

paper_content:

When we see a person's face, we can easily recognize their species, individual identity and emotional state. How does the brain represent such complex information? A substantial number of neurons in the macaque temporal cortex respond to faces1,2,3,4,5,6,7,8,9,10,11,12. However, the neuronal mechanisms underlying the processing ofcomplex information are not yet clear. Here we recorded the activity of single neurons in the temporal cortex of macaque monkeys while presenting visual stimuli consisting of geometric shapes, and monkey and human faces with various expressions. Information theory was used to investigate how well the neuronal responses could categorize the stimuli. We found that single neurons conveyed two different scales of facial information intheir firing patterns, starting at different latencies. Global information, categorizing stimuli as monkey faces, human faces or shapes, was conveyed in the earliest part of the responses. Fineinformation about identity or expression was conveyed later,beginning on average 51 ms after global information. We speculate that global information could be used as a ‘header’ to prepare destination areas for receiving more detailed information.
---
paper_title: Neurophysiology of shape processing

paper_content:

Recent physiological findings are reviewed and synthesized into a model of shape processing and object recognition. Gestalt laws (e.g. good continuation, closure) and ‘non-accidental’ image properties (e.g. colinear terminating lines) are resolved in prestriate visual cortex, (areas V2 and V3) to support the extraction of 2D shape boundaries. Processing of shape continues along a ventral route through inferior temporal (IT) cortex where a vast catalogue of 2D shape primitives is established. Each catalogue entry is size-specific (±0.5 log scale unit) and orientation-specific (±45°), but can generalize over position (±150 degree2). Several shape components are used to activate representations of the approximate appearance of one object type at one view, orientation and size. Subsequent generalization, first over orientation and size, then over view, and finally over object sub-component, is achieved in the anterior temporal cortex by combining descriptions of the same object from different orientations and views, through associative learning. This scheme provides a route to 3D object recognition through 2D shape description and reduces the problem of perceptual invariance to a series of independent analyses with an associative link established between the outputs. The system relies on parallel processing with computations performed in a series of hierarchical steps with relatively simple operations at each stage.
---
paper_title: View-based Models of 3D Object Recognition: Invariance to Imaging Transformations

paper_content:

This report describes the main features of a view-based model of object recognition. The model does not attempt to account for specific cortical structures; it tries to capture general properties to be expected in a biological architecture for object recognition. The basic module is a regularization network (RBF-like; see Poggio and Girosi, 1989; Poggio, 1990) in which each of the hidden units is broadly tuned to a specific view of the object to be recognized. The network output, which may be largely view independent, is first described in terms of some simple simulations. The following refinements and details of the basic module are then discussed: (1) some of the units may represent only components of views of the object--the optimal stimulus for the unit, its "center," is effectively a complex feature; (2) the units' properties are consistent with the usual description of cortical neurons as tuned to multidimensional optimal stimuli and may be realized in terms of plausible biophysical mechanisms; (3) in learning to recognize new objects, preexisting centers may be used and modified, but also new centers may be created incrementally so as to provide maximal view invariance; (4) modules are part of a hierarchical structure--the output of a network may be used as one of the inputs to another, in this way synthesizing increasingly complex features and templates; (5) in several recognition tasks, in particular at the basic level, a single center using view-invariant features may be sufficient.
---
paper_title: Generalization to Novel Images in Upright and Inverted Faces

paper_content:

An image of a face depends not only on its shape, but also on the viewing position, illumination conditions, and facial expression. Any face recognition system must overcome the changes in face appearance induced by these factors. To assess the ability of human vision to genralize across changes in illumination and pose of faces, we studied the performance of subjects in a discrimination task with eiether upright or inverted faces. Subjects first learned to discriminate among images of three faces, taken under fixed viewing position an illumination. They were then tested on images of the same faces taken under all combinations of four illuminations and five viewing positions. For upright faces, we found remarkably good generalization to novel conditions. For inverted faces, the generalization to novel views was significantly worse, although the performance on the training images was similar in both cases.
---

Title: Computational Models of Object Recognition in Cortex: A Review

Section 1: Object recognition is a difficult computational problem
Description 1: Introduce the problem of object recognition, its complexities, and its significance in the context of intelligent machines and neuroscience.

Section 2: Multiple tasks and strategies in object recognition
Description 2: Discuss different levels of specificity in object recognition (identification vs. categorization) and compare perspectives from computer vision and neuroscience.

Section 3: A continuum of recognition tasks along the trade-off between specificity and invariance
Description 3: Explain the theoretical spectrum between identification and categorization, emphasizing the trade-off between specificity and invariance.

Section 4: The same basic computational strategy can be used from identification to categorization
Description 4: Describe how the same computational strategies can adapt to different recognition tasks, highlighting multiple strategies in biological vision.

Section 5: Models and experiments
Description 5: Emphasize the necessity of models in understanding recognition and planning new experiments with a focus on isolated objects and recent developments.

Section 6: Models: Object-Centered and View-Based, Feedforward and Feedback
Description 6: Review the major categories of models for object recognition, including object-centered vs. view-based models and feedforward vs. feedback models.

Section 7: A basic module: feedforward and view-based
Description 7: Examine the support for feedforward and view-based models from neurophysiological data and their relevance in immediate recognition of 3D objects.

Section 8: Invariance and specificity
Description 8: Analyze the central issue of the invariance-specificity trade-off and how different transformations affect recognition performance.

Section 9: A basic module for identification and categorization: sparse population codes
Description 9: Discuss the concept of sparse population codes in object recognition and how it supports identification and categorization.

Section 10: A unified view
Description 10: Consider the implementation of different recognition tasks using a unified architectural approach, including hierarchical categorization schemes.

Section 11: Top-down and role of feedback
Description 11: Explore the role of top-down processing in recognition, including learning phases, attention control, and model matching.

Section 12: Learning
Description 12: Describe the learning mechanisms across different cortical layers and the development of invariant features.

Section 13: The time dimension
Description 13: Address the importance of incorporating the temporal aspect of recognition, including object and eye movement dynamics.

Section 14: Some key predictions
Description 14: Detail critical predictions based on the reviewed models and the potential implications of their experimental validation or falsification.