Visual Attention for Robotic Cognition: A Survey
---
paper_title: Bayesian inference and attentional modulation in the visual cortex

paper_content:

The responses of neurons in cortical areas V2 and V4 can be significantly modulated by attention to particular locations within an input image. We show that such effects emerge naturally when perception is viewed as a probabilistic inference process governed by Bayesian principles and implemented in hierarchical cortical networks. The proposed model can explain a rich variety of attention-related responses in cortical area V4 including multiplicative modulation of tuning curves, restoration of neural responses in the presence of distracting stimuli, and influence of attention on neighboring unattended locations. Our results suggest a new interpretation of attention as a cortical mechanism for reducing perceptual uncertainty by combining top-down task-relevant information with bottom-up sensory inputs in a probabilistic manner.
---
paper_title: Computational modelling of visual attention

paper_content:

Five important trends have emerged from recent work on computational models of focal visual attention that emphasize the bottom-up, image-based control of attentional deployment. First, the perceptual saliency of stimuli critically depends on the surrounding context. Second, a unique 'saliency map' that topographically encodes for stimulus conspicuity over the visual scene has proved to be an efficient and plausible bottom-up control strategy. Third, inhibition of return, the process by which the currently attended location is prevented from being attended again, is a crucial element of attentional deployment. Fourth, attention and eye movements tightly interplay, posing computational challenges with respect to the coordinate system used to control attention. And last, scene understanding and object recognition strongly constrain the selection of attended locations. Insights from these five key areas provide a framework for a computational and neurobiological understanding of visual attention.
---
paper_title: The CODE theory of visual attention: An integration of space-based and object-based attention

paper_content:

This article presents a theory that integrates space-based and object-based approaches to visual attention. The theory puts together M.P. van Oeffelen and P.G. Vos's (1982, 1983) COntour DEtector (CODE) theory of perceptual grouping by proximity with C. Bundesen's (1990) theory of visual attention (TVA). CODE provides input to TVA, accounting for spatially based between-object selection, and TVA converts the input to output, accounting for feature- and category-based within-object selection. CODE clusters nearby items into perceptual groups that are both perceptual objects and regions of space, thereby integrating object-based and space-based approaches to attention. The combined theory provides a quantitative account of the effects of grouping by proximity and distance between items on reaction time and accuracy data in 7 empirical situations that shaped the current literature on visual spatial attention.
---
paper_title: Guided Search 2.0 A revised model of visual search

paper_content:

An important component of routine visual behavior is the ability to find one item in a visual world filled with other, distracting items. This ability to performvisual search has been the subject of a large body of research in the past 15 years. This paper reviews the visual search literature and presents a model of human search behavior. Built upon the work of Neisser, Treisman, Julesz, and others, the model distinguishes between a preattentive, massively parallel stage that processes information about basic visual features (color, motion, various depth cues, etc.) across large portions of the visual field and a subsequent limited-capacity stage that performs other, more complex operations (e.g., face recognition, reading, object identification) over a limited portion of the visual field. The spatial deployment of the limited-capacity process is under attentional control. The heart of the guided search model is the idea that attentional deployment of limited resources isguided by the output of the earlier parallel processes. Guided Search 2.0 (GS2) is a revision of the model in which virtually all aspects of the model have been made more explicit and/or revised in light of new data. The paper is organized into four parts: Part 1 presents the model and the details of its computer simulation. Part 2 reviews the visual search literature on preattentive processing of basic features and shows how the GS2 simulation reproduces those results. Part 3 reviews the literature on the attentional deployment of limited-capacity processes in conjunction and serial searches and shows how the simulation handles those conditions. Finally, Part 4 deals with shortcomings of the model and unresolved issues.
---
paper_title: Computational Models of Visual Selective Attention. A Review

paper_content:

We review some of the major computational models of visual selective attention in terms of how they apply to psychological data and to theoretical concepts derived from experiments concerned with stimulus filtering, visual search and spatial cueing. We also review attempts to model neuropsychological disorders of visual object recognition and attention, including visual agnosia and neglect. We highlight how computational models can unite several dichotomies in psychological models: space-based versus object-based selection; early versus late selection; representational versus attentional accounts for neurological deficits. A comparison of the models exposes weaknesses and advantages in different accounts, but it also highlights that most models suggest that competitive interactions in visual selection are the basis for attentional effects in human data. There is also agreement across models on explaining neurological disorders as due to imbalances in competition following damage to certain areas of the brain. We propose that computational models clarify theoretical accounts of visual selective attention and integrate concepts across areas. The models provide a useful contribution to psychological research.
---
paper_title: Neural Mechanisms of Selective Visual Attention

paper_content:

The two basic phenomena that define the problem of visual attention can be illustrated in a simple example. Consider the arrays shown in each panel of Figure 1. In a typical experiment, before the arrays were presented, subjects would be asked to report letters appearing in one color (targets, here black letters), and to disregard letters in the other color (nontargets, here white letters). The array would then be briefly flashed, and the subjects, without any opportunity for eye movements, would give their report. The display mimics our. usual cluttered visual environment: It contains one or more objects that are relevant to current behavior, along with others that are irrelevant. The first basic phenomenon is limited capacity for processing information. At any given time, only a small amount of the information available on the retina can be processed and used in the control of behavior. Subjectively, giving attention to any one target leaves less available for others. In Figure 1, the probability of reporting the target letter N is much lower with two accompa­ nying targets (Figure la) than with none (Figure Ib). The second basic phenomenon is selectivity-the ability to filter out un­ wanted information. Subjectively, one is aware of attended stimuli and largely unaware of unattended ones. Correspondingly, accuracy in identifying an attended stimulus may be independent of the number of nontargets in a display (Figure la vs Ie) (see Bundesen 1990, Duncan 1980).
---
paper_title: Responses of Neurons in Inferior Temporal Cortex During MemoryGuided Visual Search

paper_content:

Chelazzi, Leonardo, John Duncan, Earl K. Miller, and Robert Desimone. Responses of neurons in inferior temporal cortex during memory-guided visual search. J. Neurophysiol. 80: 2918–2940, 1998. A typical scene will contain many different objects, few of which are relevant to behavior at any given moment. Thus attentional mechanisms are needed to select relevant objects for visual processing and control over behavior. We examined this role of attention in the inferior temporal cortex of macaque monkeys, using a visual search paradigm. While the monkey maintained fixation, a cue stimulus was presented at the center of gaze, followed by a blank delay period. After the delay, an array of two to five choice stimuli was presented extrafoveally, and the monkey was rewarded for detecting a target stimulus matching the cue. The behavioral response was a saccadic eye movement to the target in one version of the task and a lever release in another. The array was composed of one “good” stimulus (effective in driving the cell when presented alone) and one or more “poor” stimuli (ineffective in driving the cell when presented alone). Most cells showed higher delay activity after a good stimulus used as the cue than after a poor stimulus. The baseline activity of cells was also higher preceding a good cue, if the animal expected it to occur. This activity may depend on a top-down bias in favor of cells coding the relevant stimulus. When the choice array was presented, most cells showed suppressive interactions between the stimuli as well as strong attention effects. When the choice array was presented in the contralateral visual field, most cells initially responded the same, regardless of which stimulus was the target. However, within 150–200 ms of array onset, responses were determined by the target stimulus. If the target was the good stimulus, the response to the array became equal to the response to the good stimulus presented alone. If the target was a poor stimulus, the response approached the response to that stimulus presented alone. Thus the influence of the nontarget stimulus was eliminated. These effects occurred well in advance of the behavioral response. When the array was positioned with stimuli on opposite sides of the vertical meridian, the contralateral stimulus appeared to dominate the response, and this dominant effect could not be overcome by attention. Overall, the results support a “biased competition” model of attention, according to which 1 ) objects in the visual field compete for representation in the cortex, and 2 ) this competition is biased in favor of the behaviorally relevant object by virtue of “top-down” feedback from structures involved in working memory.
---
paper_title: A saliency-based search mechanism for overt and covert shifts of visual attention

paper_content:

Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested.
---
paper_title: Bayesian inference and attentional modulation in the visual cortex

paper_content:

The responses of neurons in cortical areas V2 and V4 can be significantly modulated by attention to particular locations within an input image. We show that such effects emerge naturally when perception is viewed as a probabilistic inference process governed by Bayesian principles and implemented in hierarchical cortical networks. The proposed model can explain a rich variety of attention-related responses in cortical area V4 including multiplicative modulation of tuning curves, restoration of neural responses in the presence of distracting stimuli, and influence of attention on neighboring unattended locations. Our results suggest a new interpretation of attention as a cortical mechanism for reducing perceptual uncertainty by combining top-down task-relevant information with bottom-up sensory inputs in a probabilistic manner.
---
paper_title: Visual feature integration theory: Past, present, and future

paper_content:

Visual feature integration theory was one of the most influential theories of visual information processing in the last quarter of the 20th century. This article provides an exposition of the theory and a review of the associated data. In the past much emphasis has been placed on how the theory explains performance in various visual search tasks. The relevant literature is discussed and alternative accounts are described. Amendments to the theory are also set out. Many other issues concerning internal processes and representations implicated by the theory are reviewed. The article closes with a synopsis of what has been learned from consideration of the theory, and it is concluded that some of the issues may remain intractable unless appropriate neuroscientific investigations are carried out.
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: Guided search: An alternative to the feature integration model for visual search.

paper_content:

Subjects searched sets of items for targets defined by conjunctions of color and form, color and orientation, or color and size. Set size was varied and reaction times (RT) were measured. For many unpracticed subjects, the slopes of the resulting RT X Set Size functions are too shallow to be consistent with Treisman's feature integration model, which proposes serial, self-terminating search for conjunctions. Searches for triple conjunctions (Color X Size X Form) are easier than searches for standard conjunctions and can be independent of set size. A guided search model similar to Hoffman's (1979) two-stage model can account for these data. In the model, parallel processes use information about simple features to guide attention in the search for conjunctions. Triple conjunctions are found more efficiently than standard conjunctions because three parallel processes can guide attention more effectively than two.
---
paper_title: Modeling visual attention via selective tuning

paper_content:

A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system.
---
paper_title: Computational modelling of visual attention

paper_content:

Five important trends have emerged from recent work on computational models of focal visual attention that emphasize the bottom-up, image-based control of attentional deployment. First, the perceptual saliency of stimuli critically depends on the surrounding context. Second, a unique 'saliency map' that topographically encodes for stimulus conspicuity over the visual scene has proved to be an efficient and plausible bottom-up control strategy. Third, inhibition of return, the process by which the currently attended location is prevented from being attended again, is a crucial element of attentional deployment. Fourth, attention and eye movements tightly interplay, posing computational challenges with respect to the coordinate system used to control attention. And last, scene understanding and object recognition strongly constrain the selection of attended locations. Insights from these five key areas provide a framework for a computational and neurobiological understanding of visual attention.
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: A biased competition computational model of spatial and object-based attention mediating active visual search

paper_content:

Abstract Colour is more important than form in driving the scan path during active visual feature conjunction search. A model shows how feature-based attention in V4 provides information to LIP allowing it to represent behaviourally relevant locations and attract attention to target coloured locations. Attentional effects in V4 develop such that an initial spatial attention focus, scaled according to stimulus density, becomes object-based later in the response. Thus, this is the first model to replicate, with temporal precision, different attentional effects observed in single-cell studies of V4 and IT, in addition to the systems level scan path behaviour.
---
paper_title: Configurational effects on the enumeration of dots: Counting by groups

paper_content:

The processing time for quantifying numerosity of two-dimensional dot patterns was investigated as a function of both number of dots and relative proximity between dots. A cluster algorithm (CODE) was first developed as a formal model of how human subjects organize neighboring dots into groups. CODE-based predictions of grouping effects on number processing latencies were then tested with patterns consisting of n dots (range n = 13–23). The results largely confirmed CODE-based predictions and thereby indicated that large collections of dots are preferably counted by groups. Small (n ≤ 5) groups are subitized and their partial results are summed to a running total. Based on criteria other than dot proximity, large (n > 5), proximity-based groups are subdivided into smaller groups of two or three dots, which are again subitized.
---
paper_title: Attending to Visual Motion

paper_content:

Visual motion analysis has focused on decomposing image sequences into their component features. There has been little success at re-combining those features into moving objects. Here, a novel model of attentive visual motion processing is presented that addresses both decomposition of the signal into constituent features as well as the re-combination, or binding, of those features into wholes. A new feed-forward motion-processing pyramid is presented motivated by the neurobiology of primate motion processes. On this structure the Selective Tuning (ST) model for visual attention is demonstrated. There are three main contributions: (1) a new feed-forward motion processing hierarchy, the first to include a multi-level decomposition with local spatial derivatives of velocity; (2) examples of how ST operates on this hierarchy to attend to motion and to localize and label motion patterns; and (3) a new solution to the feature binding problem sufficient for grouping motion features into coherent object motion. Binding is accomplished using a top-down selection mechanism that does not depend on a single location-based saliency representation. resentation.
---
paper_title: A Feature-Integration Theory of Attention

paper_content:

A new hypothesis about the role of focused attention is proposed. The feature-integration theory of attention suggests that attention must be directed serially to each stimulus in a display whenever conjunctions of more than one separable feature are needed to characterize or distinguish the possible objects presented. A number of predictions were tested in a variety of paradigms including visual search, texture segregation, identification and localization, and using both separable dimensions (shape and color) and local elements or parts of figures (lines, curves, etc. in letters) as the features to be integrated into complex wholes. The results were in general consistent with the hypothesis. They offer a new set of criteria for distinguishing separable from integral features and a new rationale for predicting which tasks will show attention limits and which will not.
---
paper_title: Active Object Recognition Integrating Attention and Viewpoint Control

paper_content:

We present an active object recognition strategy which combines the use of an attention mechanism for focusing the search for a 3D object in a 2D image, with a viewpoint control strategy for disambiguating recovered object features. The attention mechanism consists of a probabilistic search through a hierarchy of predicted feature observations, taking objects into a set of regions classified according to the shapes of their bounding contours. We motivate the use of image regions as a focus-feature and compare their uncertainty in inferring objects with the uncertainty of more commonly used features such as lines or corners. If the features recovered during the attention phase do not provide a unique mapping to the 3D object being searched, the probabilistic feature hierarchy can be used to guide the camera to a new viewpoint from where the object can be disambiguated. The power of the underlying representation is its ability to unify these object recognition behaviors within a single framework. We present the approach in detail and evaluate its performance in the context of a project providing robotic aids for the disabled.
---
paper_title: A saliency-based search mechanism for overt and covert shifts of visual attention

paper_content:

Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested.
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: A saliency-based search mechanism for overt and covert shifts of visual attention

paper_content:

Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested.
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: Modeling visual attention via selective tuning

paper_content:

A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system.
---
paper_title: Computational modelling of visual attention

paper_content:

Five important trends have emerged from recent work on computational models of focal visual attention that emphasize the bottom-up, image-based control of attentional deployment. First, the perceptual saliency of stimuli critically depends on the surrounding context. Second, a unique 'saliency map' that topographically encodes for stimulus conspicuity over the visual scene has proved to be an efficient and plausible bottom-up control strategy. Third, inhibition of return, the process by which the currently attended location is prevented from being attended again, is a crucial element of attentional deployment. Fourth, attention and eye movements tightly interplay, posing computational challenges with respect to the coordinate system used to control attention. And last, scene understanding and object recognition strongly constrain the selection of attended locations. Insights from these five key areas provide a framework for a computational and neurobiological understanding of visual attention.
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: Object-based visual attention for computer vision

paper_content:

In this paper, a novel model of object-based visual attention extending Duncan's Integrated Competition Hypothesis [Phil. Trans. R. Soc. London B 353 (1998) 1307-1317] is presented. In contrast to the attention mechanisms used in most previous machine vision systems which drive attention based on the spatial location hypothesis, the mechanisms which direct visual attention in our system are object-driven as well as feature-driven. The competition to gain visual attention occurs not only within an object but also between objects. For this purpose, two new mechanisms in the proposed model are described and analyzed in detail. The first mechanism computes the visual salience of objects and groupings; the second one implements the hierarchical selectivity of attentional shifts. The results of the new approach on synthetic and natural images are reported.
---
paper_title: The FeatureGate Model of Visual Selection

paper_content:

The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate.
---
paper_title: Attending to Visual Motion

paper_content:

Visual motion analysis has focused on decomposing image sequences into their component features. There has been little success at re-combining those features into moving objects. Here, a novel model of attentive visual motion processing is presented that addresses both decomposition of the signal into constituent features as well as the re-combination, or binding, of those features into wholes. A new feed-forward motion-processing pyramid is presented motivated by the neurobiology of primate motion processes. On this structure the Selective Tuning (ST) model for visual attention is demonstrated. There are three main contributions: (1) a new feed-forward motion processing hierarchy, the first to include a multi-level decomposition with local spatial derivatives of velocity; (2) examples of how ST operates on this hierarchy to attend to motion and to localize and label motion patterns; and (3) a new solution to the feature binding problem sufficient for grouping motion features into coherent object motion. Binding is accomplished using a top-down selection mechanism that does not depend on a single location-based saliency representation. resentation.
---
paper_title: The CODE theory of visual attention: An integration of space-based and object-based attention

paper_content:

This article presents a theory that integrates space-based and object-based approaches to visual attention. The theory puts together M.P. van Oeffelen and P.G. Vos's (1982, 1983) COntour DEtector (CODE) theory of perceptual grouping by proximity with C. Bundesen's (1990) theory of visual attention (TVA). CODE provides input to TVA, accounting for spatially based between-object selection, and TVA converts the input to output, accounting for feature- and category-based within-object selection. CODE clusters nearby items into perceptual groups that are both perceptual objects and regions of space, thereby integrating object-based and space-based approaches to attention. The combined theory provides a quantitative account of the effects of grouping by proximity and distance between items on reaction time and accuracy data in 7 empirical situations that shaped the current literature on visual spatial attention.
---
paper_title: A computational model of object-based selective visual attention mechanism in visual information acquisition

paper_content:

A computational model of object-based selective visual attention is introduced. What are the units of selective attention is always the focus of selective attention. Most existing computational model of selective attention is space-based, but more and more neuroscience experiments indicate that it is object-based. For object-based attention, how to define "object" is the biggest difficulty and it is generally accepted that the "object" refers to "perceptual object" which is created by Gestalt rules. However, Gestalt rules are very difficult to be calculated in images. The model presented in this paper is comprised of two independent components. One component is object-based segmentation by using perceptual color cues, the other is the generation of a object-based saliency-map in terms of the inhibition of return mechanism. Lastly, the results of experiments show that the model is valid.
---
paper_title: A saliency-based search mechanism for overt and covert shifts of visual attention

paper_content:

Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested.
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: Guided search: An alternative to the feature integration model for visual search.

paper_content:

Subjects searched sets of items for targets defined by conjunctions of color and form, color and orientation, or color and size. Set size was varied and reaction times (RT) were measured. For many unpracticed subjects, the slopes of the resulting RT X Set Size functions are too shallow to be consistent with Treisman's feature integration model, which proposes serial, self-terminating search for conjunctions. Searches for triple conjunctions (Color X Size X Form) are easier than searches for standard conjunctions and can be independent of set size. A guided search model similar to Hoffman's (1979) two-stage model can account for these data. In the model, parallel processes use information about simple features to guide attention in the search for conjunctions. Triple conjunctions are found more efficiently than standard conjunctions because three parallel processes can guide attention more effectively than two.
---
paper_title: A model of active visual search with object-based attention guiding scan paths

paper_content:

When a monkey searches for a colour and orientation feature conjunction target, the scan path is guided to target coloured locations in preference to locations containing the target orientation [Vision Res. 38 (1998b) 1805]. An active vision model, using biased competition, is able to replicate this behaviour. As object-based attention develops in extrastriate cortex, featural information is passed to posterior parietal cortex (LIP), enabling it to represent behaviourally relevant locations [J. Neurophysiol. 76 (1996) 2841] and guide the scan path. Attention evolves from an early spatial effect to being object-based later in the response of the model neurons, as has been observed in monkey single cell recordings. This is the first model to reproduce these effects with temporal precision and is reported here at the systems level allowing the replication of psychophysical scan paths.
---
paper_title: Object-based Visual Attention: a Model for a Behaving Robot

paper_content:

One of the first steps of any visual system is that of locating suitable interest points, "salient regions", in the scene, to detect events, and eventually to direct gaze toward these locations. In the last few years, object-based visual attention models have received an increasing interest in the literature, the problem, in this case, being that of creating a model of "objecthood" that eventually guides a saliency mechanism. We propose here an object-based model of visual attention and show its instantiation on a humanoid robot. The robot employs action to learn and define its own concept of objecthood.
---
paper_title: Selective Attention and the Organization of Visual Information

paper_content:

Medical Research Council, Applied Psychology Unit, Cambridge, England Theories of visual attention deal with the limit on our ability to see (and later report) several things at once. These theories fall into three broad classes. Objectbased theories propose a limit on the number of separate objects :that can be perceived simultaneously. Discrimination-based theories propose a limit on the number of separate discriminations that can be made. Space-based theories propose a limit on the spatial area from which information can be taken up. To distinguish these views, the present experiments used small (< 1 °), brief, foveal displays, each consisting of two overlapping objects (a box with a line struck through it). It was found that two judgments that concern the same object can be made simultaneously without loss of accuracy, whereas two judgments that concern different objects cannot. Neither the similarity nor the difficulty of required discriminations, nor the spatial distribution of information, could account for the results. The experiments support a view in which parallel, preattentive processes serve to segment the field into separate objects, followed by a pfocess of focal attention that deals with only one object at a time. This view is also able to account for results taken to support both discrimination-based and space-based theories. Object-Based Theories of Visual Attention Theories of visual attention are concerned with the limit on our ability to see (and later report) several things at once. This article deals with what I call object-based theories (e.g., Neisser, 1967), which propose that this limit concerns the number of separate objects that can be seen. Here some predictions of this view are tested, and object-based theories are contrasted with discrimination-based theories (e.g., Allport, 1971, 1980) and with spaced-based theories (e.g., Hoffman & Nelson, 1981; Posner, Snyder, & Davidson, 1980). The work of Neisser (1967) illustrates the object-based approach. Neisser (1967) proposed that perceptual analysis of the visual world takes place in two successive stages. The first, preattentive, stage segments the field into separate objects on the basis of such Gestalt properties as spatial proximity, continuity of contour, shared color or movement, and so on. The second stage, focal attention, analyzes a particular object in more detail. Neisser (1967) supposed that, whereas preat
---
paper_title: Objects and attention: the state of the art

paper_content:

What are the units of attention? In addition to standard models holding that attention can select spatial regions and visual features, recent work suggests that in some cases attention can directly select discrete objects. This paper reviews the state of the art with regard to such ‘object-based’ attention, and explores how objects of attention relate to locations, reference frames, perceptual groups, surfaces, parts, and features. Also discussed are the dynamic aspects of objecthood, including the question of how attended objects are individuated in time, and the possibility of attending to simple dynamic motions and events. The final sections of this review generalize these issues beyond vision science, to other modalities and fields such as auditory objects of attention and the infant’s ‘object concept’. q 2001 Elsevier Science B.V. All rights reserved.
---
paper_title: Object-based visual attention for computer vision

paper_content:

In this paper, a novel model of object-based visual attention extending Duncan's Integrated Competition Hypothesis [Phil. Trans. R. Soc. London B 353 (1998) 1307-1317] is presented. In contrast to the attention mechanisms used in most previous machine vision systems which drive attention based on the spatial location hypothesis, the mechanisms which direct visual attention in our system are object-driven as well as feature-driven. The competition to gain visual attention occurs not only within an object but also between objects. For this purpose, two new mechanisms in the proposed model are described and analyzed in detail. The first mechanism computes the visual salience of objects and groupings; the second one implements the hierarchical selectivity of attentional shifts. The results of the new approach on synthetic and natural images are reported.
---
paper_title: The FeatureGate Model of Visual Selection

paper_content:

The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate.
---
paper_title: A Feature-Integration Theory of Attention

paper_content:

A new hypothesis about the role of focused attention is proposed. The feature-integration theory of attention suggests that attention must be directed serially to each stimulus in a display whenever conjunctions of more than one separable feature are needed to characterize or distinguish the possible objects presented. A number of predictions were tested in a variety of paradigms including visual search, texture segregation, identification and localization, and using both separable dimensions (shape and color) and local elements or parts of figures (lines, curves, etc. in letters) as the features to be integrated into complex wholes. The results were in general consistent with the hypothesis. They offer a new set of criteria for distinguishing separable from integral features and a new rationale for predicting which tasks will show attention limits and which will not.
---
paper_title: Object- and space-based visual attention: An integrated framework for autonomous robots

paper_content:

This paper argues that the object- and space-based modes of visual attention can be naturally integrated in a common mathematical framework. In an earlier work we have proposed a mathematical model of visual attention for robotic system exploiting the knowledge of visual attention mechanism of the primates. This paper investigates on the validity of the proposed model for robotic systems through experimentation on a real robot. The paper sheds light on a number of real world issues involved with the design of visual attention system for physically embodied robots and explains how the proposed Bayesian model of visual attention addresses these issues. The object- and space-based modes of visual attention are naturally integrated in the model and is reflected in the sequential Monte Carlo implementation of the model on a real robot.
---
paper_title: Re-mapping of visual saliency in overt attention: A particle filter approach for robotic systems

paper_content:

‘Saliency map’, a scalar, two-dimensional representation of visual saliency, is at the center of almost all of the existing models of visual attention. Attention is directed toward the most salient location in the saliency map. The mechanism of ‘inhibition of return’ (IOR) suppresses the saliency of recently attended location and thereby enabling the shifts of attention toward different locations in the saliency map in order of decreasing saliency. This process performs well as longs as the attention is directed covertly. For overt attention with head movements, which is practically the case in robotic applications, the visual saliency as well as the frame of reference in which the IOR is expressed change after every head movement. These pose a set of computational challenges in implementing attention behavior and IOR. This paper argues that the re-mapping of visual saliency and dynamic shift of IOR emerge naturally in a particle filter based framework of visual attention. Experiments on a real camera head validate the arguments.
---
paper_title: Object- and space-based visual attention: An integrated framework for autonomous robots

paper_content:

This paper argues that the object- and space-based modes of visual attention can be naturally integrated in a common mathematical framework. In an earlier work we have proposed a mathematical model of visual attention for robotic system exploiting the knowledge of visual attention mechanism of the primates. This paper investigates on the validity of the proposed model for robotic systems through experimentation on a real robot. The paper sheds light on a number of real world issues involved with the design of visual attention system for physically embodied robots and explains how the proposed Bayesian model of visual attention addresses these issues. The object- and space-based modes of visual attention are naturally integrated in the model and is reflected in the sequential Monte Carlo implementation of the model on a real robot.
---
paper_title: VISUAL SEARCH HAS MEMORY

paper_content:

By monitoring subjects' eye movements during a visual search task, we examined the possibility that the mechanism responsible for guiding attention during visual search has no memory for which locations have already been examined. Subjects did reexamine some items during their search, but the pattern of revisitations did not fit the predictions of the memoryless search model. In addition, a large proportion of the refixations were directed at the target, suggesting that the revisitations were due to subjects' remembering which items had not been adequately identified. We also examined the patterns of fixations and compared them with the predictions of a memoryless search model. Subjects' fixation patterns showed an increasing hazard function, whereas the memoryless model predicts a flat function. Lastly, we found no evidence suggesting that fixations were guided by amnesic covert scans that scouted the environment for new items during fixations. Results do not support the claims of the memoryless search m...
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: Fast and Robust Generation of Feature Maps for Region-Based Visual Attention

paper_content:

Visual attention is one of the important phenomena in biological vision which can be followed to achieve more efficiency, intelligence, and robustness in artificial vision systems. This paper investigates a region-based approach that performs pixel clustering prior to the processes of attention in contrast to late clustering as done by contemporary methods. The foundation steps of feature map construction for the region-based attention model are proposed here. The color contrast map is generated based upon the extended findings from the color theory, the symmetry map is constructed using a novel scanning-based method, and a new algorithm is proposed to compute a size contrast map as a formal feature channel. Eccentricity and orientation are computed using the moments of obtained regions and then saliency is evaluated using the rarity criteria. The efficient design of the proposed algorithms allows incorporating five feature channels while maintaining a processing rate of multiple frames per second. Another salient advantage over the existing techniques is the reusability of the salient regions in the high-level machine vision procedures due to preservation of their shapes and precise locations. The results indicate that the proposed model has the potential to efficiently integrate the phenomenon of attention into the main stream of machine vision and systems with restricted computing resources such as mobile robots can benefit from its advantages.
---
paper_title: The iCub cognitive humanoid robot: an open-system research platform for enactive cognition

paper_content:

This paper describes a multi-disciplinary initiative to promote collaborative research in enactive artificial cognitive systems by developing the iCub : a open-systems 53 degree-of-freedom cognitive humanoid robot. At 94 cm tall, the iCub is the same size as a three year-old child. It will be able to crawl on all fours and sit up, its hands will allow dexterous manipulation, and its head and eyes are fully articulated. It has visual, vestibular, auditory, and haptic sensory capabilities. As an open system, the design and documentation of all hardware and software is licensed under the Free Software Foundation GNU licences so that the system can be freely replicated and customized. We begin this paper by outlining the enactive approach to cognition, drawing out the implications for phylogenetic configuration, the necessity for ontogenetic development, and the importance of humanoid embodiment. This is followed by a short discussion of our motivation for adopting an open-systems approach. We proceed to describe the iCub's mechanical and electronic specifications, its software architecture, its cognitive architecture. We conclude by discussing the iCub phylogeny, i.e. the robot's intended innate abilities, and an scenario for ontogenesis based on human neonatal development.
---
paper_title: Visual attention models-object-based theory of visual attention

paper_content:

According to the object-based theory, visual attention can be divided within a single object more efficiently than between separate objects. In this paper, the object-based theory of visual attention was validated through a comparison with the spatially based view, suggesting that visual attention is directed to unparsed regions of space. In testing this theory, various perceptual judgements of location were given by subjects in response to objects. Among the variables in presentation were: the direction of the objects, the distance between two locations, the number of objects involved and the display condition (joined, congruent, incongruent, etc.). In other words, the single- and two-object displays were compared by investigating the effects of these experimental factors. As a result, the object-based theory of visual attention was found to be applicable to hexagons of equal convexity, but not applicable to the other forms within the range of this experiment.
---
paper_title: A saliency-based search mechanism for overt and covert shifts of visual attention

paper_content:

Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested.
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: Spatial attention and saccadic camera motion

paper_content:

An important aspect of computer-controlled camera motion systems is that of the generation of saccadic movements, which shift the camera gaze quickly from one fixation position to another. Recent psychophysical experiments suggest that there exists a causal connection between spatial shifts in visual attention and the production of saccadic eye movements in humans. Motivated by this experimental evidence, we propose a winner-take-all based model of exogenous spatial attention, involving both sustained and transient feature detection channels, and link it to the targetting and triggering of saccadic eye movements. We show that this model accounts for a range of oculomotor phenomena observed in human subjects. We describe the application of this model to a robotic camera gaze control system.
---
paper_title: Object-based Visual Attention: a Model for a Behaving Robot

paper_content:

One of the first steps of any visual system is that of locating suitable interest points, "salient regions", in the scene, to detect events, and eventually to direct gaze toward these locations. In the last few years, object-based visual attention models have received an increasing interest in the literature, the problem, in this case, being that of creating a model of "objecthood" that eventually guides a saliency mechanism. We propose here an object-based model of visual attention and show its instantiation on a humanoid robot. The robot employs action to learn and define its own concept of objecthood.
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: A Reactive vision system: Active-dynamic saliency

paper_content:

We develop an architecture for reactive visual analysis of dynamic scenes. We specify a minimal set of system features based upon biological observations. We implement feature on a processing network based around an active stereo vision mechanism. Active rectification and mosaicing allows static stereo algorithms to operate on the active platform. Foveal zero disparity operations permit attended object extraction and ensures coordinated stereo fixation upon visual surfaces. Activedynamic inhibition of return, and task dependent biasing result in a flexible, preemptive and retrospective system that responds to unique visual stimuli and is capable of top-down modulation of attention towards regions and cues relevant to tasks.
---
paper_title: Image Mapping and Visual Attention on a Sensory Ego-Sphere

paper_content:

The Sensory Ego-Sphere (SES) is a short-term memory for a robot in the form of an egocentric, tessellated, spherical, sensory-motor map of the robot's locale. This paper reports on: (1) the mapping to the SES of images with much higher resolution than the SES itself, and (2) the processing of visual attention on the SES. Described is a procedure to store spatially-overlapping imagery in the SES database and to generate and update a spherical composite of its visual contents. The composite image serves both as a visual map of the locale and as a representation of the local contents of the underlying full-resolution imagery. Visual attention enables fast alignment of overlapping images without warping or position optimization, since an attentional point (AP) on the composite typically corresponds to one on each of the collocated regions in the images. Such alignment speeds analysis of the multiple images of the area. Compositing and attention were performed two ways and compared: (1) APs were computed directly on the composite and not on the full resolution images until the time of retrieval. (2) The attentional operator was applied to all incoming imagery. Then the locations and values of the APs on the composite were computed as a function of those points within the corresponding regions of the collocated images. It was found that although the second method was slower, it produced consistent and, thereby, more useful APs.
---
paper_title: Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

paper_content:

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.
---
paper_title: Multimodal saliency-based bottom-up attention a framework for the humanoid robot iCub

paper_content:

This work presents a multimodal bottom-up attention system for the humanoid robot iCub where the robot's decisions to move eyes and neck are based on visual and acoustic saliency maps. We introduce a modular and distributed software architecture which is capable of fusing visual and acoustic saliency maps into one egocentric frame of reference. This system endows the iCub with an emergent exploratory behavior reacting to combined visual and auditory saliency. The developed software modules provide a flexible foundation for the open iCub platform and for further experiments and developments, including higher levels of attention and representation of the peripersonal space.
---
paper_title: An adaptive detection/attention mechanism for real time robot operation

paper_content:

During the lifetime of a mobile robot, the number and complexity of the stimuli it receives may be quite high. Therefore, the construction of a detection system considering the whole sensorial space is usually not a viable proposition when aiming for real time operation. It becomes necessary to build some kind of sensorial hierarchy map to put some order into how detectors are applied. This is what is usually called an attentional system, and it provides a framework for applying detectors in a more efficient manner. In this paper, an architecture for developing attentional functions for robots that must operate in real time in dynamic environments is presented. This architecture is based on the concept of attentor and it allows for the real time adaptation to the environment and tasks to be performed in a natural manner. One of the main requirements imposed on the design of the architecture was the capability of handling different sensorial modalities and attentional streams in a transparent manner while, at the same time, being able to progressively create more complex attentional structures. The architecture is particularized for its implementation in a real robot.
---
paper_title: The FeatureGate Model of Visual Selection

paper_content:

The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate.
---
paper_title: A multi-modal object attention system for a mobile robot

paper_content:

Robot companions are intended for operation in private homes with naive users. For this purpose, they need to be endowed with natural interaction capabilities. Additionally, such robots will need to be taught unknown objects that are present in private homes. We present a multi-modal object attention system that is able to identify objects referenced by the user with gestures and verbal instructions. The proposed system can detect known and unknown objects and stores newly acquired object information in a scene model for later retrieval. This way, the growing knowledge base of the robot companion improves the interaction quality as the robot can more easily focus its attention on objects it has been taught previously.
---
paper_title: Distributed visual attention on a humanoid robot

paper_content:

Complex visual processes such as visual attention are often computationally too expensive to allow real-time implementation on a single computer. To solve this problem we study distributed computer architectures that enable us to divide complex tasks into several smaller problems. In this paper we demonstrate how to implement distributed visual attention system on a humanoid robot to achieve real-time operation at relatively high resolutions and frame rates. We start from a popular theory of bottom-up visual attention that assumes that information across various modalities is used for the early encoding of visual information. Our system uses five different modalities including color, intensity, edges, stereo, and motion. We show how to distribute the attention processing on a computer cluster and study the issues arising on such systems. The system was fully implemented on a workstation cluster comprised of eight PCs. It was used to drive the gaze of a humanoid head towards potential regions of interest
---
paper_title: A visual attention network for a humanoid robot

paper_content:

For a humanoid robot to interact easily with a person, the robot should have human-like sensory capabilities and attentional mechanisms. Particularly useful is an active vision head controlled by a visual attention system that selects viewpoints in the environment as a function of the robot's task. This paper describes a model of human visual attention called FeatureGate, which is a locally connected, pyramidal, artificial neural network that operates on 2D feature maps of the environment. Given a set of feature maps, and the description of a specific target, FeatureGate finds the location whose features most closely match those of the target. The paper describes the network, its implementation, a series of tests that characterize its performance with respect to a person's performance on a similar task, and its use in the control of an active vision system.
---
paper_title: Saccade target selection and object recognition: Evidence for a common attentional mechanism

paper_content:

Abstract The spatial interaction of visual attention and saccadic eye movements was investigated in a dual-task paradigm that required a target-directed saccade in combination with a letter discrimination task. Subjects had to saccade to locations within horizontal letter strings left and right of a central fixation cross. The performance in discriminating between the symbols “E” and “∃”, presented tachistoscopically before the saccade within the surrounding distractors was taken as a measure of visual attention. The data show that visual discrimination is best when discrimination stimulus and saccade target refer to the same object; discrimination at neighboring items is close to chance level. Also, it is not possible, in spite of prior knowledge of discrimination target position, to direct attention to the discrimination target while saccading to a spatially close saccade target. The data strongly argue for an obligatory and selective coupling of saccade programming and visual attention to one common target object. The results favor a model in which a single attentional mechanism selects objects for perceptual processing and recognition, and also provides the information necessary for motor action.
---
paper_title: A distributed model of spatial visual attention

paper_content:

Although biomimetic autonomous robotics relies on the massively parallel architecture of the brain, the key issue is to temporally organize behaviour. The distributed representation of the sensory information has to be coherently processed to generate relevant actions. In the visual domain, we propose here a model of visual exploration of a scene by the means of localized computations in neural populations whose architecture allows the emergence of a coherent behaviour of sequential scanning of salient stimuli. It has been implemented on a real robotic platform exploring a moving and noisy scene including several identical targets.
---
paper_title: Overt visual attention for a humanoid robot

paper_content:

The goal of our research is to investigate the interplay between oculomotor control, visual processing, and limb control in humans and primates by exploring the computational issues of these processes with a biologically inspired artificial oculomotor system on an anthropomorphic robot. In this paper, we investigate the computational mechanisms for visual attention in such a system. Stimuli in the environment excite a dynamical neural network that implements a saliency map, i.e., a winner-take-all competition between stimuli while simultaneously smoothing out noise and suppressing irrelevant inputs. In real-time, this system computes new targets for the shift of gaze, executed by the head-eye system of the robot. The redundant degrees-of-freedom of the head-eye system are resolved through a learned inverse kinematics with optimization criterion. We also address important issues how to ensure that the coordinate system of the saliency map remains correct after movement of the robot. The presented attention system is built on principled modules and generally applicable for any sensory modality.
---
paper_title: A Cortico-Subcortical Model for Generation of Spatially Accurate Sequential Saccades

paper_content:

This article provides a systems framework for the analysis of cortical and subcortical interactions in the control of saccadic eye movements, A major thesis of this model is that a topography of saccade direction and amplitude is preserved through multiple projections between brain regions until they are finally transformed into a temporal pattern of activity that drives the eyes to the target. The control of voluntary saccades to visual and remembered targets is modeled in terms of interactions between posterior parietal cortex, frontal eye fields, the basal ganglia (caudate and substantia nigra), superior colliculus, mediodorsal thalamus, and the saccade generator of the brainstem. Interactions include the modulation of eye movement motor error maps by topographic inhibitory projections, dynamic remapping of spatial target representations in saccade motor error maps, and sustained neural activity that embodies spatial memory. Models of these mechanisms implemented in our Neural Simulation Language simulate behavior and neural activity described in the literature, and suggest new experiments.
---
paper_title: Eye-hand coordination in object manipulation

paper_content:

We analyzed the coordination between gaze behavior, fingertip movements, and movements of the manipulated object when subjects reached for and grasped a bar and moved it to press a target-switch. Subjects almost exclusively fixated certain landmarks critical for the control of the task. Landmarks at which contact events took place were obligatory gaze targets. These included the grasp site on the bar, the target, and the support surface where the bar was returned after target contact. Any obstacle in the direct movement path and the tip of the bar were optional landmarks. Subjects never fixated the hand or the moving bar. Gaze and hand/bar movements were linked concerning landmarks, with gaze leading. The instant that gaze exited a given landmark coincided with a kinematic event at that landmark in a manner suggesting that subjects monitored critical kinematic events for phasic verification of task progress and subgoal completion. For both the obstacle and target, subjects directed saccades and fixations to sites that were offset from the physical extension of the objects. Fixations related to an obstacle appeared to specify a location around which the extending tip of the bar should travel. We conclude that gaze supports hand movement planning by marking key positions to which the fingertips or grasped object are subsequently directed. The salience of gaze targets arises from the functional sensorimotor requirements of the task. We further suggest that gaze control contributes to the development and maintenance of sensorimotor correlation matrices that support predictive motor control in manipulation.
---
paper_title: An Overt Visual Attention Mechanism Based on Saliency Dynamics

paper_content:

Abstract A visual overt attention mechanism is presented in this paper. Our algorithm chooses the next fixation point for a robot mobile camera in order to track several objects around the robot simultaneously, even if the whole set of them can not be covered by the same camera image. Our approach is based on two related measurements, liveliness and saliency, that dynamically evolve depending on the image observations and the camera movements. The attention is shared among exploring the surroundings for salient features, reobserving the tracked objects and other task-dependent points to look at. A winner-takes-all competition provides a flexible time sharing behavior with natural appearances of new objects, disappearances and inhibition of return. It also accepts top down features to look for and different priorities for them. Several experiments have been carried out with a real Pioneer robot endowed with mobile camera and are also described.
---
paper_title: Interaction rule learning with a human partner based on an imitation faculty with a simple visuo-motor mapping

paper_content:

Abstract Imitation has been receiving increasing attention from the viewpoint of not simply generating new motions but also the emergence of communication. This paper proposes a system for a humanoid who obtains new motions through learning the interaction rules with a human partner based on the assumption of the mirror system. First, a humanoid learns the correspondence between its own posture and the partner’s one on the ISOMAPs supposing that a human partner imitates the robot motions. Based on this correspondence, the robot can easily transfer the observed partner’s gestures to its own motion. Then, this correspondence enables a robot to acquire the new motion primitive for the interaction. Furthermore, through this process, the humanoid learns an interaction rule that control gesture turn-taking. The preliminary results and future issues are given.
---
paper_title: Playing Domino: A Case Study for an Active Vision System

paper_content:

We introduce a mobile robot playing at dominoes. The robot is equipped with a pan-tilt unit and a CCD camera and is solely guided by the vision system. Besides the mobile robot the overall system is distributed on a general-purpose workstation. The communication is realized by three radio links transmitting the video signal and two serial data streams for the pan-tilt and the robot controller respectively. Two client/server connections enable a bidirectional data exchange between the image processing software and the robot control and navigation software. The protocol includes requests for certain actions to be performed by the opposite module and the transmission of domino coordinates. The robot identifies the dominoes, collects them, and deposits them in form of a straight chain where two adjacent domino halves possess the same digit. Several iterations of a perception action cycle have to be executed to fulfill this task: The first cycle starts with a visual exploration of the robot's environment and a subsequent explicit foveation of detected domino point clusters. After a coordinate transformation the dominoes are identified by template matching followed by a comparison with a knowledge base to eliminate ambiguities. The positions of the dominoes are registered into the global map space of the robot which position is continuously updated while it is moving. When the robot has reached a so-called virtual point in front of a specified domino the robot requests a control view towards this domino to verify its position. After feedback from the vision system the robot grips the target, moves to the appropriate position in front of the already formed chain, again verifies its position, and deposits the domino. Afterwards the robot returns to its initial position where the cycle starts again with the selection of the next target domino.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: Providing the basis for human-robot-interaction: a multi-modal attention system for a mobile robot

paper_content:

In order to enable the widespread use of robots in home and office environments, systems with natural interaction capabilities have to be developed. A prerequisite for natural interaction is the robot's ability to automatically recognize when and how long a person's attention is directed towards it for communication. As in open environments several persons can be present simultaneously, the detection of the communication partner is of particular importance. In this paper we present an attention system for a mobile robot which enables the robot to shift its attention to the person of interest and to maintain attention during interaction. Our approach is based on a method for multi-modal person tracking which uses a pan-tilt camera for face recognition, two microphones for sound source localization, and a laser range finder for leg detection. Shifting of attention is realized by turning the camera into the direction of the person which is currently speaking. From the orientation of the head it is decided whether the speaker addresses the robot. The performance of the proposed approach is demonstrated with an evaluation. In addition, qualitative results from the performance of the robot at the exhibition part of the ICVS'03 are provided.
---
paper_title: A saliency-based search mechanism for overt and covert shifts of visual attention

paper_content:

Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested.
---
paper_title: Multimodal joint attention through cross facilitative learning based on μX principle

paper_content:

Simultaneous learning of multiple functions is one of the fundamental issues not only to design intelligent robots but also to understand humanpsilas cognitive developmental process since we, human, do so in our daily lives but we do not know how to do. Drawing an analogy to the well-known bias in child language development, we propose the mutual exclusivity selection principle (muX principle) for learning multi-modal mappings: selecting more mutually exclusive output leads experiences to make underdeveloped complementary mappings more disambiguated. The muX principle is applied to multi-modal joint attention with utterances for lexicon acquisition, and synthetically modeled in both intra- and inter-module levels of output. Through the series of computer simulations, the effects of the muX principle on the mutual facilitation in learning multi-functions and robustness against errors in segmentation of observation are analyzed. Finally, the correspondence of the synthesized development to infantpsilas one is argued based on the simulation with careful behavior by a caregiver.
---
paper_title: Simultaneous localization and map-building using active vision

paper_content:

An active approach to sensing can provide the focused measurement capability over a wide field of view which allows correctly formulated simultaneous localization and map-building (SLAM) to be implemented with vision, permitting repeatable longterm localization using only naturally occurring, automatically-detected features. In this paper, we present the first example of a general system for autonomous localization using active vision, enabled here by a high-performance stereo head, addressing such issues as uncertainty-based measurement selection, automatic map-maintenance, and goal-directed steering. We present varied real-time experiments in a complex environment.
---
paper_title: A developmental approach accelerates learning of joint attention

paper_content:

This paper argues how robot learning can be accelerated by a developmental approach that changes the capability of the learning robot (robot development) and/or the complexity of the environment (environmental development). "Robot development" means that the perceptual, cognitive, and behavioral capabilities of the learning robot change from immature to mature states in accordance with learning process. On the other hand, "environmental development" means that the complexity of the environmental structure changes from simple to complicated ones in accordance with learning process. Then, the developmental approaches can be categorized according to the relationships between what does develop and what does make the development. A case study of developmental robot learning for joint attention in which a caregiver can be regarded as a part of the environment is given to show how it can accelerate the learning. Finally, future work and discussion are given.
---
paper_title: Guided search: An alternative to the feature integration model for visual search.

paper_content:

Subjects searched sets of items for targets defined by conjunctions of color and form, color and orientation, or color and size. Set size was varied and reaction times (RT) were measured. For many unpracticed subjects, the slopes of the resulting RT X Set Size functions are too shallow to be consistent with Treisman's feature integration model, which proposes serial, self-terminating search for conjunctions. Searches for triple conjunctions (Color X Size X Form) are easier than searches for standard conjunctions and can be independent of set size. A guided search model similar to Hoffman's (1979) two-stage model can account for these data. In the model, parallel processes use information about simple features to guide attention in the search for conjunctions. Triple conjunctions are found more efficiently than standard conjunctions because three parallel processes can guide attention more effectively than two.
---
paper_title: On-line Imitative Interaction with a Humanoid Robot Using a Dynamic Neural Network Model of a Mirror System

paper_content:

This study presents experiments on the imitative interactions between a small humanoid robot and a user. A dynamic neural network model of a mirror system was implemented in a humanoid robot, based on the recurrent neural network model with parametric bias (RNNPB). The experiments showed that after the robot learns multiple cyclic movement patterns as embedded in the RNNPB, it can regenerate each pattern synchronously with the movements of a human who is demonstrating the corresponding movement pattern in front of the robot. Further, the robot exhibits diverse interactive responses when the user demonstrates novel cyclic movement patterns. Those responses were analyzed and categorized. We propose that the dynamics of coherence and incoherence between the robot’s and the user’s movements could enhance close interactions between them, and that they could also explain the essential psychological mechanism of joint attention.
---
paper_title: A Visual Attention Framework for Search Behavior by a Humanoid Robot

paper_content:

This paper presents a framework for a visual search behavior of a 3D object in a 3D environment performed by HRP-2 humanoid robot. Object search is a sensor planning problem in which appropriate sensor configuration must be selected in order to allow a proper recognition. The sensor planning is formulated as an optimization problem in which the goal is to maximize the target detection probability while minimizing the energy/distance and time to achieve the task. The knowledge of the target position is encoded in a probability distribution map which will be updated after each sensing operation. Specificities of the humanoid robot as well as the characteristics of the recognition system are taken into account to restrict the sensor configuration space to meet heavy time constraints on the behavior in order to have a reactive and near human like reaction time.
---
paper_title: Developmental learning of integrating visual attention shifts and bimanual object grasping and manipulation tasks

paper_content:

In order to achieve visual-guided object manipulation tasks via learning by example, the current neuro-robotics study considers integration of two essential mechanisms of visual attention and arm/hand movement and their adaptive coordination. The present study proposes a new dynamic neural network model in which visual attention and motor behavior are associated with task specific manners by learning with self-organizing functional hierarchy required for the cognitive tasks. The top-down visual attention provides a goal-directed shift sequence in a visual scan path and it can guide a generation of a motor plan for hand movement during action by reinforcement and inhibition learning. The proposed model can automatically generate the corresponding goal-directed actions with regards to the current sensory states including visual stimuli and body postures. The experiments show that developmental learning from basic actions to combinational ones can achieve certain generalizations in learning by which some novel behaviors without prior learning can be successfully generated.
---
paper_title: Joint attention between a humanoid robot and users in imitation game

paper_content:

This paper studies a synthetic model for joint attentions andturn takingbyconductingexperimentsofrobot-userimitative interactions. A Sony humanoid robot learns multiple cyclic movement patterns as embedded in a neural network model proposed by us and each of memorized movement patterns can be retrieved and generated by means of entrainment by external inputs in terms of users’ movement patterns perceived. Our experiments on a simple imitation game showed that multiple movement patterns are generated as synchronized between the robot and users while the shared patterns shift spontaneously from one to another. Based on the experimental results, we show possible dynamical systems accounts for the underlying mechanisms for joint attentions and turn taking.
---
paper_title: Towards a Real-Time Bayesian Imitation System for a Humanoid Robot

paper_content:

Imitation learning, or programming by demonstration (PbD), holds the promise of allowing robots to acquire skills from humans with domain-specific knowledge, who nonetheless are inexperienced at programming robots. We have prototyped a real-time, closed-loop system for teaching a humanoid robot to interact with objects in its environment. The system uses nonparametric Bayesian inference to determine an optimal action given a configuration of objects in the world and a desired future configuration. We describe our prototype implementation, show imitation of simple motor acts on a humanoid robot, and discuss extensions to the system
---
paper_title: Joint attention emerges through bootstrap learning

paper_content:

A human-like intelligent robot is expected to have the capability to develop its cognitive functions through experience without a priori knowledge or explicit teaching. In addition, the realization of this kind of robot leads us to understand the developmental mechanisms of human beings. This paper proposes a bootstrap learning model by which a robot acquires the ability of joint attention without a caregiver's evaluation or a controlled environment based on the robot's embedded mechanisms: visual attention and learning with self-evaluation. Through learning based on the proposed model, the robot finds a correlation in sensorimotor coordination when joint attention succeeds and consequently acquires the ability of joint attention by accumulating the appropriate correlation and losing the uncorrelated coordination as statistical outliers. The experimental results show the validity of the proposed model.
---
paper_title: Attentional Landmarks and Active Gaze Control for Visual SLAM

paper_content:

This paper is centered around landmark detection, tracking, and matching for visual simultaneous localization and mapping using a monocular vision system with active gaze control. We present a system that specializes in creating and maintaining a sparse set of landmarks based on a biologically motivated feature-selection strategy. A visual attention system detects salient features that are highly discriminative and ideal candidates for visual landmarks that are easy to redetect. Features are tracked over several frames to determine stable landmarks and to estimate their 3-D position in the environment. Matching of current landmarks to database entries enables loop closing. Active gaze control allows us to overcome some of the limitations of using a monocular vision system with a relatively small field of view. It supports 1) the tracking of landmarks that enable a better pose estimation, 2) the exploration of regions without landmarks to obtain a better distribution of landmarks in the environment, and 3) the active redetection of landmarks to enable loop closing in situations in which a fixed camera fails to close the loop. Several real-world experiments show that accurate pose estimation is obtained with the presented system and that active camera control outperforms the passive approach.
---
paper_title: Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

paper_content:

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.
---
paper_title: Active Learning of Joint Attention

paper_content:

Joint attention is the skill of attending to the same object another person is looking at. The acquisition of this skill is crucial in children for the development of many social and communicative abilities, and has been proposed as a critical social capability for interactive robots. Although recent attempts to model the acquisition of this skill on a robot have been moderately successful (Nagai et al., 2003; Triesch et al., 2006), they all assume that the robot remains passive during the learning process. Infants, on the other hand, have already acquired some rudimentary sensorimotor skills by the time they start to learn joint attention. We believe that these sensorimotor skills can jumpstart and considerably accelerate the learning of joint attention. In this paper we demonstrate on a humanoid robot how to use pointing and reaching to accelerate the learning of joint attention. We show that a robot can acquire this skill with a 95 % accuracy after a total of only 220 training samples compared to 85% accuracy after totals of 10,000+ samples in other approaches.
---
paper_title: Enactive Robot Vision

paper_content:

Enactivism claims that sensory-motor activity and embodiment are crucial in perceiving the environment and that machine vision could be a much simpler business if considered in this context. However, computational models of enactive vision are very rare and often rely on handcrafted control systems. In this article, we argue that the apparent complexity of the environment and of the robot brain can be significantly simplified if perception, behavior, and learning are allowed to co-develop on the same timescale. In doing so, robots become sensitive to, and actively exploit, characteristics of the environment that they can tackle within their own computational and physical constraints. We describe the application of this methodology in three sets of experiments: shape discrimination, car driving, and wheeled robot navigation. A further set of experiments, where the visual system can develop the receptive fields by means of unsupervised Hebbian learning, demonstrates that the receptive fields are consistently and significantly affected by the behavior of the system and differ from those predicted by most computational models of the visual cortex. Finally, we show that our robots can also replicate the performance deficiencies observed in experiments of motor deprivation with kittens.
---
paper_title: Multimodal Attention System for an Interactive Robot

paper_content:

Social robots are receiving much interest in the robotics community. The most important goal for such robots lies in their interaction capabilities. An attention system is crucial, both as a filter to center the robot’s perceptual resources and as a mean of letting the observer know that the robot has intentionality. In this paper a simple but flexible and functional attentional model is described. The model, which has been implemented in an interactive robot currently under development, fuses both visual and auditive information extracted from the robot’s environment, and can incorporate knowledge-based influences on attention.
---
paper_title: A multi-modal object attention system for a mobile robot

paper_content:

Robot companions are intended for operation in private homes with naive users. For this purpose, they need to be endowed with natural interaction capabilities. Additionally, such robots will need to be taught unknown objects that are present in private homes. We present a multi-modal object attention system that is able to identify objects referenced by the user with gestures and verbal instructions. The proposed system can detect known and unknown objects and stores newly acquired object information in a scene model for later retrieval. This way, the growing knowledge base of the robot companion improves the interaction quality as the robot can more easily focus its attention on objects it has been taught previously.
---
paper_title: The Challenges of Joint Attention

paper_content:

This article discusses the concept of joint attention and the different skills underlying its development. Research in developmental psychology clearly states that the development of skills to understand, manipulate and coordinate attentional behavior plays a pivotal role for imitation, social cognition and the development of language. However, beside the fact that joint attention has recently received an increasing interest in the robotics community, existing models concentrate only on partial and isolated elements of these phenomena. In the line of Tomasello’s research, we argue that joint attention is much more than simultaneous looking because it implies a shared intentional relation to the world. This requires skills for attention detection, attention manipulation, social coordination and, most importantly, intentional understanding. After defining joint attention and its challenges, the current state-of-the-art of robotic and computational models relevant for this issue is discussed in relation to a developmental timeline drawn from results in child studies. From this survey, we identify open issues and challenges that still need to be addressed to understand the development of the various aspects of joint attention and conclude with the potential contribution of robotic models.
---
paper_title: ON THE PERCEPTION OF INCONGRUITY: A PARADIGM

paper_content:

Perceiving, for all its nicety of functioning in the dark room under strict instructions for accuracy, comprises a highly complex series of little understood psychological processes. For under all conditions, perceiving represents a resultant of two complex sets of specifications. One set describes the conditions of stimulation. This is done either in terms of physical measures such as wave length, or in terms of psychological norms such as in the description of a picture as that picture is seen by "normal" observers under optimal conditions and with a set for accuracy. This first set of specifications we are used to calling "stimulus" factors.
---
paper_title: Acquisition of joint attention through natural interaction utilizing motion cues

paper_content:

Joint attention is one of the most important cognitive functions for the emergence of communication not only between humans, but also between humans and robots. In previous work, we have demonstrated how a robot can acquire primary joint attention behavior (gaze following) without external evaluation. However, this method needs the human to tell the robot when to shift its gaze. This paper presents a method that does not need such a constraint by introducing an attention selector based on a measure consisting of saliencies of object features and motion cues. In order to realize natural interaction, a self-organizing map for real-time face pattern separation and contingency learning for gaze following without external evaluation are utilized. The attention selector controls the robot gaze to switch often from the human face to an object and vice versa, and pairs of a face pattern and a gaze motor command are input to the contingency learning. The motion cues are expected to reduce the number of incorrect t...
---
paper_title: Shifts in selective visual attention: towards the underlying neural circuitry

paper_content:

A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
---
paper_title: On-line Imitative Interaction with a Humanoid Robot Using a Dynamic Neural Network Model of a Mirror System

paper_content:

This study presents experiments on the imitative interactions between a small humanoid robot and a user. A dynamic neural network model of a mirror system was implemented in a humanoid robot, based on the recurrent neural network model with parametric bias (RNNPB). The experiments showed that after the robot learns multiple cyclic movement patterns as embedded in the RNNPB, it can regenerate each pattern synchronously with the movements of a human who is demonstrating the corresponding movement pattern in front of the robot. Further, the robot exhibits diverse interactive responses when the user demonstrates novel cyclic movement patterns. Those responses were analyzed and categorized. We propose that the dynamics of coherence and incoherence between the robot’s and the user’s movements could enhance close interactions between them, and that they could also explain the essential psychological mechanism of joint attention.
---
paper_title: Object-based Visual Attention: a Model for a Behaving Robot

paper_content:

One of the first steps of any visual system is that of locating suitable interest points, "salient regions", in the scene, to detect events, and eventually to direct gaze toward these locations. In the last few years, object-based visual attention models have received an increasing interest in the literature, the problem, in this case, being that of creating a model of "objecthood" that eventually guides a saliency mechanism. We propose here an object-based model of visual attention and show its instantiation on a humanoid robot. The robot employs action to learn and define its own concept of objecthood.
---
paper_title: Joint attention between a humanoid robot and users in imitation game

paper_content:

This paper studies a synthetic model for joint attentions andturn takingbyconductingexperimentsofrobot-userimitative interactions. A Sony humanoid robot learns multiple cyclic movement patterns as embedded in a neural network model proposed by us and each of memorized movement patterns can be retrieved and generated by means of entrainment by external inputs in terms of users’ movement patterns perceived. Our experiments on a simple imitation game showed that multiple movement patterns are generated as synchronized between the robot and users while the shared patterns shift spontaneously from one to another. Based on the experimental results, we show possible dynamical systems accounts for the underlying mechanisms for joint attentions and turn taking.
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: A Reactive vision system: Active-dynamic saliency

paper_content:

We develop an architecture for reactive visual analysis of dynamic scenes. We specify a minimal set of system features based upon biological observations. We implement feature on a processing network based around an active stereo vision mechanism. Active rectification and mosaicing allows static stereo algorithms to operate on the active platform. Foveal zero disparity operations permit attended object extraction and ensures coordinated stereo fixation upon visual surfaces. Activedynamic inhibition of return, and task dependent biasing result in a flexible, preemptive and retrospective system that responds to unique visual stimuli and is capable of top-down modulation of attention towards regions and cues relevant to tasks.
---
paper_title: Image Mapping and Visual Attention on a Sensory Ego-Sphere

paper_content:

The Sensory Ego-Sphere (SES) is a short-term memory for a robot in the form of an egocentric, tessellated, spherical, sensory-motor map of the robot's locale. This paper reports on: (1) the mapping to the SES of images with much higher resolution than the SES itself, and (2) the processing of visual attention on the SES. Described is a procedure to store spatially-overlapping imagery in the SES database and to generate and update a spherical composite of its visual contents. The composite image serves both as a visual map of the locale and as a representation of the local contents of the underlying full-resolution imagery. Visual attention enables fast alignment of overlapping images without warping or position optimization, since an attentional point (AP) on the composite typically corresponds to one on each of the collocated regions in the images. Such alignment speeds analysis of the multiple images of the area. Compositing and attention were performed two ways and compared: (1) APs were computed directly on the composite and not on the full resolution images until the time of retrieval. (2) The attentional operator was applied to all incoming imagery. Then the locations and values of the APs on the composite were computed as a function of those points within the corresponding regions of the collocated images. It was found that although the second method was slower, it produced consistent and, thereby, more useful APs.
---
paper_title: Joint attention emerges through bootstrap learning

paper_content:

A human-like intelligent robot is expected to have the capability to develop its cognitive functions through experience without a priori knowledge or explicit teaching. In addition, the realization of this kind of robot leads us to understand the developmental mechanisms of human beings. This paper proposes a bootstrap learning model by which a robot acquires the ability of joint attention without a caregiver's evaluation or a controlled environment based on the robot's embedded mechanisms: visual attention and learning with self-evaluation. Through learning based on the proposed model, the robot finds a correlation in sensorimotor coordination when joint attention succeeds and consequently acquires the ability of joint attention by accumulating the appropriate correlation and losing the uncorrelated coordination as statistical outliers. The experimental results show the validity of the proposed model.
---
paper_title: Brain-based devices: intelligent systems based on principles of the nervous system

paper_content:

Our group has constructed a series of brain-based devices (BBDs); i.e. physical devices with simulated nervous systems that guide behavior, to serve as a heuristic for understanding brain function. Unlike conventional robots designed by engineering principles, BBDs are based on biological principles and alter their behavior to the environment through self-learning. The resulting systems autonomously generalize signals from the environment into perceptual categories and through adaptive behavior become increasingly successful in coping with the environment. Although the principal focus of developing BBDs has been to test theories of the nervous system, this approach may also provide a basis for robotic design and practical applications.
---
paper_title: Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

paper_content:

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.
---
paper_title: Multimodal saliency-based bottom-up attention a framework for the humanoid robot iCub

paper_content:

This work presents a multimodal bottom-up attention system for the humanoid robot iCub where the robot's decisions to move eyes and neck are based on visual and acoustic saliency maps. We introduce a modular and distributed software architecture which is capable of fusing visual and acoustic saliency maps into one egocentric frame of reference. This system endows the iCub with an emergent exploratory behavior reacting to combined visual and auditory saliency. The developed software modules provide a flexible foundation for the open iCub platform and for further experiments and developments, including higher levels of attention and representation of the peripersonal space.
---
paper_title: An adaptive detection/attention mechanism for real time robot operation

paper_content:

During the lifetime of a mobile robot, the number and complexity of the stimuli it receives may be quite high. Therefore, the construction of a detection system considering the whole sensorial space is usually not a viable proposition when aiming for real time operation. It becomes necessary to build some kind of sensorial hierarchy map to put some order into how detectors are applied. This is what is usually called an attentional system, and it provides a framework for applying detectors in a more efficient manner. In this paper, an architecture for developing attentional functions for robots that must operate in real time in dynamic environments is presented. This architecture is based on the concept of attentor and it allows for the real time adaptation to the environment and tasks to be performed in a natural manner. One of the main requirements imposed on the design of the architecture was the capability of handling different sensorial modalities and attentional streams in a transparent manner while, at the same time, being able to progressively create more complex attentional structures. The architecture is particularized for its implementation in a real robot.
---
paper_title: Development of a robot with a sense of self

paper_content:

This paper describes our efforts to develop a robot with a sense of self using a multiagent-based cognitive architecture and control with three distinctive memory systems, namely (1) spatio-temporal short-term memory, (2) procedural/declarative/episodic long-term memory and (3) a task-oriented adaptive working memory. Such a robot may be called a cognitive robot. Cognitive robots share a number of key features with conscious machines. We are exploring the interface between cognitive robots and machine consciousness through an internal model called the Self Agent.
---
paper_title: Multimodal Attention System for an Interactive Robot

paper_content:

Social robots are receiving much interest in the robotics community. The most important goal for such robots lies in their interaction capabilities. An attention system is crucial, both as a filter to center the robot’s perceptual resources and as a mean of letting the observer know that the robot has intentionality. In this paper a simple but flexible and functional attentional model is described. The model, which has been implemented in an interactive robot currently under development, fuses both visual and auditive information extracted from the robot’s environment, and can incorporate knowledge-based influences on attention.
---
paper_title: The FeatureGate Model of Visual Selection

paper_content:

The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate.
---
paper_title: A multi-modal object attention system for a mobile robot

paper_content:

Robot companions are intended for operation in private homes with naive users. For this purpose, they need to be endowed with natural interaction capabilities. Additionally, such robots will need to be taught unknown objects that are present in private homes. We present a multi-modal object attention system that is able to identify objects referenced by the user with gestures and verbal instructions. The proposed system can detect known and unknown objects and stores newly acquired object information in a scene model for later retrieval. This way, the growing knowledge base of the robot companion improves the interaction quality as the robot can more easily focus its attention on objects it has been taught previously.
---
paper_title: An attentional system for a humanoid robot exploiting space variant vision

paper_content:

This paper describes the implementation of the attentional system of a humanoid robot based completely on space variant vision (in particular logpolar). The aim is that of providing the robot with a suitable measure of position, speed and saliency of possibly interesting objects for saccading and tracking. The major advantage of log-polar based imaging is related to the reduced number of pixels while maintaining a large field of view. This arrangement is very well suited for motor control, where the high-resolution center (fovea) allows precise positioning and, at the same time, the coarse resolution periphery permits detection of potential targets. Algorithms for color processing, optic flow, and disparity computation were developed within this architecture. The attentional modules are intended as the first layer of a more complicated system, which shall include learning of object recognition, trajectory tracking, and naive physics understanding during the natural interaction of the robot with the environment.
---
paper_title: A distributed model of spatial visual attention

paper_content:

Although biomimetic autonomous robotics relies on the massively parallel architecture of the brain, the key issue is to temporally organize behaviour. The distributed representation of the sensory information has to be coherently processed to generate relevant actions. In the visual domain, we propose here a model of visual exploration of a scene by the means of localized computations in neural populations whose architecture allows the emergence of a coherent behaviour of sequential scanning of salient stimuli. It has been implemented on a real robotic platform exploring a moving and noisy scene including several identical targets.
---
paper_title: Overt visual attention for a humanoid robot

paper_content:

The goal of our research is to investigate the interplay between oculomotor control, visual processing, and limb control in humans and primates by exploring the computational issues of these processes with a biologically inspired artificial oculomotor system on an anthropomorphic robot. In this paper, we investigate the computational mechanisms for visual attention in such a system. Stimuli in the environment excite a dynamical neural network that implements a saliency map, i.e., a winner-take-all competition between stimuli while simultaneously smoothing out noise and suppressing irrelevant inputs. In real-time, this system computes new targets for the shift of gaze, executed by the head-eye system of the robot. The redundant degrees-of-freedom of the head-eye system are resolved through a learned inverse kinematics with optimization criterion. We also address important issues how to ensure that the coordinate system of the saliency map remains correct after movement of the robot. The presented attention system is built on principled modules and generally applicable for any sensory modality.
---
paper_title: ON THE PERCEPTION OF INCONGRUITY: A PARADIGM

paper_content:

Perceiving, for all its nicety of functioning in the dark room under strict instructions for accuracy, comprises a highly complex series of little understood psychological processes. For under all conditions, perceiving represents a resultant of two complex sets of specifications. One set describes the conditions of stimulation. This is done either in terms of physical measures such as wave length, or in terms of psychological norms such as in the description of a picture as that picture is seen by "normal" observers under optimal conditions and with a set for accuracy. This first set of specifications we are used to calling "stimulus" factors.
---
paper_title: A probabilistic approach for attention-based multi-modal human-robot interaction

paper_content:

The interaction between a robot and a human becomes meaningful when the robotic agent possesses some level of human-like cognition. This paper proposes an attention-based approach for multi-modal HRI. The core of the proposed approach is a bio-inspired artificial model of visual attention which enables a robot to focus on the visually salient and/or behaviorally relevant stimuli in the surrounding. The attention model provides the human partner with the opportunity to manipulate the attention behavior of the robot through natural speech command. Similarly the robot is able to manipulate the attention of the human partner using its actuators. Thus the bio-inspired visual attention mechanism along with the sensors and actuators enables the robot to establish joint attention with the human partner. Formation of this joint attention is the basis for further human-robot interaction. Experimental results validate different aspects of the proposed attention-based HRI framework.
---
paper_title: An Overt Visual Attention Mechanism Based on Saliency Dynamics

paper_content:

Abstract A visual overt attention mechanism is presented in this paper. Our algorithm chooses the next fixation point for a robot mobile camera in order to track several objects around the robot simultaneously, even if the whole set of them can not be covered by the same camera image. Our approach is based on two related measurements, liveliness and saliency, that dynamically evolve depending on the image observations and the camera movements. The attention is shared among exploring the surroundings for salient features, reobserving the tracked objects and other task-dependent points to look at. A winner-takes-all competition provides a flexible time sharing behavior with natural appearances of new objects, disappearances and inhibition of return. It also accepts top down features to look for and different priorities for them. Several experiments have been carried out with a real Pioneer robot endowed with mobile camera and are also described.
---

Title: Visual Attention for Robotic Cognition: A Survey

Section 1: Introduction
Description 1: Write about the significance of visual attention in robotic cognition, its connection to human visual attention, and the goal of the survey.

Section 2: Evolution of Research on Computational Modeling of Visual Attention
Description 2: Describe the historical development of computational models of visual attention, including influential models from psychology, computational neuroscience, and computer vision.

Section 3: Computational Models of Visual Attention and the Requirements of Robotic Visual Attention
Description 3: Discuss the characteristics of existing computational models of visual attention and explain the unique requirements and challenges of robotic visual attention systems.

Section 4: Robotic Research on Visual Attention
Description 4: Provide a comprehensive overview of the research efforts in robotic visual attention, categorized into overt attention models and application-specific models.

Section 5: General Discussion
Description 5: Present an overall discussion on the current state of robotic visual attention research, including a comparative analysis and future research directions.

Section 6: Conclusion
Description 6: Summarize the key points of the survey, highlighting the importance of integrating visual attention into robotic cognition and the potential impact on various applications.