Human-Computer Interaction : Overview on State of the Art
---
paper_title: Human-Computer Interaction: Developing Effective Organizational Information Systems

paper_content:

This book is about developing interactive information systems that support people at work or when conducting business. Specifically, it emphasizes the need to study and practice the development of HCI for real-world organizations in given contexts. Developing an effective information system means achieving a good fit among the users. In order to do this, designers need to have a good understanding of important factors that come into play. Designers need to understand why and how people interact with computers in order to accomplish their work and personal goals, what are the physical, cognitive, affective and behavioral constraints on the users’ side what pleases or annoys them, what makes human-computer interaction a satisfying experience or an effective. This knowledge is the foundation of human-computer interaction (HCI) development.
---
paper_title: Designing the User Interface. Strategies for Effective Human-Computer Interaction

paper_content:

Ben Shneiderman again provides a complete, current, and authoritative introduction to user-interface design. Students will learn practical techniques and guidelines needed to develop good systems designs - systems with interfaces the typical user can understand, predict, and control. This third edition features new chapters on the World Wide Web, information visualization, and computer-supported cooperative work. It contains expanded and earlier coverage of development methodologies, evaluation techniques, and user-interface building tools. The author provides provocative discussion of speech input/output, natural-language interaction, anthropomorphic design, virtual environments, and intelligent (software) agents.
---
paper_title: Human-Computer Interaction: Developing Effective Organizational Information Systems

paper_content:

This book is about developing interactive information systems that support people at work or when conducting business. Specifically, it emphasizes the need to study and practice the development of HCI for real-world organizations in given contexts. Developing an effective information system means achieving a good fit among the users. In order to do this, designers need to have a good understanding of important factors that come into play. Designers need to understand why and how people interact with computers in order to accomplish their work and personal goals, what are the physical, cognitive, affective and behavioral constraints on the users’ side what pleases or annoys them, what makes human-computer interaction a satisfying experience or an effective. This knowledge is the foundation of human-computer interaction (HCI) development.
---
paper_title: Introduction to Virtual Reality

paper_content:

1 Virtual Reality.- 1.1 Introduction.- 1.2 What Is VR?.- 1.3 Who Should Read This Book?.- 1.4 The Aims and Objectives of This Book.- 1.5 Assumptions Made in This Book.- 1.6 How to Use This Book.- 1.7 Some VR Concepts and Terms.- 1.8 Navigation and Interaction.- 1.9 Immersion and Presence.- 1.10 What Is Not VR?.- 1.11 The Internet.- 1.12 Summary.- 2 The Benefits of VR.- 2.1 Introduction.- 2.2 3D Visualization.- 2.3 Navigation.- 2.4 Interaction.- 2.5 Physical Simulation.- 2.6 VEs.- 2.7 Applications.- 2.8 Summary.- 3 3D Computer Graphics.- 3.1 Introduction.- 3.2 From Computer Graphics to VR.- 3.3 Modelling Objects.- 3.4 Dynamic Objects.- 3.5 Constraints.- 3.6 Collision Detection.- 3.7 Perspective Views.- 3.8 3D Clipping.- 3.9 Stereoscopic Vision.- 3.10 Rendering the Image.- 3.11 Rendering Algorithms.- 3.12 Texture Mapping.- 3.13 Bump Mapping.- 3.14 Environment Mapping.- 3.15 Shadows.- 3.16 Radiosity.- 3.17 Other Computer Graphics Techniques.- 3.18 Summary.- 4 Human Factors.- 4.1 Introduction.- 4.2 Vision.- 4.3 Vision and Display Technology.- 4.4 Hearing.- 4.5 Tactile.- 4.6 Equilibrium.- 4.7 Summary.- 5 VR Hardware.- 5.1 Introduction.- 5.2 Computers.- 5.3 Tracking.- 5.4 Input Devices.- 5.5 Output Devices.- 5.6 Glasses.- 5.7 Displays.- 5.8 Audio.- 5.9 Summary.- 6 VR Software.- 6.1 Introduction.- 6.2 VR Software Features.- 6.3 Web-Based VR.- 6.4 Division's dVISE.- 6.5 Blueberry3D.- 6.6 Boston Dynamics.- 6.7 MultiGen.- 6.8 Summary.- 7 VR Applications.- 7.1 Introduction.- 7.2 Industrial.- 7.3 Training Simulators.- 7.4 Entertainment.- 7.5 VR Centres.- 7.6 Summary.- 8 Conclusion.- 8.1 The Past.- 8.2 Today.- 8.3 Conclusion.- Appendices.- Appendix A VRML Web Sites.- Appendix B HMDs.- Appendix C Trackers.- Appendix D VRML Program.- Appendix E Web Sites for VR Products.- Referebces.
---
paper_title: Haptic interfaces and devices

paper_content:

Haptic interfaces enable person‐machine communication through touch, and most commonly, in response to user movements. We comment on a distinct property of haptic interfaces, that of providing for simultaneous information exchange between a user and a machine. We also comment on the fact that, like other kinds of displays, they can take advantage of both the strengths and the limitations of human perception. The paper then proceeds with a description of the components and the modus operandi of haptic interfaces, followed by a list of current and prospective applications and a discussion of a cross‐section of current device designs.
---
paper_title: Human-Computer Interaction: Developing Effective Organizational Information Systems

paper_content:

This book is about developing interactive information systems that support people at work or when conducting business. Specifically, it emphasizes the need to study and practice the development of HCI for real-world organizations in given contexts. Developing an effective information system means achieving a good fit among the users. In order to do this, designers need to have a good understanding of important factors that come into play. Designers need to understand why and how people interact with computers in order to accomplish their work and personal goals, what are the physical, cognitive, affective and behavioral constraints on the users’ side what pleases or annoys them, what makes human-computer interaction a satisfying experience or an effective. This knowledge is the foundation of human-computer interaction (HCI) development.
---
paper_title: Wireless Technology: Protocols, Standards, and Techniques

paper_content:

PART 1: INTRODUCTION WIRELESS NETWORK Introduction Intelligent Network Network Architecture Protocol Architecture Channel Structure Narrowband and Wideband Systems Multiple Access Summary CELLULAR PRINCIPLES Introduction Cellular Hierarchy System Management System Performance Cellular Reuse Pattern Macrocellular Reuse Pattern Microcellular Reuse Pattern Interference in Narrowband and Wideband Systems Interference in Narrowband Macrocellular Systems Interference in Narrowband Microcellular Systems Interference in Wideband Systems Network Capacity Summary MULTIPLE ACCESS Introduction Signal Domains Duplexing Multiple Access Categories Scheduled Multiple Access Random Multiple Access Controlled Multiple Access Hybrid Multiple Access Summary Part II: 2G SYSTEMS GSM Introduction Features and Services Architecture Multiple Access The Logical Channels Messages Call Management Frequency Hopping Discontinuous Transmission Power Control Spectral Efficiency Summary cdmaONE Introduction Features and Services Architecture Multiple Access Structure The Logical Channels Signaling Format Messages, Orders, and Parameters Messages and Orders and Logical Channels Mobile Station Call Processing Base Station Call Processing Authentication, Message Encryption, and Voice Privacy Authentication Message Encryption Voice Privacy Roaming Handoff Power Control Call Procedures EIA/TIA/IS-95B Summary PART III: WIRELESS DATA WIRELESS DATA TECHNOLOGY Introduction General Packet Radio Service (GPRS) EIA/TIA/IS-95B High Data Rate (HDR) Summary PART IV: 3G SYSTEMS IMT-2000 Introduction Some Definitions Frequency Allocation Features and Services Traffic Classes IMT-2000 System and IMT-2000 Family Specific Functions Network Architecture Physical Entities and Functional Entities Functional Entities and their Interrelations Application of IMT-2000 Family Member Concept Toward 3G Summary UTRA Introduction Network Architecture Protocol Architecture Radio Interface Protocol Architecture Logical Channels Transport channels and Indicators Physical Channels and Physical Signals Mapping of Channels Physical Layer Transmission Chain Channel and Frame Structures Spreading and Modulation Spreading Codes UTRA Procedures Interference Issues Summary cdma2000 Introduction Network Architecture Radio Interface Protocol Architecture Logical Channels Physical Channels Mapping of Channels Achievable Rates Forward Link Reverse Link Forward Physical Channels Reverse Physical Channels High Rate Packet Data Access Summary PART V: APPENDICES OPEN SYSTEMS INTERCONNECTION SIGNALING SYSTEM NUMBER 7 SPREAD SPECTRUM Correlation Pseudonoise Sequence Walsh Codes Orthogonal Variable Spreading Factor Codes Rake Receiver Processing Gain Direct Sequence Spread Spectrum Frequency Hopping Spread Spectrum POSITIONING OF THE INTERFERERS IN A MICROCELLULAR GRID Collinear Type Even Noncollinear Type Odd Nonprime Noncollinear Prime Noncollinear
---
paper_title: The importance of the sense of touch in virtual and real environments

paper_content:

What would be worse, losing your sight or your sense of touch? Although touch (more generally, somesthesis) is commonly underrated, major somesthetic loss can't be adequately compensated for by sight. It results in catastrophic impairments of hand dexterity, haptic capabilities, walking, perception of limb position, and so on. Providing users with inadequate somesthetic feedback in virtual environments might impair their performance, just as major somesthetic loss does
---
paper_title: A probabilistic approach to reference resolution in multimodal user interfaces

paper_content:

Multimodal user interfaces allow users to interact with computers through multiple modalities, such as speech, gesture, and gaze. To be effective, multimodal user interfaces must correctly identify all objects which users refer to in their inputs. To systematically resolve different types of references, we have developed a probabilistic approach that uses a graph-matching algorithm. Our approach identifies the most probable referents by optimizing the satisfaction of semantic, temporal, and contextual constraints simultaneously. Our preliminary user study results indicate that our approach can successfully resolve a wide variety of referring expressions, ranging from simple to complex and from precise to ambiguous ones.
---
paper_title: Fundamentals Of Speech Recognition

paper_content:

1. Fundamentals of Speech Recognition. 2. The Speech Signal: Production, Perception, and Acoustic-Phonetic Characterization. 3. Signal Processing and Analysis Methods for Speech Recognition. 4. Pattern Comparison Techniques. 5. Speech Recognition System Design and Implementation Issues. 6. Theory and Implementation of Hidden Markov Models. 7. Speech Recognition Based on Connected Word Models. 8. Large Vocabulary Continuous Speech Recognition. 9. Task-Oriented Applications of Automatic Speech Recognition.
---
paper_title: The Visual Analysis of Human Movement: A Survey

paper_content:

The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human-inhabited environment. Because of many potentially important applications, “looking at people” is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D approaches. Where appropriate, systems are reviewed. We conclude with some thoughts about future directions.
---
paper_title: Readings in Intelligent User Interfaces

paper_content:

Introduction I: Multimedia Input Analysis II: Multimedia Presentation Design III: Automated Graphic Design IV: Automated Layout V: User and Discourse Modeling VI: Model Based Interfaces VII: Agent Interfaces VIII: Evaluation
---
paper_title: Human-Computer Interaction: Developing Effective Organizational Information Systems

paper_content:

This book is about developing interactive information systems that support people at work or when conducting business. Specifically, it emphasizes the need to study and practice the development of HCI for real-world organizations in given contexts. Developing an effective information system means achieving a good fit among the users. In order to do this, designers need to have a good understanding of important factors that come into play. Designers need to understand why and how people interact with computers in order to accomplish their work and personal goals, what are the physical, cognitive, affective and behavioral constraints on the users’ side what pleases or annoys them, what makes human-computer interaction a satisfying experience or an effective. This knowledge is the foundation of human-computer interaction (HCI) development.
---
paper_title: Evaluation of eye gaze interaction

paper_content:

Eye gaze interaction can provide a convenient and natural addition to user-computer dialogues. We have previously reported on our interaction techniques using eye gaze [10]. While our techniques seemed useful in demonstration, we now investigate their strengths and weaknesses in a controlled setting. In this paper, we present two experiments that compare an interaction technique we developed for object selection based on a where a person is looking with the most commonly used selection method using a mouse. We find that our eye gaze interaction technique is faster than selection with a mouse. The results show that our algorithm, which makes use of knowledge about how the eyes behave, preserves the natural quickness of the eye. Eye gaze interaction is a reasonable addition to computer interaction and is convenient in situations where it is important to use the hands for other tasks. It is particularly beneficial for the larger screen workspaces and virtual environments of the future, and it will become increasingly practical as eye tracker technology matures.
---
paper_title: Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions

paper_content:

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human-computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably under realistic and challenging usage conditions. In this article, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error-handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multiperson use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems.
---
paper_title: Electronic tongue for quality assessment of ethanol, vodka and eau-de-vie

paper_content:

Rapid quality assessment of alcoholic beverages, including brand identification and detection of products of unacceptable quality or counterfeits is an important practical task. In the present work the multisensor electronic tongue system (ET), based on array of potentiometric chemical sensors was applied to recognition and classification of spirits such as vodka and ethanol used for vodka production and also for eau-de-vie in cognac production. The ET system was capable of detecting presence of contaminant substances in vodka in concentrations exceeding allowed levels as well as of distinguishing vodka complying and not complying with state quality standards. Ten brands of vodka produced at the same distillery using water and ethanol of different purity and various taste additives were discriminated using the instrument. The ET could distinguish synthetic and alimentary grain ethanol as well as alimentary ethanol of different grades (i.e. different degree of purification). A feasibility study was run on several eau-de-vie samples, which included fresh and aged eau-de-vie as well as samples produced using different distillation technology and samples kept in contact with different kinds of oak. The electronic tongue showed a promise as an analytical instrument for rapid quality assessment of spirits.
---
paper_title: Multimodal human-computer interaction: A survey

paper_content:

In this paper we review the major approaches to multimodal human computer interaction from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition, and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for Multimodal Human Computer Interaction (MMHCI) research.
---
paper_title: A Breadth-First Survey of Eye Tracking Applications

paper_content:

Eye-tracking applications are surveyed in a breadth-first manner, reporting on work from the following domains: neuroscience, psychology, industrial engineering and human factors, marketing/advertising, and computer science. Following a review of traditionally diagnostic uses, emphasis is placed on interactive applications, differentiating between selective and gaze-contingent approaches.
---
paper_title: Evaluation of eye gaze interaction

paper_content:

Eye gaze interaction can provide a convenient and natural addition to user-computer dialogues. We have previously reported on our interaction techniques using eye gaze [10]. While our techniques seemed useful in demonstration, we now investigate their strengths and weaknesses in a controlled setting. In this paper, we present two experiments that compare an interaction technique we developed for object selection based on a where a person is looking with the most commonly used selection method using a mouse. We find that our eye gaze interaction technique is faster than selection with a mouse. The results show that our algorithm, which makes use of knowledge about how the eyes behave, preserves the natural quickness of the eye. Eye gaze interaction is a reasonable addition to computer interaction and is convenient in situations where it is important to use the hands for other tasks. It is particularly beneficial for the larger screen workspaces and virtual environments of the future, and it will become increasingly practical as eye tracker technology matures.
---
paper_title: Perceptual user interfaces using vision-based eye tracking

paper_content:

We present a multi-camera vision-based eye tracking method to robustly locate and track user's eyes as they interact with an application. We propose enhancements to various vision-based eye-tracking approaches, which include (a) the use of multiple cameras to estimate head pose and increase coverage of the sensors and (b) the use of probabilistic measures incorporating Fisher's linear discriminant to robustly track the eyes under varying lighting conditions in real-time. We present experiments and quantitative results to demonstrate the robustness of our eye tracking in two application prototypes.
---
paper_title: Designing, Playing, and Performing with a Vision-based Mouth Interface

paper_content:

The role of the face and mouth in speech production as well as non-verbal communication suggests the use of facial action to control musical sound. Here we document work on the Mouthesizer, a system which uses a headworn miniature camera and computer vision algorithm to extract shape parameters from the mouth opening and output these as MIDI control changes. We report our experience with various gesture-to-sound mappings and musical applications, and describe a live performance which used the Mouthesizer interface.
---
paper_title: Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction

paper_content:

Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities to control the computer such as voice, gesture, and force-feedback are emerging. Among these, voice and vision are two natural modalities in human-to-human communication. Automatic speech recognition (ASR) technology has matured enough to allow users to dictate to a word processor or operate the computer using voice commands. Computer vision techniques have enabled the computer to see. Interacting with computers in these modalities is much more natural for people, and the progression is towards the kind of interaction between humans. Despite these advances, one necessary ingredient for natural interaction is still missing—emotions. Emotions play an important role in human-to-human communication and interaction, allowing people to express themselves beyond the verbal domain. The ability to understand human emotions is desirable for the computer in some applications such as computer-aided learning or user-friendly online help. ::: This thesis addresses the problem of detecting human emotional expressions by computer from the voice and facial motions of the user. The computer is equipped with a microphone to listen to the user's voice, and a video camera to look at the user. Prosodic features in the audio and facial motions exhibited on the face can help the computer make some inferences about the user's emotional state, assuming the users are willing to show their emotions. Another problem it addresses is the coupling between voice and the facial expression. Sometimes the user moves the lips to produce the speech, and sometimes the user only exhibits facial expression without speaking any words. Therefore, it is important to handle these two modalities accordingly. In particular, a pure “facial expression detector” will not function properly when the person is speaking, and a pure “vocal emotion recognizer” is useless when the user is not speaking. In this thesis, a complementary relationship between audio and video is proposed. Although these two modalities do not couple strongly in time, they seem to complement each other. In some cases, similar facial expressions may have different vocal characteristics, and vocal emotions having similar properties may have distinct facial behaviors.
---
paper_title: Perception of non-verbal emotional listener feedback

paper_content:

This paper reports on a listening test assessing the perception of short non-verbal emotional vocalisations emitted by a listener as feedback to the speaker. We clarify the concepts backchannel and feedback, and investigate the use of affect bursts as a means of giving emotional feedback via the backchannel. Experiments with German and Dutch subjects confirm that the recognition of emotion from affect bursts in a dialogical context is similar to their perception in isolation. We also investigate the acceptability of affect bursts when used as listener feedback. Acceptability appears to be linked to display rules for emotion expression. While many ratings were similar between Dutch and German listeners, a number of clear differences was found, suggesting language-specific affect bursts.
---
paper_title: Fundamentals Of Speech Recognition

paper_content:

1. Fundamentals of Speech Recognition. 2. The Speech Signal: Production, Perception, and Acoustic-Phonetic Characterization. 3. Signal Processing and Analysis Methods for Speech Recognition. 4. Pattern Comparison Techniques. 5. Speech Recognition System Design and Implementation Issues. 6. Theory and Implementation of Hidden Markov Models. 7. Speech Recognition Based on Connected Word Models. 8. Large Vocabulary Continuous Speech Recognition. 9. Task-Oriented Applications of Automatic Speech Recognition.
---
paper_title: Speaker recognition: A tutorial

paper_content:

A tutorial on the design and development of automatic speaker-recognition systems is presented. Automatic speaker recognition is the use of a machine to recognize a person from a spoken phrase. These systems can operate in two modes: to identify a particular person or to verify a person's claimed identity. Speech processing and the basic components of automatic speaker-recognition systems are shown and design tradeoffs are discussed. Then, a new automatic speaker-recognition system is given. This recognizer performs with 98.9% correct decalcification. Last, the performances of various systems are compared.
---
paper_title: The importance of the sense of touch in virtual and real environments

paper_content:

What would be worse, losing your sight or your sense of touch? Although touch (more generally, somesthesis) is commonly underrated, major somesthetic loss can't be adequately compensated for by sight. It results in catastrophic impairments of hand dexterity, haptic capabilities, walking, perception of limb position, and so on. Providing users with inadequate somesthetic feedback in virtual environments might impair their performance, just as major somesthetic loss does
---
paper_title: Multimodal human-computer interaction: A survey

paper_content:

In this paper we review the major approaches to multimodal human computer interaction from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition, and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for Multimodal Human Computer Interaction (MMHCI) research.
---
paper_title: Determining driver visual attention with one camera

paper_content:

This paper presents a system for analyzing human driver visual attention. The system relies on estimation of global motion and color statistics to robustly track a person's head and facial features. The system is fully automatic, it can initialize automatically, and reinitialize when necessary. The system classifies rotation in all viewing directions, detects eye/mouth occlusion, detects eye blinking and eye closure, and recovers the three dimensional gaze of the eyes. In addition, the system is able to track both through occlusion due to eye blinking, and eye closure, large mouth movement, and also through occlusion due to rotation. Even when the face is fully occluded due to rotation, the system does not break down. Further the system is able to track through yawning, which is a large local mouth motion. Finally, results are presented, and future work on how this system can be used for more advanced driver visual attention monitoring is discussed.
---
paper_title: A Survey of Research on Context-Aware Homes

paper_content:

The seamless integration of people, devices and computation will soon become part of our daily life. Sensors, actuators, wireless networks and ubiquitous devices powered by intelligent computation will blend into future environments in which people will live. Despite showing great promise, research into future computing technologies is often far removed from the needs of users. The nature of such future systems is often too obtrusive, seemingly denying their purpose. Furthermore, most research on context-aware environments and ubiquitous computing conducted so far has concentrated on supporting people while at work. This paper presents research issues that need to be addressed to enhance the quality of life for people living in context-aware homes. We survey current research and present strategies that facilitate the diffusion of information technology into homes in order to inspire positive emotions, encourage effortless exploration of content and help occupants to achieve tasks at hand.
---
paper_title: Automatic analysis of multimodal group actions in meetings

paper_content:

This paper investigates the recognition of group actions in meetings. A framework is employed in which group actions result from the interactions of the individual participants. The group actions are modeled using different HMM-based approaches, where the observations are provided by a set of audiovisual features monitoring the actions of individuals. Experiments demonstrate the importance of taking interactions into account in modeling the group actions. It is also shown that the visual modality contains useful information, even for predominantly audio-based events, motivating a multimodal approach to meeting analysis.
---
paper_title: Building Multi-modal Personal Sales Agents as Interfaces to E-commerce Applications

paper_content:

The research presented explores a new paradigm for human-computer interaction with electronic retailing applications. A paradigm that deploys face-to-face interaction with intelligent, visual, lifelike, multimodal conversational agents, which take on the role of electronic sales assistants. This paper discusses the motivations for enriching current e-commerce application interfaces with multi-modal interface agents and discusses the technical development issues they raise, as realised in the MAPPA (EU project EP28831) system architecture design and development.The paper addresses three distinct components of an overall framework for developing lifelike, multi-modal agents for real-time and dynamic applications: Knowledge Representation and Manipulation, Grounded Affect Models, and the convergence of both into support for multimedia visualisation of lifelike, social behaviour. The research presents a novel specification for such a medium and a functional agent-based system scenario (e-commerce) that is implemented with it. Setting forth a framework for building multi-modal interface agents and yielding a conversational form of human-machine interaction, which may have potential for shaping tomorrows interface to the world of e-commerce.
---
paper_title: “Put-that-there”: Voice and gesture at the graphics interface

paper_content:

Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality. The work described herein involves the user commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.
---
paper_title: A probabilistic approach to reference resolution in multimodal user interfaces

paper_content:

Multimodal user interfaces allow users to interact with computers through multiple modalities, such as speech, gesture, and gaze. To be effective, multimodal user interfaces must correctly identify all objects which users refer to in their inputs. To systematically resolve different types of references, we have developed a probabilistic approach that uses a graph-matching algorithm. Our approach identifies the most probable referents by optimizing the satisfaction of semantic, temporal, and contextual constraints simultaneously. Our preliminary user study results indicate that our approach can successfully resolve a wide variety of referring expressions, ranging from simple to complex and from precise to ambiguous ones.
---
paper_title: Rules of Play: Game Design Fundamentals

paper_content:

This text offers an introduction to game design and a unified model for looking at all kinds of games, from board games and sports to computer and video games. Also included are concepts, strategies, and methodologies for creating and understanding games.
---
paper_title: MATCHkiosk: A Multimodal Interactive City Guide

paper_content:

Multimodal interfaces provide more flexible and compelling interaction and can enable public information kiosks to support more complex tasks for a broader community of users. MATCHKiosk is a multimodal interactive city guide which provides users with the freedom to interact using speech, pen, touch or multimodal inputs. The system responds by generating multimodal presentations that synchronize synthetic speech with a life-like virtual agent and dynamically generated graphics.
---
paper_title: Assistive multimodal system based on speech recognition and head tracking

paper_content:

In this paper the assistive multimodal system is presented, which is aimed for the disabled people, which need other kinds of interfaces than ordinary people. The group of users of this system is persons with hands disabilities. The interaction between a user and a machine is performed by voice and head movements. It gives the opportunity for disabled people to carry out a work with PC. The work of the multimodal systems is presented during EUSIPCO-2005 Conference in framework of Similar Demonstration Session “Multimedia Tools for Disabled”.
---
paper_title: A probabilistic approach to reference resolution in multimodal user interfaces

paper_content:

Multimodal user interfaces allow users to interact with computers through multiple modalities, such as speech, gesture, and gaze. To be effective, multimodal user interfaces must correctly identify all objects which users refer to in their inputs. To systematically resolve different types of references, we have developed a probabilistic approach that uses a graph-matching algorithm. Our approach identifies the most probable referents by optimizing the satisfaction of semantic, temporal, and contextual constraints simultaneously. Our preliminary user study results indicate that our approach can successfully resolve a wide variety of referring expressions, ranging from simple to complex and from precise to ambiguous ones.
---
paper_title: MATCHkiosk: A Multimodal Interactive City Guide

paper_content:

Multimodal interfaces provide more flexible and compelling interaction and can enable public information kiosks to support more complex tasks for a broader community of users. MATCHKiosk is a multimodal interactive city guide which provides users with the freedom to interact using speech, pen, touch or multimodal inputs. The system responds by generating multimodal presentations that synchronize synthetic speech with a life-like virtual agent and dynamically generated graphics.
---
paper_title: Assistive multimodal system based on speech recognition and head tracking

paper_content:

In this paper the assistive multimodal system is presented, which is aimed for the disabled people, which need other kinds of interfaces than ordinary people. The group of users of this system is persons with hands disabilities. The interaction between a user and a machine is performed by voice and head movements. It gives the opportunity for disabled people to carry out a work with PC. The work of the multimodal systems is presented during EUSIPCO-2005 Conference in framework of Similar Demonstration Session “Multimedia Tools for Disabled”.
---
paper_title: Human computing and machine understanding of human behavior: a survey

paper_content:

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, humanlike interactive functions including understanding and emulating certain human behaviors such as affecti0ve and social signaling. This article discusses how far are we from enabling computers to understand human behavior.
---
paper_title: Automatic prediction of frustration

paper_content:

Predicting when a person might be frustrated can provide an intelligent system with important information about when to initiate interaction. For example, an automated Learning Companion or Intelligent Tutoring System might use this information to intervene, providing support to the learner who is likely to otherwise quit, while leaving engaged learners free to discover things without interruption. This paper presents the first automated method that assesses, using multiple channels of affect-related information, whether a learner is about to click on a button saying ''I'm frustrated.'' The new method was tested on data gathered from 24 participants using an automated Learning Companion. Their indication of frustration was automatically predicted from the collected data with 79% accuracy (chance=58%). The new assessment method is based on Gaussian process classification and Bayesian inference. Its performance suggests that non-verbal channels carrying affective cues can help provide important information to a system for formulating a more intelligent response.
---
paper_title: Analysis of emotion recognition using facial expressions, speech and multimodal information

paper_content:

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on facial expressions or speech, relatively limited work has been done to fuse these two, and other, modalities to improve the accuracy and robustness of the emotion recognition system. This paper analyzes the strengths and the limitations of systems based only on facial expressions or acoustic information. It also discusses two approaches used to fuse these two modalities: decision level and feature level integration. Using a database recorded from an actress, four emotions were classified: sadness, anger, happiness, and neutral state. By the use of markers on her face, detailed facial motions were captured with motion capture, in conjunction with simultaneous speech recordings. The results reveal that the system based on facial expression gave better performance than the system based on just acoustic information for the emotions considered. Results also show the complementarily of the two modalities and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.
---
paper_title: Bi-modal emotion recognition from expressive face and body gestures

paper_content:

Psychological research findings suggest that humans rely on the combined visual channels of face and body more than any other channel when they make judgments about human communicative behavior. However, most of the existing systems attempting to analyze the human nonverbal behavior are mono-modal and focus only on the face. Research that aims to integrate gestures as an expression mean has only recently emerged. Accordingly, this paper presents an approach to automatic visual recognition of expressive face and upper-body gestures from video sequences suitable for use in a vision-based affective multi-modal framework. Face and body movements are captured simultaneously using two separate cameras. For each video sequence single expressive frames both from face and body are selected manually for analysis and recognition of emotions. Firstly, individual classifiers are trained from individual modalities. Secondly, we fuse facial expression and affective body gesture information at the feature and at the decision level. In the experiments performed, the emotion classification using the two modalities achieved a better recognition accuracy outperforming classification using the individual facial or bodily modality alone.
---
paper_title: Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures

paper_content:

This paper presents an architecture for fusion of multimodal input streams for natural interaction with a humanoid robot as well as results from a user study with our system. The presented fusion architecture consists of an application independent parser of input events, and application specific rules. In the presented user study, people could interact with a robot in a kitchen scenario, using speech and gesture input. In the study, we could observe that our fusion approach is very tolerant against falsely detected pointing gestures. This is because we use speech as the main modality and pointing gestures mainly for disambiguation of objects. In the paper we also report about the temporal correlation of speech and gesture events as observed in the user study.
---
paper_title: Building a Multimodal Human-Robot Interface

paper_content:

When we begin to build and interact with machines or robots that either look like humans or have human functionalities and capabilities, then people may well interact with their human-like machines in ways that mimic human-human communication. For example, if a robot has a face, a human might interact with it similarly to how humans interact with other creatures with faces, Specifically, a human might talk to it, gesture to it, smile at it, and so on. If a human interacts with a computer or a machine that understands spoken commands, the human might converse with the machine, expecting it to have competence in spoken language. In our research on a multimodal interface to mobile robots, we have assumed a model of communication and interaction that, in a sense, mimics how people communicate. Our interface therefore incorporates both natural language understanding and gesture recognition as communication modes. We limited the interface to these two modes to simplify integrating them in the interface and to make our research more tractable. We believe that with an integrated system, the user is less concerned with how to communicate (which interactive mode to employ for a task), and is therefore free to concentrate on the tasks and goals at hand. Because we integrate all our system's components, users can choose any combination of our interface's modalities. The onus is on our interface to integrate the input, process it, and produce the desired results.
---

Title: Human-Computer Interaction: Overview on State of the Art

Section 1: Introduction
Description 1: Provide an introduction to the evolution and the importance of Human-Computer Interaction (HCI), including the main focus of the paper.

Section 2: Overview on HCI
Description 2: Present a general overview of HCI, including an examination of existing technologies and the future direction of HCI research.

Section 3: Existing HCI Technologies
Description 3: Discuss the different existing HCI technologies, categorized by the human senses they engage (vision, audition, touch) and example devices.

Section 4: Recent Advances in HCI
Description 4: Explore recent advancements in the field, with a focus on intelligent and adaptive interfaces, and ubiquitous computing.

Section 5: Intelligent and Adaptive HCI
Description 5: Detail the design and functionality of intelligent and adaptive interfaces, including examples and applications.

Section 6: Ubiquitous Computing and Ambient Intelligence
Description 6: Describe the concept of ubiquitous computing, its history, and its significance as the third wave of computing.

Section 7: HCI Systems Architecture
Description 7: Explain the different configurations and designs of HCI systems, focusing on unimodal and multimodal systems.

Section 8: Unimodal HCI Systems
Description 8: Discuss unimodal HCI systems, including visual-based, audio-based, and sensor-based interactions.

Section 9: Multimodal HCI Systems
Description 9: Describe multimodal HCI systems which combine multiple modalities to enhance user interaction.

Section 10: Applications
Description 10: Highlight different real-world applications of multimodal HCI systems and their advantages over traditional interfaces.

Section 11: Multimodal Systems for Disabled people
Description 11: Explain how multimodal systems can be used to assist disabled individuals using alternative interfaces.

Section 12: Emotion Recognition Multimodal Systems
Description 12: Discuss systems that recognize human emotions through multimodal signals for better HCI.

Section 13: Map-Based Multimodal Applications
Description 13: Detail map-based multimodal applications and the integration of multiple input methods like speech and gestures.

Section 14: Multimodal Human-Robot Interface Applications
Description 14: Explain human-robot interfaces using multimodal inputs and their practical applications.

Section 15: Multi-Modal HCI in Medicine
Description 15: Highlight the applications of multimodal HCI in medical fields, specifically in neuro-surgery.

Section 16: Conclusion
Description 16: Summarize the key points discussed and emphasize the importance of evolving HCI towards natural, adaptive methods.