A survey of assistive technologies and applications for blind users on mobile platforms: a review and foundation for research
---
paper_title: Pygmalion: A COMPUTER PROGRAM TO Model and Stimulate Creative Thought

paper_content:

iii Preface The following is a map of this document. Chapters 1,2 --A psychological model of creative thought, forming the basis for the PYGMALION design principles. Chapter 3 --Other projects which adhere to some of the same principles. Chapters 4,5 --The PYGMALION programming environment in detail. Chapter 6 --Examples of PYGMALION programs and data structures. Chapter 7 --Conclusions and suggestions for the future. This paper places equal emphasis oil presenting a psychological model of thought and using the model in a computer environment. Readers interested in aspects of creative thoug.ht which can be assisted by a computer should read chapters I and 2. Readers interested in how the PYGMALION system attempts to stimulate creative thought should look at chapter 6 (mostly pictures) to get the flavor, then read chapters 4 and 5. The works of others which deal with the same aspects are described in chapter 3. Chapter 7 suggests areas for future exploration. Thorough readers will read the chapters in order. Chapter 6 and 4-A through 4-D are a minimal set for readers in a hurry. There are three parts to this report.
---
paper_title: Overview of auditory representations in human-machine interfaces

paper_content:

In recent years, a large number of research projects have focused on the use of auditory representations in a broadened scope of application scenarios. Results in such projects have shown that auditory elements can effectively complement other modalities not only in the traditional desktop computer environment but also in virtual and augmented reality, mobile platforms, and other kinds of novel computing environments. The successful use of auditory representations in this growing number of application scenarios has in turn prompted researchers to rediscover the more basic auditory representations and extend them in various directions. The goal of this article is to survey both classical auditory representations (e.g., auditory icons and earcons) and those auditory representations that have been created as extensions to earlier approaches, including speech-based sounds (e.g., spearcons and spindex representations), emotionally grounded sounds (e.g., auditory emoticons and spemoticons), and various other sound types used to provide sonifications in practical scenarios. The article concludes by outlining the latest trends in auditory interface design and providing examples of these trends.
---
paper_title: Tactons: Structured Tactile Messages for Non-Visual Information Display

paper_content:

Tactile displays are now becoming available in a form that can be easily used in a user interface. This paper describes a new form of tactile output. Tactons, or tactile icons, are structured, abstract messages that can be used to communicate messages non-visually. A range of different parameters can be used for Tacton construction including: frequency, amplitude and duration of a tactile pulse, plus other parameters such as rhythm and location. Tactons have the potential to improve interaction in a range of different areas, particularly where the visual display is overloaded, limited in size or not available, such as interfaces for blind people or in mobile and wearable devices. This paper describes Tactons, the parameters used to construct them and some possible ways to design them. Examples of where Tactons might prove useful in user interfaces are given.
---
paper_title: Perceptual design of haptic icons

paper_content:

The bulk of applications for haptic feedback employ direct rendering approaches wherein a user touches a virtual model of some "real" thing, often displayed graphically as well. We propose a new class of applications based on abstract messages, ranging from "haptic icons" - brief signals conveying an ob- ject's or event's state, function or content - to an expressive haptic language for interpersonal communication. Building this language requires us to understand how synthetic haptic signals are perceived, and what they can mean to us. Experiments presented here address the perception question by using an effi- cient version of Multidimensional Scaling (MDS) to extract perceptual axes for complex haptic icons: once this space is mapped, icons can be designed to maximize both differentiability and individual salience. Results show that a set of icons constructed by varying the frequency, magnitude and shape of 2-sec, time-invariant wave shapes map to two perceptual axes, which differ depending on the signals' frequency range; and suggest that expressive capability is maxi- mized in one frequency subspace.
---
paper_title: Spectral discrimination thresholds comparing audio and haptics for complex stimuli

paper_content:

Individuals with normal hearing are generally able to discriminate auditory stimuli that have the same fundamental frequency but different spectral content. This study concerns to what extent it is possible to perform the same differentiation considering vibratory tactile stimuli. Three perceptual experiments have been carried out in an attempt to compare discrimination thresholds in terms of spectral differences between auditory and vibratory tactile stimulations. The first test consists of assessing the subject's ability in discriminating between three signals with distinct spectral content. The second test focuses on the measurement of the discrimination threshold between a pure tone signal and a signal composed of two pure tones, varying the amplitude and frequency of the second tone. Finally, in the third test the discrimination threshold is measured between a tone with even harmonic components and a tone with odd ones. The results show that it is indeed possible to discriminate between haptic signals having the same fundamental frequency but different spectral. The threshold of sensitivity for detection is markedly less than for audio stimuli.
---
paper_title: Pygmalion: A COMPUTER PROGRAM TO Model and Stimulate Creative Thought

paper_content:

iii Preface The following is a map of this document. Chapters 1,2 --A psychological model of creative thought, forming the basis for the PYGMALION design principles. Chapter 3 --Other projects which adhere to some of the same principles. Chapters 4,5 --The PYGMALION programming environment in detail. Chapter 6 --Examples of PYGMALION programs and data structures. Chapter 7 --Conclusions and suggestions for the future. This paper places equal emphasis oil presenting a psychological model of thought and using the model in a computer environment. Readers interested in aspects of creative thoug.ht which can be assisted by a computer should read chapters I and 2. Readers interested in how the PYGMALION system attempts to stimulate creative thought should look at chapter 6 (mostly pictures) to get the flavor, then read chapters 4 and 5. The works of others which deal with the same aspects are described in chapter 3. Chapter 7 suggests areas for future exploration. Thorough readers will read the chapters in order. Chapter 6 and 4-A through 4-D are a minimal set for readers in a hurry. There are three parts to this report.
---
paper_title: Tone-2 tones discrimination task comparing audio and haptics

paper_content:

To investigating the capabilities of human beings to differentiate between tactile-vibratory stimuli with the same fundamental frequency but with different spectral content, this study concerns discrimination tasks comparing audio and haptic performances. Using an up-down 1 dB step adaptive procedure, the experimental protocol consists of measuring the discrimination threshold between a pure tone signal and a stimulus composed of two concurrent pure tones, changing the amplitude and frequency of the second tone. The task is performed employing exactly the same experimental apparatus (computer, AD-DA converters, amplifiers and drivers) for both audio and tactile modalities. The results show that it is indeed possible to discriminate between signals having the same fundamental frequency but different spectral content for both haptic and audio modalities, the latter being notably more sensitive. Furthermore, particular correlations have been found between the frequency of the second tone and the discrimination threshold values, for both audio and tactile modalities.
---
paper_title: Auditory Icons: Using Sound in Computer Interfaces

paper_content:

Carroll and Campbell have exercised themselves over a straw man not subscribed to by us. In doing so, they have misrepresented our position and even the statements in our paper. In reply, we restate as clearly as we can the position for which we actually did and do argue and give examples of their misrepresentations. The underlying issue seems to concern the advantages of using technical psychological theories to identify underlying mechanisms in human-computer interaction. We argue that such theories are an important part of a science of human-computer interaction. We argue further that technical theories must be considered in the context of the uses to which they are put. Such considerations help the theorist to determine what is a good approximation, the degree of formalization that is justified, the appropriate commingling of qualitative and quantitative techniques, and encourages cumulative progress through the heuristic of divide and conquer.
---
paper_title: Earcons and icons: their structure and common design principles

paper_content:

In this article we examine earcons, which are audio messages used in the user-computer interface to provide information and feedback to the user about computer entities. (Earcons include messages and functions, as well as states and labels.) We identify some design principles that are common to both visual symbols and auditory messages, and discuss the use of representational and abstract icons and earcons. We give some examples of audio patterns that may be used to design modules for earcons, which then may be assembled into larger groupings called families. The modules are single pitches or rhythmicized sequences of pitches called motives. The families are constructed about related motives that serve to identify a family of related messages. Issues concerned with learning and remembering earcons are discussed.
---
paper_title: The SonicFinder: an interface that uses auditory icons

paper_content:

The appropriate use of nonspeech sounds has the potential to add a great deal to the functionality of computer interfaces. Sound is a largely unexploited medium of output, even though it plays an integral role in our everyday encounters with the world, a role that is complementary to vision. Sound should be used in computers as it is in the world, where it conveys information about the nature of sound-producing events. Such a strategy leads to auditory icons, which are everyday sounds meant to convey information about computer events by analogy with everyday events. Auditory icons are an intuitively accessible way to use sound to provide multidimensional, organized information to users. ::: ::: These ideas are instantiated in the SonicFinder, which is an auditory interface I developed at Apple Computer, Inc. In this interface, information is conveyed using auditory icons as well as standard graphical feedback. I discuss how events are mapped to auditory icons in the SonicFinder, and illustrate how sound is used by describing a typical interaction with this interface. ::: ::: Two major gains are associated with using sound in this interface: an increase in direct engagement with the model world of the computer and an added flexibility for users in getting information about that world. These advantages seem to be due to the iconic nature of the mappings used between sound and the information it is to convey. I discuss sound effects and source metaphors as methods of extending auditory icons beyond the limitations implied by literal mappings, and I speculate on future directions for such interfaces.
---
paper_title: The hapticon editor: a tool in support of haptic communication research

paper_content:

We define haptic icons, or "hapticons", as brief programmed forces applied to a user through a haptic interface, with the role of communicating a simple idea in manner similar to visual or auditory icons. In this paper we present the design and implementation of an innovative software tool and graphical interface for the creation and editing of hapticons. The tool's features include various methods for creating new icons including direct recording of manual trajectories and creation from a choice of basis waveforms; novel direct-manipulation icon editing mechanisms, integrated playback and convenient storage of icons to file. We discuss some ways in which the tool has aided our research in the area of haptic iconography and present an innovative approach for generating and rendering simple textures on a low degree of freedom haptic device using what we call terrain display.
---
paper_title: Human Spatial Navigation via a Visuo-Tactile Sensory Substitution System

paper_content:

Spatial navigation within a real 3-D maze was investigated to study space perception on the sole basis of tactile information transmitted by means of a ‘tactile vision substitution system' (TVSS) allowing the conversion of optical images—collected by a micro camera—into ‘tactile images’ via a matrix in contact with the skin. The development of such a device is based on concepts of cerebral and functional plasticity, enabling subjective reproduction of visual images from tactile data processing. Blindfolded sighted subjects had to remotely control the movements of a robot on which the TVSS camera was mounted. Once familiarised with the cues in the maze, the subjects were given two exploration sessions. Performance was analysed according to an objective point of view (exploration time, discrimination capacity), as well as a subjective one (speech). The task was successfully carried out from the very first session. As the subjects took a different path during each navigation, a gradual improvement in perform...
---
paper_title: Brain plasticity: ‘visual’ acuity of blind persons via the tongue

paper_content:

Abstract The ‘visual’ acuity of blind persons perceiving information through a newly developed human–machine interface, with an array of electrical stimulators on the tongue, has been quantified using a standard Ophthalmological test (Snellen Tumbling E). Acuity without training averaged 20/860. This doubled with 9 h of training. The interface may lead to practical devices for persons with sensory loss such as blindness, and offers a means of exploring late brain plasticity.
---
paper_title: Characteristics of reading rate and manual scanning patterns of blind Optacon readers.

paper_content:

Tactual reading is slow compared with sighted reading, and the rate-limiting constraints imposed on tactile readers are only vaguely understood. Like the eye movements of sighted reading, the text-scanning hand movements of tactual reading provide a means to investigate the operative system. We examined the reading hand movements of 10 blind readers using the Optacon, the electronic reading aid used most commonly by blind readers. Subjects read texts of graded difficulty, and their reading hand movements were recorded.Rate and scanning measures were used to characterize reading performance. The group mean reading rate was found to be 28.2 words/min. Reading rate measured in letter spaces per minute was independent of text difficulty.
---
paper_title: Waypoint navigation with a vibrotactile waist belt

paper_content:

Presenting waypoint navigation on a visual display is not suited for all situations. The present experiments investigate if it is feasible to present the navigation information on a tactile display. Important design issue of the display is how direction and distance information must be coded. Important usability issues are the resolution of the display and its usefulness in vibrating environments. In a pilot study with 12 pedestrians, different distance-coding schemes were compared. The schemes translated distance to vibration rhythm while the direction was translated into vibration location. The display consisted of eight tactors around the user's waist. The results show that mapping waypoint direction on the location of vibration is an effective coding scheme that requires no training, but that coding for distance does not improve performance compared to a control condition with no distance information. In Experiment 2, the usefulness of the tactile display was shown in two case studies with a helicopter and a fast boat.
---
paper_title: Vibrotactile pattern perception: extraordinary observers.

paper_content:

Two sighted peopled showed a remarkable ability to perceive vibrotactile patterns generated by the Optacon, a reading aid for the blind. These individuals were able to read at very high rates, 70 to 100 words per minute, through their fingertips. Additional testing showed them to be much better than other people at discriminating and recognizing vibrotactile patterns.
---
paper_title: Tactile Guidance of Movement

paper_content:

In some prototype mobility aids for the blind information about the environment is obtained with the aid of patterns within a matrix of tactile point stimuli. The aim of this report is to summarize some experiments on the possibilities of guiding movements in 3-D space by devices of this kind, the movements studied being: (1) batting a ball, (2) walking and pointing to a target, and (3) slalom walking. The results were that movements could be guided by such a matrix with reasonable precision and time consumption. There are many remaining problems, especially in a cluttered environment, but they can be expected to be decreased if we are able to increase our knowledge about how touch, or rather the haptic system, is functioning, and if we can utilize this knowledge in constructing more effective tactile displays.
---
paper_title: Seeing with the brain

paper_content:

We see with the brain, not the eyes (Bach-y-Rita, 1972); images that pass through our pupils go no further than the retina. From there image information travels to the rest of the brain by means of coded pulse trains, and the brain, being highly plastic, can learn to interpret them in visual terms. Perceptual levels of the brain interpret the spatially encoded neural activity, modified and augmented by nonsynaptic and other brain plasticity mechanisms (Bach-y-Rita, 1972, 1995, 1999, in press). However, the cognitive value of that information is not merely a process of image analysis. Perception of the image relies on memory, learning, contextual interpretation (e.g., we perceive intent of the driver in the slight lateral movements of a car in front of us on the highway), cultural, and other social factors that are probably exclusively human characteristics that provide “qualia” (Bach-y-Rita, 1996b). This is the basis for our tactile vision substitution system (TVSS) studies that, starting in 1963, have demonstrated that visual information and the subjective qualities of seeing can be obtained tactually using sensory substitution systems.1 The description of studies with this system have been taken INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION, 15(2), 285–295 Copyright © 2003, Lawrence Erlbaum Associates, Inc.
---
paper_title: Braille Display by Lateral Skin Deformation with the STReSS2 Tactile Transducer

paper_content:

Earlier work with a 1D tactile transducer demonstrated that lateral skin deformation is sufficient to produce sensations similar to those felt when brushing a finger against a line of Braille dots. Here, we extend this work to the display of complete 6-dot Braille characters using a general purpose 2D tactile transducer called STReSS2. The legibility of the produced Braille was evaluated by asking seven expert Braille readers to identify meaningless 5-letter strings as well as familiar words. Results indicate that reading was difficult but possible for most individuals. The superposition of texture to the sensation of a dot improved performance. The results contain much information to guide the design of a specialized Braille display operating by lateral skin deformation. They also suggest that rendering for contrast rather than realism may facilitate Braille reading when using a weak tactile transducer
---
paper_title: Tactile letter recognition: Pattern duration and modes of pattern generation

paper_content:

Measurements were made of the ability of subjects to identify vibrotactile patterns presented to their fingertips. The patterns were letters of the alphabet generated on the tactile display of the Optacon. Five different modes of pattern generation were examined. Two of the modes, static and scan, involved full-field presentations of the letters. In the remaining three modes, patterns were generated by presenting parts of the letters sequentially. In one mode, the letters were exposed by a slit passing across them. In the other two modes, the patterns were generated as though the letter were being drawn on the skin. Performance in all five modes was examined as a function of pattern duration, with durations ranging from 4 to 1,000 msec. Increasing duration, up to 400 msec, resulted in generally improved performance, although the functions relating performance and duration differed according to the mode of presentation. Contrary to previous results, the static mode produced the best overall performance level. Some possible reasons for the disagreement between the present results and previous results and some models of cutaneous pattern recognition are discussed.
---
paper_title: SWAN: System for Wearable Audio Navigation

paper_content:

Wearable computers can certainly support audio-only presentation of information; a visual interface need not be present for effective user interaction. A system for wearable audio navigation (SWAN) is being developed to serve as a navigation and orientation aid for persons temporarily or permanently visually impaired. SWAN is a wearable computer consisting of audio-only output and tactile input via a handheld interface. SWAN aids a user in safe pedestrian navigation and includes the ability for the user to author new GIS data relevant to their needs of wayfinding, obstacle avoidance, and situational awareness support. Emphasis is placed on representing pertinent data with non-speech sounds through a process of sonification. SWAN relies on a geographic information system (GIS) infrastructure for supporting geocoding and spatialization of data. Furthermore, SWAN utilizes novel tracking technology.
---
paper_title: Generalized learning of visual-to-auditory substitution in sighted individuals

paper_content:

Abstract Visual-to-auditory substitution involves delivering information about the visual world using auditory input. Although the potential suitability of sound as visual substitution has previously been demonstrated, the basic mechanism behind crossmodal learning is largely unknown; particularly, the degree to which learning generalizes to new stimuli has not been formally tested. We examined learning processes involving the use of the image-to-sound conversion system developed by Meijer [Meijer, P., 1992. An experimental system for auditory image representations. IEEE Trans Biom Eng. 39 (2), 112-121.] that codes visual vertical and horizontal axes into frequency and time representations, respectively. Two behavioral experiments provided training to sighted individuals in a controlled environment. The first experiment explored the early learning stage, comparing performance of individuals who received short-term training and those who were only explicitly given the conversion rules. Both groups performed above chance, suggesting an intuitive understanding of the image–sound relationship; the lack of group difference indicates that this intuition could be acquired simply on the basis of explicit knowledge. The second experiment involved training over a three-week period using a larger variety of stimuli. Performance on both previously trained and novel items was examined over time. Performance on the familiar items was higher than on the novel items, but performance on the latter improved over time. While the lack of improvement with the familiar items suggests memory-based performance, the improvement with novel items demonstrated generalized learning, indicating abstraction of the conversion rules such that they could be applied to interpret auditory patterns coding new visual information. Such generalization could provide a basis for the substitution in a constantly changing visual environment.
---
paper_title: A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution

paper_content:

The rehabilitation of blindness, using noninvasive methods, requires sensory substitution. A theoretical frame for sensory substitution has been proposed (C. Veraart, 1989) which consists of a model of the deprived sensory system connected to an inverse model of the substitutive sensory system. This paper addresses the feasibility of this conceptual model in the case of auditory substitution, and its implementation as a rough model of the retina connected to an inverse linear model of the cochlea. The authors have developed an experimental prototype. It aims at allowing optimization of the sensory substitution process. This prototype is based on a personal computer which is connected to a miniature head-fixed video camera and to headphones. A visual scene is captured. Image processing achieves edge detection and graded resolution. Each picture element (pixel) of the processed image is assigned a sinusoidal tone; weighted summation of these sinewaves builds up a complex auditory signal which is transduced by the headphones. On-line selection of various parameters and real-time functioning of the device allow optimization of parameters during psychophysical experimentations. Assessment of this implementation has been initiated, and has so far demonstrated prototype usefulness for pattern recognition. An integrated circuit of this system is to be developed.
---
paper_title: Navigation Performance With a Virtual Auditory Display: Effects of Beacon Sound, Capture Radius, and Practice

paper_content:

OBJECTIVE: We examined whether spatialized nonspeech beacons could guide navigation and how sound timbre, waypoint capture radius, and practice affect performance. BACKGROUND: Auditory displays may assist mobility and wayfinding for those with temporary or permanent visual impairment, but they remain understudied. Previous systems have used speech-based interfaces. METHOD: Participants (108 undergraduates) navigated three maps, guided by one of three beacons (pink noise, sonar ping, or 1000-Hz pure tone) spatialized by a virtual reality engine. Dependent measures were efficiency of time and path length. RESULTS: Overall navigation was very successful, with significant effects of practice and capture radius, and interactions with beacon sound. Overshooting and subsequent hunting for waypoints was exacerbated for small radius conditions. A human-scale capture radius (1.5 m) and sonar-like beacon yielded the optimal combination for safety and efficiency. CONCLUSION: The selection of beacon sound and capture radius depend on the specific application, including whether speed of travel or adherence to path are of primary concern. Extended use affects sound preferences and quickly leads to improvements in both speed and accuracy. APPLICATION: These findings should lead to improved wayfinding systems for the visually impaired as well as for first responders (e.g., firefighters) and soldiers. Language: en
---
paper_title: An application of bio-feedback in the rehabilitation of the blind.

paper_content:

The long term blind exhibit diminished awareness of limb position; kinaesthetic feedback alone providing insufficient positional information. Experiments to evaluate simple hand held electronic travel aids for the blind have shown that failure to hold the aid in the correct orientation leads to a failure to detect important hazards. Consequently some potential users may come to reject the aid. Experimental results show that by providing an auditory alarm signal whenever an aid is not held horizontally it is possible to train subjects to maintain the correct holding position. The positive effects of short periods of training with feedback are retained when the feedback is removed. It is proposed to make available a small number of modified (with feedback) aids for use during the early stages of client training.
---
paper_title: A sonar aid to enhance spatial perception of the blind: engineering design and evaluation

paper_content:

The design of an air sonar device with a new form of binaural display is described which aids the blind in perceiving their environment. Some of the limitations of knowledge of human perception and the influence this has on a specification for the device are discussed. Inherent limitations in the binaural aid both in terms of technology development and performance are also explained. The paper described what is expected of the man-machine control system in a mobility setting and discusses the technique of evaluating a manmachinesystem so as to assess the machine performance.
---
paper_title: Real-time assistance prototype — A new navigation aid for blind people

paper_content:

This paper presents a new prototype for being used as a travel aid for blind people. The system is developed to complement traditional navigation systems such as white cane and guide dogs. The system consists of two stereo cameras and a portable computer for processing the environmental information. The aim of the system is to detect the static and dynamic objects from the surrounding environment and transform them into acoustical signals. Through stereophonic headphones, the user perceives the acoustic image of the environment, the volume of the objects, moving object direction and trajectory, its distance relative to the user and the free paths in a range of 5m to 15m. The acoustic signals represent short train of delta sounds externalized with non-individual Head-Related Transfer Functions generated in an anechoic chamber. Experimental results show that users were able to control and navigate with the system safety both in familiar and unfamiliar environments.
---
paper_title: CREATING AND ACCESSING AUDIOTACTILE IMAGES WITH "HFVE" VISION SUBSTITUTION SOFTWARE

paper_content:

The HFVE (Heard and Felt Vision Effects) vision substitution system uses moving speech-like sounds and tactile effects to present aspects of visual images. This paper describes several audio- and interaction-related improvements. A separate "buzz track" allows more accurate perception of shape, and additional sound cues can be added to this new track, instead of distorting the speech. Details are given of improved ways of presenting image “layout”, and the HFVE approach is compared to other audio vision substitution systems. Blind users can create or add to images using a standard computer mouse (or joystick), by hearing similar sound cues. Finally, a facility for defining and capturing material visible on a computer screen is described.
---
paper_title: Auditory Information Design

paper_content:

The prospect of computer applications making “noises” is disconcerting to some. Yet the soundscape of the real world does not usually bother us. Perhaps we only notice a nuisance? This thesis is an approach for designing sounds that are useful information rather than distracting “noise”. The approach is called TaDa because the sounds are designed to be useful in a Task and true to the Data. ::: Previous researchers in auditory display have identified issues that need to be addressed for the field to progress. The TaDa approach is an integrated approach that addresses an array of these issues through a multifaceted system of methods drawn from HCI, visualisation, graphic design and sound design. A task-analysis addresses the issue of usefulness. A data characterisation addresses perceptual faithfulness. A case-based method provides semantic linkage to the application domain. A rule-based method addresses psychoacoustic control. A perceptually linearised sound space allows transportable auditory specifications. Most of these methods have not been used to design auditory displays before, and each has been specially adapted for this design domain. ::: The TaDa methods have been built into computer-aided design tools that can assist the design of a more effective display, and may allow less than experienced designers to make effective use of sounds. The case-based method is supported by a database of examples that can be searched by an information analysis of the design scenario. The rule-based method is supported by a direct manipulation interface which shows the available sound gamut of an audio device as a 3D coloured object that can be sliced and picked with the mouse. These computer-aided tools are the first of their kind to be developed in auditory display. ::: The approach, methods and tools are demonstrated in scenarios from the domains of mining exploration, resource monitoring and climatology. These practical applications show that sounds can be useful in a wide variety of information processing activities which have not been explored before. The sounds provide information that is difficult to obtain visually, and improve the directness of interactions by providing additional affordances.
---
paper_title: Accessing Audiotactile Images with HFVE Silooet

paper_content:

In this paper, recent developments of the HFVE vision-substitution system are described; and the initial results of a trial of the "Silooet" software are reported. The system uses audiotactile methods to present features of visual images to blind people. Included are details of presenting objects found in prepared media and live images; object-related layouts and moving effects (including symbolic paths); and minor enhancements that make the system more practical to use. Initial results are reported from a pilot study that tests the system with untrained users.
---
paper_title: A computer-vision based sensory substitution device for the visually impaired (See ColOr)

paper_content:

Audio-based Sensory Substitution Devices (SSDs) perform adequately when sensing and mapping low-level visual features into sound. Yet, their limitations become apparent when it comes to represent high-level or conceptual information involved in vision. We introduce See ColOr as an SSD that senses color and depth to convert them into musical instrument sounds. In addition and unlike any other approach, our SSD extends beyond a sensing prototype, by integrating computer vision methods to produce reliable knowledge about the physical world (effortlessly for the user). Experiments reported in this thesis reveal that our See ColOr SSD is learnable, functional, and provides easy interaction. In moderate time, participants were able to grasp visual information from the environment out of which they could derive: spatial awareness, ability to find someone, location of daily objects, and skill to walk safely avoiding obstacles. Our encouraging results open a door towards autonomous mobility of the blind.
---
paper_title: From software product lines to software ecosystems

paper_content:

Software product line companies increasingly expand their platform outside their organizational boundaries, in effect transitioning to a software ecosystem approach. In this paper, we discuss the emerging trend of software ecosystems and provide a overview of the key concepts and implications of adopting a software ecosystem approach. We define the notion of software ecosystems and introduce a taxonomy. Finally, we explore the implications of software ecosystems to the way companies build software.
---
paper_title: From continuous improvement to collaborative innovation: the next challenge in supply chain management

paper_content:

This paper considers the growing importance of inter-company collaboration, and develops the concept of intra-company continuous improvement through to what may be termed collaborative innovation between members of an extended manufacturing enterprise (EME). The importance of ICTs to such company networks is considered but research has shown that no amount of technology can overcome a lack of trust and ineffective goal setting between key partners involved in the cross-company projects. Different governance models may also impact on the success or otherwise of the network. This paper provides an overview of the main topics considered in this Special Issue.
---
paper_title: Gyrophone: recognizing speech from gyroscope signals

paper_content:

We show that the MEMS gyroscopes found on modern smart phones are sufficiently sensitive to measure acoustic signals in the vicinity of the phone. The resulting signals contain only very low-frequency information (<200Hz). Nevertheless we show, using signal processing and machine learning, that this information is sufficient to identify speaker information and even parse speech. Since iOS and Android require no special permissions to access the gyro, our results show that apps and active web content that cannot access the microphone can nevertheless eavesdrop on speech in the vicinity of the phone.
---
paper_title: Towards Real-time Emergency Response using Crowd Supported Analysis of Social Media

paper_content:

This position paper outlines an ongoing research project that aims to incorporate crowdsourcing as part of an emergency response system. The proposed system's novelty is that it integrates crowdsourcing into its architecture to analyze and structure social media content posted by microbloggers and service users, including emergency response coordinators and victims, during the event or disaster. An important challenge in this approach is identifying appropriate tasks to crowdsource, and adopting effective motivation strategies. Author Keywords Emergency response, social media, crowdsourcing, text mining.
---
paper_title: A crowdsourcing platform for the construction of accessibility maps

paper_content:

We present in this article a crowdsourcing platform that enables the collaborative creation of accessibility maps. The platform provides means for integration of different kind of data, collected automatically or with user intervention, to augment standard maps with accessibility information. The article shows the architecture of the platform, dedicating special attention to the smartphone applications we developed for data collection. The article also describes a preliminar experiment conducted on field, showing how the analysis of data produced by our solution can bring novel insights in accessibility challenges that can be found in cities.
---
paper_title: Medusa: a programming framework for crowd-sensing applications

paper_content:

The ubiquity of smartphones and their on-board sensing capabilities motivates crowd-sensing, a capability that harnesses the power of crowds to collect sensor data from a large number of mobile phone users. Unlike previous work on wireless sensing, crowd-sensing poses several novel requirements: support for humans-in-the-loop to trigger sensing actions or review results, the need for incentives, as well as privacy and security. Beyond existing crowd-sourcing systems, crowd-sensing exploits sensing and processing capabilities of mobile devices. In this paper, we design and implement Medusa, a novel programming framework for crowd-sensing that satisfies these requirements. Medusa provides high-level abstractions for specifying the steps required to complete a crowd-sensing task, and employs a distributed runtime system that coordinates the execution of these tasks between smartphones and a cluster on the cloud. We have implemented ten crowd-sensing tasks on a prototype of Medusa. We find that Medusa task descriptions are two orders of magnitude smaller than standalone systems required to implement those crowd-sensing tasks, and the runtime has low overhead and is robust to dynamics and resource attacks.
---
paper_title: Combining crowdsourcing and google street view to identify street-level accessibility problems

paper_content:

Poorly maintained sidewalks, missing curb ramps, and other obstacles pose considerable accessibility challenges; however, there are currently few, if any, mechanisms to determine accessible areas of a city a priori. In this paper, we investigate the feasibility of using untrained crowd workers from Amazon Mechanical Turk (turkers) to find, label, and assess sidewalk accessibility problems in Google Street View imagery. We report on two studies: Study 1 examines the feasibility of this labeling task with six dedicated labelers including three wheelchair users; Study 2 investigates the comparative performance of turkers. In all, we collected 13,379 labels and 19,189 verification labels from a total of 402 turkers. We show that turkers are capable of determining the presence of an accessibility problem with 81% accuracy. With simple quality control methods, this number increases to 93%. Our work demonstrates a promising new, highly scalable method for acquiring knowledge about sidewalk accessibility.
---
paper_title: Dissemination in opportunistic mobile ad-hoc networks: The power of the crowd

paper_content:

Opportunistic ad-hoc communication enables portable devices such as smartphones to effectively exchange information, taking advantage of their mobility and locality. The nature of human interaction makes information dissemination using such networks challenging. We use three different experimental traces to study fundamental properties of human interactions. We break our traces down in multiple areas and classify mobile users in each area according to their social behavior: Socials are devices that show up frequently or periodically, while Vagabonds represent the rest of the population. We find that in most cases the majority of the population consists of Vagabonds. We evaluate the relative role of these two groups of users in data dissemination. Surprisingly, we observe that under certain circumstances, which appear to be common in real life situations, the effectiveness of dissemination predominantly depends on the number of users in each class rather than their social behavior, contradicting some of the previous observations. We validate and extend the findings of our experimental study through a mathematical analysis.
---
paper_title: Real-time emergency response: improved management of real-time information during crisis situations

paper_content:

The decision-making process during crisis and emergency scenarios intertwines human intelligence with infocommunications. In such scenarios, the tasks of data acquisition, manipulation, and analysis involve a combination of cognitive processes and information and communications technologies, all of which are vital to effective situational awareness and response capability. To support such capabilities, we describe our real time emergency response (rtER) system, implemented with the intention of helping to manage the potential torrents of data that are available during a crisis, and that could easily overwhelm human cognitive capacity in the absence of technological mediation. Specifically, rtER seeks to address the research challenges surrounding the real-time collection of relevant data, especially live video, making this information rapidly available to a team of humans, and giving them the tools to manipulate, tag, and filter the most critical information of relevance to the situation.
---
paper_title: Collaborative navigation of visually impaired

paper_content:

A navigation system for visually impaired users can be much more efficient if it is based on collaboration among visually impaired persons and on utilising distributed knowledge about the environment in which the navigation task takes place. To design a new system of this kind, it is necessary to make a study of communication among visually impaired users while navigating in a given environment and on their regularly walked routes. A qualitative study was conducted to gain insight into the issue of communication among visually impaired persons while they are navigating in an unknown environment, and our hypotheses were validated by a quantitative study with a sample of 54 visually impaired respondents. A qualitative study was conducted with 20 visually impaired participants aimed at investigating regularly walked routes used by visually impaired persons. The results show that most visually impaired users already collaborate on navigation, and consider an environment description from other visually impaired persons to be adequate for safe and efficient navigation. It seems that the proposed collaborative navigation system is based on the natural behaviour of visually impaired persons. In addition, it has been shown that a network of regularly walked routes can significantly expand the urban area in which visually impaired persons are able to navigate safely and efficiently.
---
paper_title: Supporting Accessibility for Blind and Vision-impaired People With a Localized Gazetteer and Open Source Geotechnology

paper_content:

Disabled people, especially the blind and vision-impaired, are challenged by many transitory hazards in urban environments such as construction barricades, temporary fencing across walkways, and obstacles along curbs. These hazards present a problem for navigation, because they typically appear in an unplanned manner and are seldom included in databases used for accessibility mapping. Tactile maps are a traditional tool used by blind and vision-impaired people for navigation through urban environments, but such maps are not automatically updated with transitory hazards. As an alternative approach to static content on tactile maps, we use volunteered geographic information (VGI) and an Open Source system to provide updates of local infrastructure. These VGI updates, contributed via voice, text message, and e-mail, use geographic descriptions containing place names to describe changes to the local environment. After they have been contributed and stored in a database, we georeference VGI updates with a detailed gazetteer of local place names including buildings, administrative offices, landmarks, roadways, and dormitories. We publish maps and alerts showing transitory hazards, including location-based alerts delivered to mobile devices. Our system is built with several technologies including PHP, JavaScript, AJAX, Google Maps API, PostgreSQL, an Open Source database, and PostGIS, the PostgreSQL's spatial extension. This article provides insight into the integration of user-contributed geospatial information into a comprehensive system for use by the blind and vision-impaired, focusing on currently developed methods for geoparsing and georeferencing using a gazetteer.
---
paper_title: THE WALKING STRAIGHT MOBILE APPLICATION: HELPING THE VISUALLY IMPAIRED AVOID VEERING

paper_content:

The visually impaired community still faces many challenges with safely navigating their environment. They rely heavily on speechbased GPS in addition to their usual guiding help. However, GPSbased systems do not help with veering issues, which affect the ability of the visually impaired to maintain a straight path. Some research systems provide feedback intended to correct veering, but these tend to employ bulky, custom hardware. In response, we implemented our “Walking Straight” application on an existing consumer device, taking advantage of the built-in sensors on smartphones. First, we investigated whether a continuous or discrete form of non-speech audio feedback was more effective in keeping participants on a straight path. The most effective form was then tested with nine blind participants. The promising results demonstrate that Walking Straight significantly reduced the participants’ deviationfrom astraight pathascompared totheir usual behaviour, e.g., with a guide dog or cane, without affecting their pace.
---
paper_title: Navigational 3D audio-based game-training towards rich auditory spatial representation of the environment

paper_content:

As the number of people suffering from visual impairments continuously increases, there is strong need for efficient sensory substitution devices, that can support creating a rich mental spatial depiction of the environment. The use of the auditory sense has proved to be an effective approach towards creating a method of interaction with the elements of the surrounding space in a way which resembles the natural 3D visual representation of normal sighted people. Training is an essential component in the process of employing an auditory-based visual substitution device for blind people, as it helps them to learn and become proficient to process and decode the audio information and convert it into spatial mental representation. Taking into account the well-known advantages of game based learning, we propose a new method of training, consisting in a navigational 3D audio-based game. In this exploratory, goal-directed application, the player has to perform route-navigational tasks under different conditions, with the purpose of training and testing their orientation and mobility skills, relying exclusively on the perception of 3D audio cues. Experimental results showed that this game-based learning strategy leads to substantial improvements and can be a starting point for developing more enhanced sound-based navigational applications. The ludic-oriented, motivational training approach achieved straightforward immersion and concentration on the cognitive depiction of the environment, ensuring behavioral gains in the sound-directed spatial orientation.
---
paper_title: THE DESIGN OF AN AUDIO FILM FOR THE VISUALLY IMPAIRED

paper_content:

1. ABSTRACT Nowadays, Audio Description is used to enable visually impaired people to access films. However, it presents an important limitation, which consists in the need of the visually impaired audiences to rely on a describer, not being able to access the work directly. The aim of this project was to design a format of sonic art called audio film that eliminates the need of visual elements and of a describer, by providing information solely through sound, sound processing and spatialization, and which might be considered as an alternative to Audio Description. In order to explore the viability of this format an example has been designed based on Roald Dahl’s Lamb to the Slaughter (1954) using a 6.1 surround sound configuration. Through the design of this example it could be noticed that this format can successfully convey a story without the need of either visual elements or of a narrator.
---
paper_title: Definition and Synergies of Cognitive Infocommunications

paper_content:

In this paper, we provide the finalized definition of Cognitive Infocommunications (CogInfoCom). Following the definition, we briefly describe the scope and goals of CogInfoCom, and discuss the common interests between CogInfoCom and the various research disciplines which contribute to this new field in a synergistic way.
---
paper_title: Sensory dominance in combinations of audio, visual and haptic stimuli

paper_content:

Participants presented with auditory, visual, or bi-sensory audio–visual stimuli in a speeded discrimination task, fail to respond to the auditory component of the bi-sensory trials significantly more often than they fail to respond to the visual component—a ‘visual dominance’ effect. The current study investigated further the sensory dominance phenomenon in all combinations of auditory, visual and haptic stimuli. We found a similar visual dominance effect also in bi-sensory trials of combined haptic–visual stimuli, but no bias towards either sensory modality in bi-sensory trials of haptic–auditory stimuli. When presented with tri-sensory trials of combined auditory–visual–haptic stimuli, participants made more errors of responding only to two corresponding sensory signals than errors of responding only to a single sensory modality, however, there were no biases towards either sensory modality (or sensory pairs) in the distribution of both types of errors (i.e. responding only to a single stimulus or to pairs of stimuli). These results suggest that while vision can dominate both the auditory and the haptic sensory modalities, it is limited to bi-sensory combinations in which the visual signal is combined with another single stimulus. However, in a tri-sensory combination when a visual signal is presented simultaneously with both the auditory and the haptic signals, the probability of missing two signals is much smaller than of missing only one signal and therefore the visual dominance disappears.
---
paper_title: Visual Touch in Virtual Environments: An Exploratory Study of Presence, Multimodal Interfaces, and Cross-Modal Sensory Illusions

paper_content:

How do users generate an illusion of presence in a rich and consistent virtual environment from an impoverished, incomplete, and often inconsistent set of sensory cues? We conducted an experiment to explore how multimodal perceptual cues are integrated into a coherent experience of virtual objects and spaces. Specifically, we explored whether inter-modal integration contributes to generating the illusion of presence in virtual environments. ::: ::: To discover whether intermodal integration might play a role in presence, we looked for evidence of intermodal integration in the form of cross-modal interactions---perceptual illusions in which users use sensory cues in one modality to “fill in” the “missing” components of perceptual experience. One form of cross-modal interaction, a cross-modal transfer, is defined as a form of synesthesia, that is, a perceptual illusion in which stimulation to a sensory modality connected to the interface (such as the visual modality) is accompanied by perceived stimulation to an unconnected sensory modality that receives no apparent stimulation from the virtual environment (such as the haptic modality). Users of our experimental virtual environment who manipulated the visual analog of a physical force, a virtual spring, reported haptic sensations of “physical resistance”, even though the interface included no haptic displays. A path model of the data suggested that this cross-modal illusion was correlated with and dependent upon the sensation of spatial and sensory presence. ::: ::: We conclude that this is evidence that presence may derive from the process of multi-modal integration and, therefore, may be associated with other illusions, such as cross-modal transfers, that result from the process of creating a coherent mental model of the space. Finally, we suggest that this perceptual phenomenon might be used to improve user experiences with multimodal interfaces, specifically by supporting limited sensory displays (such as haptic displays) with appropriate synesthetic stimulation to other sensory modalities (such as visual and auditory analogs of haptic forces).
---
paper_title: The Spiral Discovery Method: An Interpretable Tuning Model for CogInfoCom Channels

paper_content:

Cognitive Infocommunications (CogInfoCom) messages that are used to carry information on the state of the same high-level concept can be regarded as belonging to a CogInfoCom channel. Such channels can be generated using any kind of parametric model. By changing the values of the parameters, it is possible to arrive at a large variety of CogInfoCom messages, a subset of which can belong to a CogInfoCom channel -provided they are perceptually well-suited to the purpose of conveying information on the same highlevel concept. Thus, for any CogInfoCom channel, we may speak of a parameter space and a perceptual space that is created by the totality of messages in the CogInfoCom channel. In this paper, we argue that in general, the relationship between the parameter space and the perceptual space is highly non-linear. For this reason, it is extremely difficult for the designer of a CogInfoCom channel to tune the parameters in such a way that the resulting CogInfoCom messages are perceptually continuous, and suitable to carry information on a single high-level concept. To address this problem, we propose a cognitive artifact that uses a rank concept available in tensor algebra to provide the designer of CogInfoCom channels with practical tradeoffs between complexity and interpretability. We refer to the artifact as the Spiral Discovery Method (SDM).
---
paper_title: Oversketching and associated audio-based feedback channels for a virtual sketching application

paper_content:

With the growing relevance of human interaction with infocommunications in general and augmented/virtual environments in particular, it is becoming increasingly important to provide users with a “virtual physics” capable of emulating the richness and subtle informativeness of multimodal feedback in the physical world. In this paper, we describe an extension to an existing immersive virtual sketching application which consists of an oversketching interaction mode, and a set of associated audio-based feedback signals. The implemented oversketching functionality allows users make incremental corrections to the drawing, helping them to emphasize certain aspects while deemphasizing others. The audio-based feedback signals, in turn, support a better understanding of the progress of oversketching, when the goal is to transition from curved to straight line segments, or vice versa.
---
paper_title: Visual cues and virtual touch: Role of visual stimuli and intersensory integration in cross-modal haptic illusions and the sense of presence.

paper_content:

Summary Intermodal integration (sometimes referred to as intersensory integration) may be a key psychological mechanism contributing to a sense of presence in virtual environments. Sensorimotor processes associated with multimodal integration may integrate perceptual cues and motor actions into a coherent experience and relatively consistent model of objects and spaces. When the cues come from virtual environments, intermodal integration may generate a sense of presence in a coherent virtual world. Because stimuli from virtual environments frequently fail to provide coherent and consistent cues, evidence of the role of intermodal integration might be found in intersensory illusions, the results of the user’s attempt to integrate an inconsistent environment. Secondly, if intermodal integration plays a role in the generation of presence, then intersensory illusions should be correlated with the illusion of presence in a coherent virtual world.
---

Title: A survey of assistive technologies and applications for blind users on mobile platforms: a review and foundation for research

Section 1: Introduction
Description 1: Introduce the prevalence of state-of-the-art technology among visually impaired users, discuss the challenges in designing user interfaces, and present the focus and structure of the paper.

Section 2: Overview of auditory and haptic feedback methods
Description 2: Describe basic auditory and haptic feedback techniques, including historical context, theoretical underpinnings, and various representations used in these domains.

Section 3: Systems in assistive engineering based on tactile solutions
Description 3: Discuss assistive engineering systems that rely on tactile solutions, covering specific examples, historical evolution, and current trends.

Section 4: Systems in assistive engineering based on auditory solutions
Description 4: Examine assistive engineering systems that utilize auditory solutions, detailing early systems, technological advancements, and significant examples in the field.

Section 5: Systems in assistive engineering based on auditory and tactile solutions
Description 5: Explore systems that combine auditory and tactile modalities, focusing on recent developments, key examples, and the integration of multimodal feedback.

Section 6: Summary of key observations
Description 6: Summarize the key findings from previous sections, emphasizing the range of feedback solutions, their usability, and effectiveness.

Section 7: Generic capabilities of mobile computing platforms
Description 7: Highlight the general-purpose computing, advanced sensory capabilities, and data integration features of mobile platforms that support assistive technologies.

Section 8: State-of-the-art applications for mobile platforms
Description 8: Provide an overview of current applications available on Android and iOS platforms to assist visually impaired users in performing everyday tasks.

Section 9: Applications for Android
Description 9: Detail various existing assistive solutions specifically developed for Android, including text-to-speech, typing aids, and navigation support.

Section 10: Applications for iOS
Description 10: Examine assistive applications available on iOS, covering similar categories as Android, and highlighting unique features and solutions.

Section 11: Towards a research agenda for mobile assistive technology for visually impaired users
Description 11: Propose a research agenda for the further development of mobile assistive technology, identifying key areas and issues that need to be addressed.

Section 12: Summary
Description 12: Conclude the paper with a brief summary of discussed topics, emphasizing the significance and potential of assistive technologies on mobile platforms.