Font Size
Search Results for ""
Top Stories

Published On October 3, 2015


Sensory Substitution

Research on synesthesia has led to devices that blur the lines between the senses, and may offer new hope for the blind.

FOR ARTIST CAROL STEEN, the elevator bell in her Manhattan apartment building elicits a burst of bright magenta in her mind, and she frequently paints to music, putting the colors she “hears” on the canvas. And for Steen, letters and numbers present themselves in colors. To her, the letter A appears a beautiful shade of pink, while she describes X as “a mousy gray.”

Steen has a form of synesthesia, a condition in which the senses cross their typical boundaries. The word translates from its Greek origins roughly as “joined sensation,” and it can take dozens of forms: sounds that can be smelled, visions that produce taste, touch that elicits colors. But while synesthesia was long dismissed as a product of mere mental associations or memories, a growing body of science suggests that it’s a genuine perceptual phenomenon. Cognitive neuroscientist Jamie Ward, of the University of Sussex in the United Kingdom, studies synesthesia and says most once-skeptical colleagues have stopped asking him if it’s for real. “People are a lot more open-minded about it today,” says Ward, though he notes that some still ask: “If it’s real, so what?”

One answer may be that synesthesia serves as a model for sensory substitution devices (SSDs), an emerging category of aids for people who have lost one of their senses. Most SSDs help the blind, though at least one device is now being developed for the hearing-impaired. The notion is that because the brain is capable of synesthesia, one sense may be able to be experienced through input normally associated with another sense. So if the eyes no longer provide signals to the brain, why not stimulate the visual cortex with input intended for sound or touch? “Our goal is to create a form of synesthesia that people can learn,” says experimental psychologist Michael Proulx, who leads the Crossmodal Cognition Laboratory at the University of Bath, also in the United Kingdom, and is studying the benefits of SSDs for the blind.

Various kinds of SSDs take different approaches, but all are based on the idea that someone who is deprived of one sense may be able to learn to translate patterns of sensations that are received into something very much like visual images. How that happens is complicated and a little mysterious, but clinical studies by Proulx and others have found that blind people trained to use SSDs can perform basic visual functions, such as identifying everyday objects and navigating around obstacles in hallways. One SSD, the BrainPort, translates visual images into vibrations felt on the tongue and has been approved by the U.S. Food and Drug Administration. Another SSD, called The vOICe (the spelling is meant to suggest “Oh, I see”), converts images to sound. It is already available and has been studied by neuroscientists in the United Kingdom and Israel. One longtime user, Pat Fletcher, who lost her vision in an industrial accident, is unequivocal about what occurs when she uses The vOICe. “I can see,” says Fletcher. “How, I don’t know, but I can see.”

Most researchers who study SSDs tend to be more circumspect when describing the benefits, although they do believe that SSDs create visual-like experiences in the brains of the blind. They argue that SSDs could offer a less expensive, noninvasive alternative to current and future options for the blind, such as prosthetic eyes and retinal implants.

Neuroscientist Paul Bach-y-Rita, who created the apparatus that eventually became the BrainPort, was fond of saying, “We see with the brain, not with the eyes.” But is it folly to believe that inducing synesthesia could allow the blind to see with their skin or ears?

SIR FRANCIS GALTON, perhaps best known for his study of eugenics, is believed to have offered one of the earliest scientific descriptions of synesthesia in a chapter in his 1883 book, Inquiries into Human Faculty and Its Development. Galton’s research inspired other scientists of the era to study synesthesia, though interest then waned during much of the 20th century. Neurologist Richard E. Cytowic is credited with reawakening scientific curiosity about people who have these unusual experiences, notably in his 1989 treatise, Synesthesia: A Union of the Senses. (He later published a popular book, The Man Who Tasted Shapes.)

About 4% of adults have synesthesia, according to a 2006 study, though earlier estimates suggest the condition is much rarer. Synesthetes’ sensory and perceptual associations seem to remain remarkably stable over time. For instance, when self-reported synesthetes are asked to specify the colors evoked by various words and characters, their associations remain 70% to 90% consistent when retested months or even years later. In contrast, non-synesthetes come up with the same associations only 20% to 38% of the time.

Some imaging studies have revealed distinctive functional and structural differences in the brains of synesthetes, but other work has found nothing to distinguish them neurologically from non-synesthetes. Among the many theories about what might cause synesthesia is one suggesting that all infants are born with an overabundance of neuronal linkages among different sensory regions in the brain; in most people, the majority of these connections are lost, or “pruned,” during development, but more of them are retained by synesthetes.

Synesthesia-like experiences have been described by people who’ve taken hallucinogenic drugs, and there are reports of people developing synesthesia after suffering neurological damage. Yet most scientists who study the phenomenon have concluded that you have it from early childhood or you don’t. Now Ward, Proulx and others are proposing that a form of synesthesia can be induced, or artificially acquired, as a way to restore a lost sense. But well before they reached that conclusion, Bach-y-Rita had begun experimenting with the radical theory that blind people can retrain their brains to see.

IN A LETTER TO NATURE magazine in 1969, Bach-y-Rita and several colleagues described what many consider the first sensory substitution device, which he called a tactile vision substitution system (TVSS). Bach-y-Rita had spent much of the 1960s studying the physiology of eye movement at the Smith-Kettlewell Institute of Visual Sciences in San Francisco, and he later became known for helping promote the idea of neuroplasticity—the now widely accepted theory that the brain maintains the ability to adapt, rewire itself and form new connections between neurons throughout a person’s life span.

Bach-y-Rita knew of earlier work using vibrations on the skin to form letters, an attempt to help the blind read with their skin. So, he thought, why not use vibrations to create images in the brain? His TVSS consisted of a dentist’s chair equipped with 400 tiny Teflon “tips,” or stimulators, arrayed 12 millimeters apart on the seat back. Adjacent to the chair was a movable television camera. Images captured by the camera were transduced into specific patterns of vibration for the stimulators. Bach-y-Rita and his colleagues first trained blind subjects to recognize those patterns vibrating on their backs as types of lines, and then as circles, triangles and squares. Then they moved on to everyday objects, such as a telephone, chair and toy horse.

At the University of Wisconsin–Madison, Bach-y-Rita refined his apparatus to make it mobile. He planted a tiny video camera into sunglasses and replaced the back stimulator with a small pad-like device that is placed on the tongue, with a wire running up to the sunglasses. The tongue turns out to be surprisingly well suited for sensory stimulation. It’s lined with receptors, making it very sensitive, but the tongue is also coated with an electrolytic solution—saliva—that promotes good electrical contact. (Bach-y-Rita had considered using the fingertip, but the skin there is less sensitive than the tongue, and thus would have required a painful level of electrical current for adequate stimulation.)

Bach-y-Rita died in 2006, but the company he started, Wicab, lives on. Now known as the BrainPort, his tongue-stimulating device has been approved for use in the European Union and the United States. Optometrist Amy Nau first heard about the BrainPort at an ophthalmology conference in 2009 and found the idea of tongue stimulation as an aid for the blind to be “completely kooky.” She nonetheless ended up studying the BrainPort with more than 100 blind subjects at the University of Pittsburgh, where she founded a sensory substitution laboratory. Nau and her colleagues have found that, with adequate training, most BrainPort users are able to identify objects such as a plastic banana or coffee mug and navigate obstacle courses in long corridors while picking out signs for EXIT or DANGER.

WHILE BACH-Y-RITA WAS developing the BrainPort, a Dutch physicist named Peter Meijer was quietly working on his own SSD for the blind. Meijer first had the idea of converting images into sound as a university student in the early 1980s. He worked on the project in his spare time before publishing an article describing his system in a biomedical engineering journal in 1992.

The vOICe converts images into sound, or “soundscapes,” as Meijer prefers, with a three-part system: a tiny video camera embedded into sunglasses; a small laptop computer (carried in a backpack) or a smartphone loaded with software that runs an image-to-sound algorithm; and headphones. The software scans from left to right, and converts each pixel (or point in the digital image) into a distinct sound piped into the headphones. Bright pixels are translated into loud sounds, while darker pixels sound softer. The position of a pixel in the view dictates its pitch, high or low, and its timing in the stereo mix. Meijer currently gives away the software that runs The vOICe for free on the Internet, though he may eventually charge a fee for a more advanced version; the user supplies the other components. (Sunglasses with embedded cameras are available online for as little as $30.)

At Hebrew University of Jerusalem, neuroscientist Amir Amedi and his colleagues trained eight congenitally blind people to use The vOICe, then administered a standard eye-chart vision exam. More than half of the study subjects achieved a visual acuity of 20/360—which pushed them above “blind” to “low vision” on the World Health Organization’s classification system. Amedi and his team are developing an SSD of their own, called EyeMusic, which converts the shape, location and color of objects into musical notes.

Proulx at the University of Bath has studied both people who are blind and those wearing blindfolds using The vOICe and says the two groups perform about equally well in testing. In one test, some blindfolded users achieved a visual acuity of 20/400. “That’s still legally blind,” says Proulx, “but essentially that’s what I can see without my contact lenses.” Some people could read letters on a computer screen that were only five pixels by five pixels—the smallest size the computer could generate. Proulx and a colleague are developing a training program to make it less intimidating to learn to use The vOICe—an experience that is currently, according to Proulx, “like being dropped into Moscow and told to learn Russian.”

After her accident, Pat Fletcher learned about The vOICe while surfing the Internet—using software that converts text to spoken word—for a color-identifying program to help with her clothes. When she downloaded the software and tried it, “at first it was very annoying and made no sense at all,” she says. But she gradually began to associate the electronic squawks and bleeps with the shapes and textures of objects she held in her hand. One day, while wearing the apparatus, she stepped out of her bedroom and was shocked to realize she could see the blinds hanging from the window at the end of the hallway.

From that day on, Fletcher says, her acuity steadily sharpened. “I see my world around me,” she says. Though she struggles somewhat with depth perception, she can identify the furniture in her home, do household chores and mow the lawn. Fletcher describes the images she sees as being varying shades of gray, though she says that recently colors have begun to appear.

BUT ARE THE USERS OF BrainPort and other SSDs truly seeing? A better first question might be whether it’s possible for the brain’s visual cortex to accept and process nerve signals from another sensory channel, such as the ears or skin. “Absolutely” it’s possible, says neuroscientist Stephen Lomber, who heads the Cerebral Systems Lab at the University of Western Ontario. Although not involved in research on SSDs, Lomber has studied the visual cortex and currently focuses on how the brain responds when hearing is restored (as in deaf people who receive cochlear implants). “We’ve found that when the brain is deprived of input, whether visual or auditory, those regions that would normally process that input don’t seem to have any trouble at all accepting a different type of sensory input,” says Lomber.

During human development, according to Lomber, specific regions of the brain appear to become predisposed to process a specific form of information. For instance, one part of the auditory cortex normally processes sound in motion, such as the whoosh of a car racing past. According to his research, when someone loses the ability to hear, that same area in the auditory cortex switches to processing images of moving objects. The task performed by that part of the brain—processing the perception of motion—is what matters, rather than whether the input comes from the ears or the eyes.

Several studies using functional magnetic resonance imaging (fMRI) show blood flow increasing in the visual cortexes of blind people using SSDs. Yet that evidence doesn’t prove their brains are converting sound into vision, Lomber says. A better experiment, he says, would temporarily deactivate the visual cortex of someone using The vOICe or BrainPort to see whether that causes the tools to stop working.

Fletcher let researchers do just that to her brain in a 2008 experiment led by neuroscientist Alvaro Pascual-Leone of Beth Israel Deaconess Medical Center in Boston. After Fletcher, using The vOICe, scored well on tests to identify objects and letters, Pascual-Leone used transcranial magnetic stimulation (TMS) to disrupt activity in the visual-processing region of her brain. Tested again, her discernment plummeted. (The effects of TMS took longer than expected to wear off, Fletcher recalls. “That scared the dickens out of me,” she says.)

Pascual-Leone’s study suggests that the activity in the visual cortex of vOICe users found by the fMRI is not a meaningless artifact or side effect, but instead is probably “functionally relevant,” says Meijer, The vOICe inventor, because blocking normal visual cortex activity prevented Fletcher from interpreting the soundscapes. But he’s not sure whether that means Fletcher and other vOICe users can truly see. “At a scientific level, we do not know,” says Meijer.

But Proulx believes that Fletcher can see. He feels that people who have lost their vision later in life are able to combine The vOICe’s soundscapes with their visual memories to form accurate images of objects and their environments. That may not happen, however, for those who are born blind or who lost their sight early in life. “They can have seeing-like behavior, but not have the same qualitative experience of vision that someone like Pat can have,” Proulx says.

The field of sensory substitution is barely out of its infancy, though it may be poised to grow up fast. “It might seem kind of wacky and fringe, but it’s getting traction in the scientific world,” says Nau, who now works in an optometry practice in Boston and continues to study visual SSDs. She believes the models currently available are best thought of as ambulatory aids—tools to help the blind navigate daily life. But a growing interest may help to refine the technology; both Google and the U.S. Department of Defense have contributed to BrainPort.

Researchers who have trained the blind to use SSDs stress that it’s crucial to manage expectations. Sight doesn’t return the instant you slip on one of these apparatuses, and a great deal of hard work is necessary to reach the proficiency of a Pat Fletcher. And Fletcher reports there’s something else about her senses that has changed since she began using The vOICe: She has developed synesthesia. Certain images now trigger her sensation of touch; staring at a Christmas tree, for instance, makes her hands tingle as if she were brushing them against evergreen needles.

Fletcher knows that skeptics question whether she can truly see, but that doesn’t change her conviction that sensory substitution has changed her life forever. “There’s nothing there until I put on The vOICe,” says Fletcher. “Then the world is full and rich.” 


Sensory Substitution: Closing the Gap Between Basic Research and Widespread Practical Visual Rehabilitation,” by Shachar Maidenbaum et al., Neuroscience and Biobehavioral Reviews, April 2014. An overview of the potential of SSDs written by researchers at Hebrew University in Jerusalem who are designing several of their own.

The Synesthesia Battery ( This online test for synesthesia was created by the lab of neuroscientist David Eagleman, which is also working on an SSD vest.

AUDIO “Oh, I See”: Pat Fletcher and The vOICe ( Long-time The vOICe user Pat Fletcher describes her experience with the device, with commentary from creator Peter Meijer and scientist Michael Proulx.