Edition: U.S. / Global

Controls Rely on the Twitch of a Muscle, Not the Twitch of a Mouse

Published: February 8, 2001

COMPUTERS, cell phones and other gadgets continue to shrink, but the means of interacting with them, like keyboards and keypads, are either too large and cumbersome or too small for the human hand. That is the belief of a group of NASA scientists who have been working on more direct ways to control machines, starting with jets and spacecraft.

Speech recognition is one alternative, but it is not ideal for cockpits or other noisy places. So the scientists are attempting to show how gestures can link humans more directly to the devices they control. They recently demonstrated that electronic signals picked up from human muscles could be used to land a jetliner in a simulation. These kinds of controls could someday be used in things like video games as well.

"What's driving this project is the belief that we've become increasingly disassociated from the actual state of our physical environment," said Dr. Charles Jorgensen, the lead scientist on the project. "We're looking at whether we can find a way of more tightly integrating ourselves with the machines around us." Dr. Jorgensen is head of the neuroengineering laboratory at the space agency's Ames Research Center in Moffett Field, Calif.

To demonstrate gesture-based, or bioelectric, control, Dr. Jorgensen and his team fashioned a five-inch-wide armband from exercise tights, and sewed on eight metallic buttons to function as dry electrodes, receiving electrical signals from muscles in the pilot's arm. Wires carry signals from the electrodes to a computer.

The NASA team has tested the armband in about 100 trial runs in the past six months, landing a 757 passenger jet simulator at a virtual version of San Francisco Airport. The simulation software ran on two personal computers with 17-inch computer screens; one displayed a simulation of a cockpit control panel, while the other, a flight simulator, showed the position of the jet from multiple angles.

Before the trial runs, the scientists designed pattern recognition software, a variation of neural net software, which can be programmed to identify repeated physical patterns.

Dr. Kevin Wheeler, a scientist who worked on the project, acted as the pilot during the trials. Once the software was able to recognize the electrical signals from the arm as movements, Dr. Wheeler sat in front of the simulated cockpit at the control terminal, closing his fist as if he were closing it around an actual control stick, then moving his arm as if he were shifting the control stick into banking and pitching motions. As he moved, the electrical signals in his arm were picked up by the button electrodes on the armband and transmitted to a computer a few feet away through a wire. Pattern recognition software then digitized the signals and recognized the patterns as gestures; signals keyed to each gesture were then transmitted over a network to the flight simulator on the other terminal.

Dr. Jorgensen said controls based on the electrical signals from muscle movements would allow pilots to connect more directly and intuitively with aircraft and spaceships, without joysticks or other controls. Flying a plane with gestures instead of manual controls, he said, would also cut costs.

"A lot of the cost differentials in the building of jets are made up of having a separate compartment in them specifically designed to wrap around a pilot in a particular location in the aircraft," he said. "There's big design savings by taking away that need."

Eliminating conventional control devices in a space vehicle would have several benefits, Dr. Jorgensen said. It would certainly help an astronaut wearing a spacesuit with bulky gloves, which would make it difficult to type and manipulate controls. An armband with electrodes that detected the astronaut's muscle movements inside the spacesuit and transmitted commands would let the astronaut carry out these functions more easily.

"If there's an accident and there's depressurization," Dr. Wheeler said, "you can't get out of a spacesuit. If the computers were still functional, you'd still have limited means to interact with them."

In the event that an astronaut sends out a robot on a space walk, the astronaut could use his own arm movements to control the robot, instead of pushing buttons or using other manual controls. The robot would mimic the astronaut's own motions.

Dr. Jorgensen said that NASA had applied for a patent on the pattern recognition method and bioelectric armband concept and that he hoped to see the first commercial applications for the technology within a couple of years. The first commercial products would probably be computer game controllers, which do not require the same level of precision as a virtual aircraft controller, he said. Virtual keyboards and mice would be the next logical step.

"We're envisioning something that looks like a wristwatch on your arm," Dr. Jorgensen said, "which would enable someone to type on a photocopy of a keyboard that could be thrown away." The virtual keyboard could also be a blank, flat surface, assuming that someone is a good typist and does not need to see the keys.

Further down the road, it may also be possible to use bioelectric signals to produce "silent speech." The scientists are exploring ways in which electrodes could transmit signals from muscle movements in the tongue as words were formed within the mouth — but not actually spoken — to a computer that could translate the signals into digitized speech.

"The idea would be to get away from having to hear people talking on their cell phones," Dr. Jorgensen said. "You could more or less sit there and communicate and have it be translated electronically."

For now, the main challenge is honing the system to recognize the subtle arm and wrist turns that pilots typically make during a flight. The group is also working on ways to make the software recognize and compensate for different arm sizes and for the different ways individuals move.

"We obviously have a ways to go yet," Dr. Jorgensen said, "and haven't licked all these things. The analogy would be with speech recognition software. It works pretty well out of the box, but you need to custom- train it for a while."