Eavesdropping Technique Reconstructs Sound from Moving Images Alone

When you speak, surrounding objects quake and quiver in response to your voice—you just can’t see their vibrations because they’re so minute.

But now, engineers at MIT led by Abe Davis have devised a way to image those tiny vibrations and translate them back into sound. By using a special high-frame-rate camera to film objects down to about one one-thousandth of a pixel at a time, they were able to recreate the original frequencies, in order, that caused those vibrations. What’s more, they could also discern human speech from behind soundproof glass using the same technique.

A new technique devised by a team at MIT makes sound visible in a way scientists had never conceived before.

Even more impressive, they modified their approach to use an everyday digital camera. The trick was a technique known as a rolling shutter, in which a scene is scanned rapidly from top to bottom rather than taken in all at once. Their reconstructed “Mary Had a Little Lamb” included frequencies up to six times higher than the camera’s actual frame rate of 60 frames per second.

Watch this video for a demonstration of the team's experiments.

Here’s Hal Hodson, writing for New Scientist:



Davis says that while a regular digital camera with a rolling shutter can’t recover speech, it can recover vocal qualities like gender. In many ways, their demonstration illustrates how sound leaves a literal imprint on our environment.