« Retro Information Management Continued... | Main | Call for Head First Design Meditations »

The effect of sound on users

In Hollywood, some say that in a movie the visuals tell you what you're seeing, while the soundtrack tells you how to feel about it. I used to teach new media and interaction design at UCLA Extension's Entertainment Studies Department, and in the film scoring classes (which I didn't teach) students often started out with a classic exercise: Watch the shower scene in the movie Psycho, without the sound. Without that Ee-Ee-Ee-Knife-Slashing audio it just doesn't feel like the scene that scared me into being more of a bath than shower girl.

And audio has even been shown to affect the audience perception of the quality of a presentation more than the visuals. I don't have a link handy, but there's a study that showed that the quality of the audio causes people to change their evaluation of the quality of the visuals, but that it doesn't work in reverse. In other words, given a film clip or animation, raising the quality of the audio caused people to say, "Hey, the visuals are good!", and when the audio quality sucked, the audience rated the visuals as worse even when the visual quality was the same in both the bad and good audio versions. But... changing the quality of the visuals did NOT have that same power. If the sound sucked but the visuals were fabulous, it didn't cause viewers to say, "Wow... good sound too."

So, sound has the power to raise (or lower) audience perception of visuals, but visual doesn't have the power to change how the audience perceives the audio.

But sound is usually the second-class citizen in the non-professional multimedia world, while visuals take center stage in everything. (Unless the photographer, videographer, or animator happens to also be a musician). Everyone has a digital camera now--that makes everyone at least an amateur photographer. And everyone has some kind of digital editing software like Photoshop Elements that brings high-end photo manipulation to the home user. And why stop with still pictures when digital camcorders are so cheap now? With editing software like iMovie shipping with every Mac, anyone can become a video editor now.

But while the emphasis on developing visual sensibilities and skills has continued to build (almost everyone with a digital camera today knows design fundamentals like the "rule of thirds" or how not to cut people off at their joints), what about the poor stepchild audio? Sure we could all listen to music, but where were the tools that would bring music creation to non-musicians in the way that the visual tools (and books and references) brought graphic and photographic editing to non-photographers and non-graphic designers?

I gave a presention on this discrepancy ten years ago to a new media group in Los Angeles, and while I stood there ranting about how nobody had made the Photoshop equivalent of audio (or even the Kid Pix equivalent) one guy in the audience, Kevin Klinger, started thinking about it. He went home, thought some more, and decided to start a company to do just that. I was at Mac World many years later, and there was the Smart Sound booth, and Kevin said, "Hey, thanks for the idea." I couldn't believe it. He actually did it.

SmartSound's focus is on giving people a way to create sound tracks for videos, without being a musician. And the "smart" part comes from the cleverly-engineered ways in which the software fits itself to the movie. Because another problem with most home movies is that the music often doesn't finish, it often just fades out (or worse, cuts off), because the music was too long for the video. SmartSound is an amazing program and goes way past what I had in mind when I gave the talk. But it's main focus is on making sound tracks for, say, corporate videos, while I was still waiting for that low-end, home-use music creation program for non (or very weak) musicians.

Something that would encourage "regular people" to start developing music and sound sensitivities in the way that we've developed our design and visual awareness and creative skills. In other words, when will "the rest of us" get to work on the "A" in "AV"?

Of course, Apple's done that now with the phenomenal Garage Band! A tool that threatens to turn people who have no business making music into musicians. (When I say "no business making music" I'm referring to the notion some have that there must be clear boundaries between those who create art and those who appreciate it. I think that's bullshit... we're all born creative even if many of us will never EVER hope to be professionals. Heck, we all spent our first nine months listening to a 24-7 dance beat.)

I blogged earlier about the way that teens and twenty-somethings today often treat turntabilism the way forty-year olds used to treat guitars. They treat it like an instrument. Something to use for creating music. So the audio world is definitely changing, and Garage Band is, I think, the single most important step in bringing music into the world formerly reserved only for graphics programs.

The four of us care a lot about sound here, because sound (and especially music) has a powerful effect on learning, in two ways. First, it manipulates emotions, and emotions play a huge role in memory formation. Second, the more senses you can involve in learning, the greater your chances of retaining and later recalling the knowledge. Think about it--if you file something in two places instead of one, you've doubled your chances of getting it back again, and when you remember something as, say, a sound, image, and text--you've just given yourself three potentially different ways in which that info is stored in the brain. Triple the chances of getting it out when you need it.

But... we're still doing books and books don't give us a way to do that. We are planning some multimedia formats for the fairly near future, but for now, we're doing what we can with things we expect you--the reader--to do to fully realize the power of audio. When you come across a limerick, poem, or song lyric in the book, for the love of god PLEASE say it out loud! Don't just read it silently (although even then, you often will hear yourself saying it with a rhythm, and even that helps).

Anyway, I'll have a lot more to say about audio in the future, but for now, here's my really BAD version of the film school exercise for those who aren't musicians or sound designers. I have three videos, all with the exact same little scene, but using three different songs. (The songs are just simple sequences I put together in GarageBand, which means no copyright issues : ) [Disclaimer: I'm not a musician (which will be painfully obvious if you play these). The point is the effect the music has, regardless of what you're looking at.]

Your assignment is to play the videos, in order, and with each one, write down the following info before moving on to the next video:

1) What kind of movie is this? Speculate on what the movie might be about, based on the feeling you get from the audio. Do NOT use your brain to try to think something up, just go with what pops into your head based on the feeling.

2) Speculate on what might be happening in this character's life. Make up what the next scene might be, based on the feeling you have from the sound.

Once you've done that with all three, play one of them without any sound at all and see how you feel about it now. (You can experiment by playing one of the others, and see if the last music you heard affects how you view it when there's no music at all.)

Finally, pick an emotion/feeling you want to evoke, and find some music you have that you think will create that feeling. Play that while the video is playing, and test it on someone else and see if your victim gets the feeling.

Here they are, and remember... watch them in this order (note: they are Quicktime movies, about 1.5 Mbs, sorry dial-up users : (

Version one movie

Version two movie

Version three movie

So, how are you using the power of sound in your teaching and learning and communicating?

Posted by Kathy on January 15, 2005 | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference The effect of sound on users:


This looks related to what you're talking about for non-musicians to help create (through remixing to make something new) music:


Posted by: Zeus | Jan 15, 2005 6:02:31 PM

I bought Design Patterns yesterday because I was looking for a book to prepare me for studying JAVA next summer. I found this website/blog tonight.

I LOVE the book. I'm a visual thinker so I really appreciate the opportunity to learn more about object-oriented programming/design without plowing through pages of never-ending words.

I'm a mid-career school psychologist who works with middle and high school students. I have some background with computer music technology (as a hobby).I recently completed my first programming class (VB.Net) and an introductory game design/development class.

The following is the link you might have seen about the use of sound with video.


I came across the link to CAST and the on-line interactive book, "Teaching Every Student in the Digital Age" while doing research for a paper last year.

The title of my paper? "Thinking, Learning, and Communicating with Multimedia: Views from a School Psychologist".

I wrote the paper because most of the students I work with at the middle school and high school find themselves failing or at risk of failing one or more of their classes each semester. Most of them are strong visual learners who have talent in music, art, and/or computers. These students do not have many opportunities to shine at school.

Students who have learning and/or attention difficulties often read slowly and inefficiently. They have auditory processing deficits that make it difficult for them to learn through lectures and discussions, yet they find themselves in classrooms where their highly-verbal teachers prefer to teach through lectures, discussions, and volumes of reading assignments.

Why is this so?

Sadly, SmartBoards, wireless mobile computer labs, PDA sets, Inspiration software, are available at my schools, but rarely used. As a consequence, students with learning differences are brain dead 30 minutes into a 90-minute class.

You should get into the high school text book business!

(Sorry if my message is a bit disjointed- it is past my early bedtime.)

Posted by: Lynn Marentette | Jan 15, 2005 9:49:21 PM

When building a home theater a common rule of thumb is that 60%-70% of the experience is audio, and so surprisingly your A/V budget should be split near the same (or at least 50%/50%) between your video and audio components.

As Kathy mentioned, in home theater tests users that watch a video with a good audio will actually report that the video quaility was better than when watched with the same video and poor audio.

Posted by: Eric | Jan 16, 2005 10:15:43 PM

The comments to this entry are closed.