VR Gives Journalism a New Dimension

Virtual reality is entering the world of news, creating engaging, real-life reports.

I'm standing in a dark room. I hear rain falling outside. A man says, "It was raining. And I stood for a few minutes lost in the beauty of it. If only there could be something equivalent to rain falling inside. Then the whole of a room would take on shape and dimension."

Suddenly, the sound of the rain is coming from within the room.

As I gaze around, I see dim shapes of ordinary household items—a pot, a pan, a bowl. They mysteriously change color and then transform into a slow burst of iridescent light. I can almost feel the raindrops. The man says, "Why should this experience strike one as being beautiful? Cognition is beautiful. It's beautiful to know." The rain becomes a downpour, and the sound blends with beautiful, melancholy music.

It may sound like I'm dreaming (or hallucinating). But this is a description of what I experienced in "Notes on Blindness," a powerful virtual reality "experience" (the term for these 360-degree interactive movies), which accompanies a documentary on the writer and philosopher, John Hull, as he began to lose his sight.

A New Kind of VR

"Notes on Blindness" is just one example of how VR is taking off in a new direction. It's a turn from the more commonly known fictional VR worlds in gaming, which use 360-degree video and computer-generated graphics (CGI) to immerse you in, say, an ethereal landscape on a make-believe distant planet or a highly detailed reconstructed set of your favorite science-fiction movie.

That's not to say the technology used in "Notes on Blindness" is different from what's used to create entertainment VR: In fact, it's very much the same (360-degree video, CGI, and so on). The aim is not to escape the real world, though, but to feel more engaged with it. "Notes on Blindness" gives you a sense of what it feels like to be John Hull, as his eyesight diminishes and then vanishes altogether.

These types of VR experiences are beginning to appear more frequently. What if you could dive under the ice in Antarctica and swim next to a seal? Or see the devastation and hear the falling bombs in a city that's been ravaged by war, such as Aleppo, Syria? Or move alongside refugees fleeing their homes to avoid persecution? These are some examples of how journalists, creators of documentary movies, and other non-fiction storytellers are beginning to experiment with VR.

Some of the most ambitious endeavors in VR and 360-degree video have come from The New York Times, which, in 2015, sent Google Cardboard headsets to more than a million subscribers to use with their smartphones.

"That was really one of the watershed moments in VR in terms of exposing a broad group of people who probably have not seen anything in that medium," said Adam Sheppard, CEO and co-founder of 8ninths, a Seattle-based virtual- and mixed-reality studio. In late 2016, The New York Times also introduced a feature called the Daily 360, which posts a new 360-degree video and VR experience every day.

Immersive Storytelling

Not surprisingly, The New York Times believes immersive virtual journalistic experiences can make a huge impact. According to Marcelle Hopkins, co-director of virtual reality and deputy director of video at The New York Times, "We see virtual reality, as well as 360 video, AR, MR, and whatever comes next, as part of the same spectrum, which is immersive platforms. We see that as part of the future of how people consume media, including journalism."

For Hopkins, as well as many others in this emerging field, it's the immersive quality that holds the biggest draw. But for journalism, documentaries, news, and other non-fiction genres, VR is relatively new territory. "It's a very young medium," said Hopkins, "and we're just learning how to use it. As we're telling stories in this way, we're learning a lot each time we do it."

"Cutting yourself off from the rest of the world in a headset is a very immersive experience," explained Jessica Lauretti (below), vice president of RYOT Studio Oath's creative studio, which creates VR content. She noted that VR can be powerful for storytellers, since it forces the viewer to be engaged. "You can't see anything else. So it does have this ability to transport you to another place, another country, another time."

Jessica Lauretti, VP of RYOT Studio Oath's creative studio

In 2016, The Guardian used this isolating quality of VR to great effect in "6 x 9," which aimed to replicate the experience living in solitary confinement in prison.

"We're always looking at new ways to express our journalism at The Guardian and find ways to innovate," said Francesca Panetta, executive editor, virtual reality, Guardian News & Media. "Virtual reality was a form we had been thinking about and wanted to experiment with and simultaneously, editorially at The Guardian, we had been talking about solitary confinement. In '6 x 9' the two things came together: VR is a medium, which is all about space, and solitary confinement is too, albeit a small and very undesirable space.

"It also is a piece about psychology: the impact on the mind when you are in isolation. We also wanted to portray the possible effects of this, such as blurred vision, audio, and visual hallucinations. With all this considered, it felt obvious that '6 x 9' would be a good story for the form."

The Guardian's 6x9

Panetta also wanted to engage viewers in other ways, including interactive elements—although technically, she said, that was difficult to implement. "One scene has hot spots which you trigger by looking at them," said Panetta. "This sounds easy, but it wasn't."

Another consideration was time, said Panetta. "'6x9' is a piece about being in a space and having very little to do, with minimal interaction for days, months, years, even decades. We needed to consider how we could make a piece that wasn't deadly boring and [in] which people didn't take the headsets off halfway through."

Actually, "6 x 9" is the opposite of dull—it's riveting and produces a strong visceral reaction. I had the opportunity to try this experience using an Oculus Rift. During the roughly 10-minute piece, I was moved by hearing the voices of other prisoners outside "my cell" while I looked at the plain space around me, which contained a bed, a bench, a tiny stool, a combination toilet and sink, and a few books and magazines. In such a sparse setting, the objects took on the gravitas of a Chardin still-life.

During the experience, a variety of statistics, quotes, and phrases from former inmates, guards, and even psychologists, are superimposed on the walls. At one point, you read, "Solitary confinement alters neural and psychological states" and "Even short-term isolation may alter brain activity."

You begin to feel as if you're floating. As you virtually hover near the ceiling of your cell, your "vision" (actually the video itself) starts to blur. This part of the experience is meant to give you the feeling of what it's like to feel disorientation and even hallucinate in solitary confinement. It's a powerful and unsettling effect.

"Many people have told us that it demonstrates in nine minutes what they can't begin to express in words," said Panetta.

Nonlinear Storytelling

Ryot Studio's Lauretti noted this first-person, point-of-view quality enables viewers to feel as though they're inside the story in a physical way, which is often referred to as the "sense of presence." It's the quality that makes you feel like you're truly there on top of Mount Everest or swimming under the ocean. And because a viewer can employ gestures or body language, such as a turn of the head to view different scenes or trigger actions, there's a profound shift in how the story is told.

"You're in control of what you see," said Lauretti, "and what information you have access to."

Niko Chauls, former director of emerging technology at USA Today Networks, who has led many of the company's VR and AR teams, concurs with Lauretti on the importance of choice and interactivity in VR, although he noted that it's difficult to do for those trained in traditional media.

"Giving control to consumers can be pretty scary to traditional storytellers," said Chauls, "but it can be powerful if it's embraced."

USA Today's USS Eisenhower

Chauls and his team tackled restructuring how a story is told in "USS Eisenhower VR," a project USA Today published this past summer. "It was our first large-scale, nonlinear, immersive storytelling experience," said Chauls.

"USS Eisenhower VR" documents life aboard the ship while it underwent sea trials before being deployed to the Middle East. Viewers first explore a large-scale model of the ship, where they can click on various hot spots and content on the model's deck. They can also choose the content they want to explore, which includes a variety of photo slideshows or 360-degree video.

Some videos nearly convince you that you're taking off or landing on the deck of the carrier via jet or a helicopter and can produce a very real bout of vertigo. Others are less dramatic—you're on the bridge and listening to an interview with the captain, or below deck with crew members.

"It essentially documents life aboard a nuclear aircraft carrier. But it's really meant to be explored and discovered, instead of watched from beginning to end," said Chauls.

Sound Matters

VR teams are experimenting with other elements, in addition to nonlinear narrative structures and immersive 360-degree video. One is audio.

"As any filmmaker knows, audio is extremely important. In VR, it is just as—if not more—important, because it is one of the ways people understand the space," noted the Times' Hopkins. "We can use spatial audio so that we can place sounds in space, so that when they hear something, they can hear it coming from a specific direction."

Zahra Rasool, editorial lead for Contrast VR

Zahra Rasool (above), editorial lead for Contrast VR, an immersive-media studio that creates VR experiences for Al Jazeera, said, "Audio gives you the sense of scale and location in VR. We use spatial audio in all of our productions and can convey a sense of environment and a sense of space. As a storyteller, it's powerful when you feel you need someone to be there in order to understand the gravity of the story and the situation."

"Sensations of Sound," from The New York Times, is a powerful example of using spatial audio to help tell a story. This VR creation is centered on Rachel Kolb and her experience of music. Kolb had been profoundly deaf her whole life—until a few years ago, when she was 20 and underwent surgery for cochlear implants, which allowed her to experience partial hearing.

Although Kolb, the story's narrator, hadn't been able to hear music for most of her life, she had still been able to experience it. As a child, she played piano and guitar. "She saw and felt music," said Hopkins, "even in ways that we as hearing people don't." But when Kolb first heard live music, "It was a jarring experience for her," said Hopkins, since it was previously far less dynamic.

"Sound in this piece is obviously important," said Hopkins. "We were able to use spatial audio as well as an interesting sound design to express some of the things that she is talking about in conveying her story."

At the end of "Sensations of Sound," Kolb asks, "Can you hear the music? Even though I now can, I think this question misses the point. Music is also visual, physical, tactile. It weaves its rhythms though our lives. I believe music becomes more remarkable when we experience it with our whole bodies."

Other elements of multimedia that are being explored in VR projects are motion graphics and animation elements. One of Contrast VR's first experiences for Al Jazeera, "I Am Rohingya," chronicles the life of Jamalida, a young woman from Myanmar who now lives in a refugee camp in Bangladesh. In one section, Jamalida describes her persecution in Myanmar. Since there was no footage of Jamilda in particular, Rasool said, "The best way to represent those memories and recollections was through digital animations." By focusing on just one point of view, Rasool and her team generated a powerful sense of empathy.

The Future of VR Journalism

Nonfiction VR hasn't fully arrived just yet. Lauretti, Chauls, and others note that one challenge is in demand and distribution. Many publishing and news organizations are still struggling with mainstream digital platforms and how to monetize those more accessible forms of media. And because VR projects generally require lots of people and time to produce, they're just too expensive for most outlets.

"Right now, the biggest challenge is about reach and scale," said Lauretti. "Comparing it to the expense of VR, written journalism is really quick and really cheap. And you can get massive scale with written journalism... On the distribution side, you also have a problem. Headsets aren't mainstream yet. The average consumer doesn't have the Microsoft HoloLens right now, or even the Samsung Gear [VR]. We haven't seen the mainstream adoption yet."

But like most digital technology, VR will doubtless become cheaper and more widely adopted. Sheppard predicted, "What you're going to see, in the short-term, is 360-degree video becoming just another format that consumers will come to expect for media consumption." Platforms including Facebook and YouTube already support 360-degree video.

Longer-term, Sheppard sees some really interesting opportunities around video and cloud-based technology: "If you can imagine a future where most people are wearing a small camera, and it's constantly gathering information (storing it both locally and to the cloud), I think citizen journalism will become one of the primary ways we will be involved in the news."

If we're all carrying connected cameras, we have the opportunity to be VR journalists ourselves. But Sheppard also noted that the public, media, and government will need to be vigilant in paying attention to the negative ramifications of VR. For example, he suggested that the confusion between fake news and real news may increase.

"We can already create highly realistic faces that appear to be saying any line you want them to say. And you wouldn't have any idea whether that was real or not," said Sheppard. "It may become difficult to unwind what's real from the unreal. How do we think about authenticity and authoritative sources when almost everything can be fabricated?"

Despite its challenging aspects (and sensitivity concerns—think back to the controversy surrounding Mark Zuckerberg's VR "tour" of storm-ravaged Puerto Rico), some see VR journalism as a possible way to solve current problems in the media and journalism.

"VR journalism really has the potential to rebuild trust between an audience and a reporter, because of the nature of 360-degree video capture," said Chauls. "You are removing layers of interpretation or layers of gatekeeping between the audience and the event." That is to say, with 360-degree video, there's generally very little editing other than the length of the video. So viewers might be less skeptical that the photographer or journalist is leaving out important information or footage.

Lauretti suggested something similar: "I would argue that every time [traditional] photographers put a camera up to their eye, they are cropping out or including just certain parts in a particular photo or video." So there's already an edit taking place at the beginning of shooting a traditional video or photo. "In a way, 360-degree video actually democratizes that process. Because we're actually not leaving anything out. We're actually showing you everything."

This could be one of the most important ways for VR to empower the public. "In a way," says Lauretti, "it almost leaves nothing to the imagination, but gives you, as the viewer, the opportunity to really see and take away whatever you want."

About Terry Sullivan