Ben Grosser

Artist Ben Grosser has explored computer vision through a project that investigates how a machine might watch movies such as The Matrix, Inception, 2001: A Space Odyssey, Taxi Driver and Annie Hall.

He wrote software with computer vision algorithms that watch for areas of prominence, patterns, colours and other aspects of each frame in a film in order to identify and track items of interest. These might be faces, buildings, signs or landscape elements.

Computers Watching Movies (2001: A Space Odyssey)benjamin grosser

"Whatever it is, how the software decides what to focus on is driven by simple intelligence algorithms -- namely stochastic processes -- that give the software some degree of agency," Grosser told "When it chooses something to 'look' at, it represents that act of looking through an interactive temporal 'sketching' process. In other words, the system draws in response to what it sees."

In the resulting artworks you hear the audio of the original film overlaid with a blank white canvas and an evolving sketch dancing across the screen. Grosser is particularly interested in what the computer focuses on compared with what a human might focus on.

The project was born after Grosser noticed that "portrayals of computer vision are typically based on what they do for humans, such as face identification for automated surveillance or car tracking for traffic analysis".

"I wondered what computer vision would look like if it wasn't so instrumental, but instead showed the computer watching something for its own purposes instead of ours. What would computers see if they watched movies?"

According to Grosser, the biggest challenge of the project was conceptualising the computer as its own vision machine while trying to understand what that vision reveals about our culturally developed ways of looking. "When viewing a movie, why do we watch what we watch when we watch it? Will a system without our sense of narrative or historical patterns of vision watch the same things?"

Computers Watching Movies (Inception)benjamin grosser

The Inception clip he analysed illustrates the difference between computer vision and human vision. When the computer watches explosions it views them as a series of small but rapid changes across the visual field. It is drawn to the movement of each little piece flying around the frame. It focuses on so many of these movements that its new sketches completely obliterate those it created before.

"But when I view that clip, I mostly watch the origins of the explosions, and as much as anything else, focus my attention on those aspects of the frame that aren't moving. I watch the actors and look for their reactions to the situation," Grosser says, adding that he wonders how much his understanding of narrative plays into this difference.

Check out all of the clips on Grosser's site.


Latest on