Deep-learning algorithm creates videos of the future

November 29, 2016 by Adam Conner-Simons

Given a still image from a scene, the CSAIL team's deep-learning algorithm can create a brief video that simulates the future of that scene. Credit: Massachusetts Institute of Technology

Living in a dynamic physical world, it's easy to forget how effortlessly we understand our surroundings. With minimal thought, we can figure out how scenes change and objects interact.


Recently, researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have moved a step closer, developing a deep-learning algorithm that, given a still image from a , can create a brief video that simulates the future of that scene.

Trained on 2 million unlabeled videos that include a year's worth of footage, the algorithm generated videos that human subjects deemed to be realistic 20 percent more often than a baseline .

The team says that future versions could be used for everything from improved security tactics and safer self-driving cars. According to CSAIL PhD student and first author Carl Vondrick, the algorithm can also help machines recognize people's activities without expensive human annotations.

"These videos show us what computers think can happen in a scene," says Vondrick. "If you can predict the future, you must have understood something about the present."

Vondrick wrote the paper with MIT professor Antonio Torralba and Hamed Pirsiavash, a former CSAIL postdoc who is now a professor at the University of Maryland Baltimore County (UMBC). The work will be presented at next week's Neural Information Processing Systems (NIPS) conference in Barcelona.

How it works

Multiple researchers have tackled similar topics in computer vision, including MIT Professor Bill Freeman, whose new work on "visual dynamics" also creates future frames in a scene. But where his model focuses on extrapolating videos into the future, Torralba's model can also generate completely new videos that haven't been seen before.


"Building up a scene frame-by-frame is like a big game of 'Telephone,' which means that the message falls apart by the time you go around the whole room," says Vondrick. "By instead trying to predict all frames simultaneously, it's as if you're talking to everyone in the room at once."

The video will load shortly.



The team used a deep-learning method called "adversarial learning" that involves training two competing neural networks. One network generates video, and the other discriminates between the real and generated videos. Over time, the generator learns to fool the discriminator.


Testing the scene


Vondrick stresses that the model still lacks some fairly simple common-sense principles. For example, it often doesn't understand that objects are still there when they move, like when a train passes through a scene. The model also tends to make humans and objects look much larger in size than reality.


"It's difficult to aggregate accurate information across long time periods in videos," says Vondrick. "If the video has both cooking and eating activities, you have to be able to link those two together to make sense of the scene."


"In the future, this will let us scale up vision systems to recognize objects and scenes without any supervision, simply by training them on ," says Vondrick.

Explore further: Deep-learning vision system anticipates human interactions using videos of TV shows

More information: Paper: "Generating Videos with Scene Dynamics"

Provided by Massachusetts Institute of Technology

add to favorites email to friend print save as pdf