Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.
 The ability to fill in the gaps in high-dimensional data is a fundamental cognitive skill. Suppose you glance out of the window and see a person in uniform approaching your gate carrying a letter. You can easily imagine what will (probably) happen a few seconds later. The person will walk up the path and push the letter through your door. Now suppose you glance out of the window the following day and see a person in the same uniform walking down the path, away from the house. You can easily imagine what (probably) happened a few seconds earlier. The person came through your gate, walked up the path, and delivered a letter. Moreover, in both instances, you can visualize the scene from different viewpoints. From your vantage point at the window, you can imagine how things might look from the gate, or from the front door, or even from your neighbour's roof. Essentially, you have learned from your past experience of unfolding visual scenes how to flexibly extrapolate and interpolate in both time and space. Replicating this ability is a significant challenge for AI.To make this more precise, let's first consider the traditional video prediction setup. Video prediction is typically framed as a sequential forward prediction task. Given a sequence of frames f 1 , ..., f t , a model is tasked to generate future frames that follow, f t+1 , ..., f T . Existing models BID0 BID7 carry out the prediction in an autoregressive manner, by sequentially sampling frame f t+n from p(f t+n |f 1 , ..., f t+n−1 ).This traditional setup is often limiting -in many cases, we are interested in what happens a few seconds after the input sequence. For example, in model based planning tasks, we might want to predict what frame f T is, but might not care what intermediate frames f t+1 , ..., f T −1 are. Existing models must still produce the intermediate frames, which is inefficient. Instead, our model is "jumpy" -it can directly sample frames in the future, bypassing intermediate frames. For example , in a 40-frame video, our model can sample the final frame 12 times faster than an autoregressive model like SV2P BID0 .More generally , existing forward video prediction models are not flexible at filling in gaps in data. Instead, our model can sample frames at arbitrary time points of a video, given a set of frames at arbitrary time points as context. So it can, for example, be used to infer backwards or interpolate between frames, as easily as forwards. Our model is also not autoregressive on input or output frames -it takes in each input frame in parallel, and samples each output frame in parallel.In our setup of "jumpy" video prediction, a model is given frames f 1 , ..., f n from a single video along with the arbitrary time-points t 1 , ..., t n at which those frames occurred. To be clear, t 1 , ..., t n need not be consecutive or even in increasing order. The model is then asked to sample plausible frames at arbitrary time points t 1 , ..., t k . In many cases there are multiple possible predicted frames given the context, making the problem stochastic. For example, a car moving towards an intersection could turn left or turn right. Although our model is not autoregressive, each set of k sampled frames is consistent with a single coherent possibility, while maintaining diversity across sets of samples. That is, in each sampled set all k sampled frames correspond to the car moving left, or all correspond to the car moving right.At a high level, our model, JUMP, takes in the input frames and samples a stochastic latent that models the stochasticity in the video. Given an arbitrary query time-point, the model uses the sampled latent to render the frame at that time-point. The model is not autoregressive on input or output frames, but still captures correlations over multiple target frames. Our method is not restricted to video prediction. When conditioned on camera position , our model can sample consistent sets of images for an occluded region of a scene, even if there are multiple possibilities for what that region might contain.We test JUMP on multiple synthetic datasets. To summarize our results: on a synthetic video prediction task involving five shapes, we show that JUMP produces frames of a similar image quality to modern video prediction methods. Moreover, JUMP converges more reliably, and can do jumpy predictions. To showcase the flexibility of our model we also apply it to stochastic 3D reconstruction, where our model is conditioned on camera location instead of time. For this, we introduce a dataset that consists of images of 3D scenes containing a cube with random MNIST digits engraved on each of its faces. JUMP outperforms GQN (Eslami et al., 2018) on this dataset, as GQN is unable to capture several correlated frames of occluded regions of the scene. We focus on synthetic datasets so that we can quantify how each model deals with stochasticity. For example, in our 3D reconstruction dataset we can control when models are asked to sample image frames for occluded regions of the scene, to quantify the consistency of these samples. We strongly encourage the reader to check the project website https://bit.ly/2O4Pc4R to view videos of our experiments.To summarize, our key contributions are as follows.1. We motivate and formulate the problem of jumpy stochastic video prediction, where a model has to predict consistent target frames at arbitrary time points, given context frames at arbitrary time points. We observe close connections with consistent stochastic 3D reconstruction tasks, and abstract both problems in a common framework.2. We present a model for consistent jumpy predictions in videos and scenes. Unlike existing video prediction models, our model consumes input frames and samples output frames entirely in parallel. It enforces consistency of the sampled frames by training on multiple correlated targets, sampling a global latent, and using a deterministic rendering network.3. We show strong experimental results for our model. Unlike existing sequential video prediction models, our model can also do jumpy video predictions. We show that our model also produces images of similar quality while converging more reliably. Our model is not limited to video prediction -we develop a dataset for stochastic 3D reconstruction. Here, we show that our model significantly outperforms GQN. We have presented an architecture for learning generative models in the visual domain that can be conditioned on arbitrary points in time or space. Our models can extrapolate forwards or backwards in time, without needing to generate intermediate frames. Moreover, given a small set of contextual frames they can be used to render 3D scenes from arbitrary camera positions. In both cases, they generate consistent sets of frames for a given context, even in the presence of stochasticity. One limitation of our method is that the stochastic latent representation is of a fixed size, which may limit its expressivity in more complicated applications -fixing this limitation and testing on more complex datasets are good avenues for future work. Among other applications, video prediction can be used to improve the performance of reinforcement learning agents on tasks that require lookahead BID21 . In this context, the ability to perform jumpy predictions that look many frames ahead in one go is an important step towards agents that can explore a search space of possible futures, as it effectively divides time into discrete periods. This is an avenue we will be exploring in future work.
We present a model for consistent 3D reconstruction and jumpy video prediction e.g. producing image frames multiple time-steps in the future without generating intermediate frames. This paper proposes a general method for indexed data modeling by encoding index information together with observation into a neural network, and then decode the observation condition on the target index. Proposes using a VAE that encodes input video in a permuation invariant way to predict future frames of a video.