Prediction is arguably one of the most basic functions of an intelligent system. In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult. However, most phenomena naturally pass through relatively predictable bottlenecks---while we cannot predict the precise trajectory of a robot arm between being at rest and holding an object up, we can be certain that it must have picked the object up. To exploit this, we decouple visual prediction from a rigid notion of time. While conventional approaches predict frames at regularly spaced temporal intervals, our time-agnostic predictors (TAP) are not tied to specific times so that they may instead discover predictable "bottleneck" frames no matter when they occur. We evaluate our approach for future and intermediate frame prediction across three robotic manipulation tasks. Our predictions are not only of higher visual quality, but also correspond to coherent semantic subgoals in temporally extended tasks. Imagine taking a bottle of water and laying it on its side. Consider what happens to the surface of the water as you do this: which times can you confidently make predictions about? The surface is initially flat, then becomes turbulent, until it is flat again, as shown in Fig 1. Predicting the exact shape of the turbulent liquid is extremely hard, but its easy to say that it will eventually settle down.Prediction is thought to be fundamental to intelligence BID2 BID3 BID10 . If an agent can learn to predict the future, it can take anticipatory actions, plan through its predictions, and use prediction as a proxy for representation learning. The key difficulty in prediction is uncertainty. Visual prediction approaches attempt to mitigate uncertainty by predicting iteratively in heuristically chosen small timesteps, such as, say, 0.1s. In the bottle-tilting case, such approaches generate blurry images of the chaotic states at t = 0.1s, 0.2s, . . ., and this blurriness compounds to make predictions unusable within a few steps. Sophisticated probabilistic approaches have been proposed to better handle this uncertainty BID0 BID19 BID4 BID39 .What if we instead change the goal of our prediction models? Fixed time intervals in prediction are in many ways an artifact of the fact that cameras and monitors record and display video at fixed frequencies. Rather than requiring predictions at regularly spaced future frames, we ask: if a frame prediction is treated as a bet on that frame occurring at some future point, what should we predict? Such time-agnostic prediction (TAP) has two immediate effects: (i) the predictor may skip more uncertain states in favor of less uncertain ones, and (ii) while in the standard approach, a prediction is wrong if it occurs at t ± rather than at t, our formulation considers such predictions equally correct.Recall the bottle-tilting uncertainty profile. Fig 1 depicts uncertainty profiles for several other prediction settings, including both forward/future prediction (given a start frame) and intermediate prediction (given start and end frames). Our time-agnostic reframing of the prediction problem targets the minima of these profiles, where prediction is intuitively easiest. We refer to these minima states as "bottlenecks."At this point, one might ask: are these "easy" bottlenecks actually useful to predict? Intuitively, bottlenecks naturally correspond to reliable subgoals-an agent hoping to solve the maze in Fig 1 (e) would do well to target its bottlenecks as subgoals. In our experiments, we evaluate the usefulness of our predictions as subgoals in simulated robotic manipulation tasks.Figure 1: (a) Over time as the bottle is tilted, the uncertainty first rises and then falls as the bottle is held steady after tilting. (b)-(e) Similar uncertainty profiles corresponding to various scenarios-a ball rolling down the side of a bowl, a car driving on a highway with an exit 100m away, an iron pellet tossed in the direction of a magnet, and intermediate frame prediction in a maze traversal given start and end states. The red asterisks along the x-axis correspond to the asterisks in the maze-these "bottleneck" states must occur in any successful traversal.Our main contributions are: (i) we reframe the video prediction problem to be time-agnostic, (ii) we propose a novel technical approach to solve this problem, (iii) we show that our approach effectively identifies "bottleneck states" across several tasks, and (iv) we show that these bottlenecks correspond to subgoals that aid in planning towards complex end goals. The standard paradigm for prediction tasks demands that a predictor not only make good predictions, but that it make them on a set schedule. We have argued for redefining the prediction task so that the predictor need only care that its prediction occur at some time, rather than that it occur at a specific scheduled time. We define this time-agnostic prediction task and propose novel technical approaches to solve it, that require relatively small changes to standard prediction methods. Our results show that reframing prediction objectives in this way yields higher quality predictions that are also semantically coherent-unattached to a rigid schedule of regularly specified timestamps, model predictions instead naturally attach to specific semantic "bottleneck" events, like a grasp. In our preliminary experiments with a hierarchical visual planner, our results suggest that such predictions could serve as useful subgoals for complex tasks.In future work, we would like to address some limitations of our TAP formulation, of which we will mention two here. First, TAP currently benefits not only from selecting which times to predict, but also from not having to provide timestamps attached to its predictions. We would like to study: could we retain the benefits of time-agnostic prediction while also providing timestamps for when each predicted state will occur? Second, our current TAP formulation may not generalize to prediction problems in all settings of interest. As an example, for videos of juggling or waving, which involve repeated frames, TAP might collapse to predicting the input state repeatedly. We would like to investigate more general TAP formulations: for example, rather than choosing E t in Eq 5 to encourage predicting farther away times, we could conceivably penalize predictions that look too similar to the input context frames. More broadly, we believe that our results thus far hold great promise for many applications of prediction including hierarchical planning and model-based reinforcement learning, and we hope to build further on these results. In these appendices, we provide details omitted in the main text for space. Note that more supplementary material, such as video examples, is hosted at: https://sites.google.com/view/ ta-pred
In visual prediction tasks, letting your predictive model choose which times to predict does two things: (i) improves prediction quality, and (ii) leads to semantically coherent "bottleneck state" predictions, which are useful for planning. A method on prediction of frames in a video, the approach including that target prediction is floating, resolved by a minimum on the error of prediction. Reformulates the task of video prediction/interpolation so that a predictor is not forced to generate frames at fixed time intervals, but instead is trained to generate frames that happen at any point in the future.