The Challenge of Latency in Virtual Reality

By Wesley Fenlon

Immersive VR with head tracking is so close we can taste it, but the issue of latency still has to be conquered to bring convincing virtual worlds to life.

The Oculus Rift bears a heavy burden of expectation. After a strong showing at E3 2012, the developer prototype VR helmet raked in $2.5 million in Kickstarter funding. Suddenly, the 80s dream of fully immersive virtual reality with head tracking and high definition displays seems within our grasp. And it's actually affordable. But will it be good enough?

In 1990, a VR setup far inferior to the Oculus Rift would've cost tens of thousands of dollars. Oculus Rift brings us so much closer to feasible virtual reality that the shortcomings of the technology will become glaringly apparent. It's sort of like the ninety-ninety rule: Finishing the last 10 percent of a project can take just as long as the first 90 percent. For virtual reality, overcoming latency will be that final challenge.

Photo Credit: NASA.

Valve's Michael Abrash posted a detailed blog in December about the challenges of latency in virtual reality. Just as our brains react differently to different framerates in film, they react differently to regular computer displays and virtual reality setups that encompass our full vision.

"When it comes to VR and AR, latency is fundamental – if you don’t have low enough latency, it’s impossible to deliver good experiences, by which I mean virtual objects that your eyes and brain accept as real," writes Abrash. "By 'real,' I don’t mean that you can’t tell they’re virtual by looking at them, but rather that your perception of them as part of the world as you move your eyes, head, and body is indistinguishable from your perception of real objects. The key to this is that virtual objects have to stay in very nearly the same perceived real-world locations as you move; that is, they have to register as being in almost exactly the right position all the time. Being right 99 percent of the time is no good, because the occasional mis-registration is precisely the sort of thing your visual system is designed to detect, and will stick out like a sore thumb."

Abrash says that most games operate fine with 50ms or more of lag between the movement of a mouse and action happening on screen. But when you throw in head tracking and screens that totally blot out the world, our visual expectations change. 50ms won't cut it.

"With AR/VR, all the [brain] processing power that originally served to detect anomalies that might indicate the approach of a predator or the availability of prey is brought to bear on bringing virtual images that are wrong by more than a tiny bit to your attention. That includes images that shift when you move, rather than staying where they’re supposed to be – and that’s exactly the effect that latency has.

"Suppose you rotate your head at 60 degrees/second. That sounds fast, but in fact it’s just a slow turn; you are capable of moving your head at hundreds of degrees/second. Also suppose that latency is 50 ms and resolution is 1K x 1K over a 100-degree FOV. Then as your head turns, the virtual images being displayed are based on 50 ms-old data, which means that their positions are off by three degrees, which is wider than your thumb held at arm’s length. Put another way, the object positions are wrong by 30 pixels. Either way, the error is very noticeable."

Predictive head tracking offers a solution--pre-rendering the image for where players will look before they turn their heads--but it's extremely difficult for interactive games. Players may move unpredictably, and it's hard to know when they'll stop moving. Inaccurate prediction is worse than none at all.

So where does that leave the Oculus Rift?

Apparently, the Oculus Rift will perform around that 90 percent point. Ars Technica writes that the Oculus Rift's dev kit will achieve latency as low as 30 or 40 milliseconds, not accounting for the delay of the display itself drawing pixels.

Abrash suggests VR will need latency as low as 15ms or even 7ms to work perfectly. How do we get there? 120Hz or 240Hz displays are part of the solution, though Abrash writes in depth about other improvements that need to be made to optical tracking and CPU/GPU tech to get latency down into the 15ms range.

The first version of Oculus Rift won't be the perfect VR helmet, but it will move us far closer to the ideal technology. A consumer version of the headset is expected after the dev kits are released with higher resolution displays and other improvements. That won't be the perfect VR headset, either, but it might be the piece of technology to blow virtual reality wide open, which Abrash is hoping for:

"...there is no way to get low enough display latency out of existing hardware that also has high enough resolution, low enough cost, appropriate image size, compact enough form factor and low enough weight, and suitable pixel quality for consumer-scale AR/VR. (It gets even more challenging when you factor in wide FOV for VR, or see-through for AR.) Someone has to step up and change the hardware rules to bring display latency down. It’s eminently doable, and it will happen – the question is when, and by whom. It’s my hope that if the VR market takes off in the wake of the Rift’s launch, the day when display latency comes down will be near at hand."