Achieving the AR dream through games

In a laboratory in Atlanta there is a group of people whose job it is to play games. In fact, there’s a good part of Georgia Tech University that’s been largely converted for the purpose, but you’d be wrong if you thought that the Augmented Environment Labs have been sponsored by Microsoft or were developing console titles for PlayStation. What the team is actually studying is augmented reality. It just so happens that the best field do so is in gaming.

“Gaming is going to be the main driver of AR,” explains director of the AEL, Professor Blair MacIntyre.

“Up till now, we’ve seen two main uses of AR. First, there’s been the web based advertising stuff. The technology has been good enough and, since Flash, people can just use their browser to view it. There’s no need to have anything else. It’s used as a bit of a gimmick to generate a buzz within an industry because that’s how marketing works.

“The second use has been in mobile information browsing with apps like Layar, Wikitude, Nearest Tube, Near Tweet and others. There have been some attempts at games but, until recently, it’s all about ‘look out there at what’s floating around you.’”

Even from half way round the world down the end of the telephone, it doesn’t take a genius to tell that MacIntyre is none too impressed with the current available crop mobile AR apps. It’s of little surprise when you consider that, as one of the pioneers of augmented reality, he created a more profound and useful experience nearly 20 years ago working at Columbia University with colleagues Steve Feiner and Doree Seligmann, both of whom have also gone on to become top researchers in the field.

The prototype they created was called KARMA (Knowledge-based Augmented Reality for Maintenance Assistance) and it used a head-mounted display (HMD) to show simple end-user instructions for how to service a laser printer by using a ghost image on the real background. Granted, computers, HMDs and, of course, laser printers have come on a fair bit since 1991 but consumer clutch on AR, sadly, doesn’t seem to have kicked on at the same rate. According to MacIntyre, the trouble seems to be our expectations:

“People have latched onto the dream of walking down the street and seeing what the Terminator does and we’re just not anywhere near there yet.”

The good news, though, is that we seem to be on the way.

“Devices are finally able to do the graphics. A few years ago it was all backpacks and HMDs. We took this game and made a whole room out of it with massive devices and trackers and huge HMDs everywhere and it just about worked. You could do all that now with a handheld.”

What MacIntyre is referring to is what many considered to be a tour de force of AR when it was created in 2006. AR Facade was the AEL’s take on the hit game Facade which was a one act interactive drama based on a story of a couple in the virtual world of their apartment. To give it an AR twist, the team at Georgia Tech recreated that apartment in the real world to the exact measurements with all the identical furniture precisely where it should be. They then sent the player into the real space, backpack and all, to wander around mixing the actual with the virtual experience in true augemented reality fashion. What it proved, rather than just a novel way to play the game was that they could track the user perfectly laying the two worlds over each other precisely and without a hitch.

Just a few years on and those more powerful handheld devices were readily available and it was the original Nvidia Tegra system-on-a-chip that brought about what MacIntyre describes as as “sweet spot in technology” from the test kits and their image based tracking devices. The result was the infinitely more playable and hilariously named ARhrrrr! zombie survival shooter tabletop game that you can enjoy in the video below and so successful was it that it caught the eye of a certain mobile chip giant.

“Qualcomm brought all the technology we have now and made a lab with the best tracking team out there in Vienna. We get all sorts of experimental hardware and software from them including Snapdragon systems, we’ve turned a large part of the lab turned into a games studio at Georgia tech and we’ve even got all the desks configured for maximum collaboration of ideas with art and design crossing over. It’s a big space for people to work in.”

Since Qualcomm has been onboard, the AEL has 30 people working on AR games and where the barriers were all about expense of equipment and painstaking coding, the time has begun to tackle a new issue.

“Using big old HMDs and tracking devices as well as low level programming languages meant it all took a lot of time before, but we’ve built a lot of tools within the AR community since. Finally we have good tools and good devices, the hard part now is figuring out what makes a good game.”

And that’s exactly what the AEL team has been up to ever since. Head over to the AELYouTube channel and you’ll find a raft of different demos and ideas mostly based around the table top gaming experience and, even though the decision to keep it to that format has largely been one of convenience, it brings an interesting return to a more traditional social experience, something in which MacIntyre has seen a significant advantage.

“Multiplayer games on handhelds and computers are often very solitary experiences. In World of Warcraft and others, it’s about playing together playing alone. You don’t interact in a physical way or look at the other players in the face. We’re recreating the experience more like a board game or card game. It’s also much more doable for AR.”

Doable has been the key for much of the work at Georgia Tech which is why they’ve also been working on the recently launched AR tools Kharma and Argon. The former - with the same name as MacIntyre’s very first project in the 1990s, an irony not lost on the professor - is an open source platform which allows developers to write AR programs with the normal html and kml web tools used on the Internet. The idea is that there’s no particular specialist knowledge required to write for augmented reality.


“Most AR tools are for very specific program environments. Normally, a browser calls up the server which sends down a load of html to decode into what we see. The idea is to repeat that process for AR information. Your browser will then translate what comes down into an AR experience if people have accounted for that up in the server.”

At the other end of the equation, Argon is a mobile browser built for the iPhone that makes it possible for someone to create an augmented reality app without any real knowledge of how to do so beyond a few standard scripts.

“We take the servers, add some code to recognise Argon requests and send back kml based on location and html and images but now re-purposed to fit your environment. This means less number crunching needed on your device, so it can happen on simpler devices and also saves on battery life too. It only needs to pass down information of whatever is relevant right now. Not everything at once.”

The goal, of course, is to put easy to use AR tools in the hands of everyone rather than people who’re willing to invest both a lot of time and money into an app. The ability to run 3D is expected to be added shortly to the system which will eventually be fully open source. So once the power to create AR has gone mass market, will that mean that the augmented reality has finally arrived in its full form? As one might predict, the answer is “not quite”.

“We’re 80 per cent of the way there with AR but the last 20 is going to be hard. That first 80 is the geocodes, the tweets, map directions and all that information, and I can get them out but GPS on mobiles isn’t good enough. Stuff doesn’t line up. Even if it’s off by 5-20m it won’t line up and it won’t work properly.

“All AR in sport is beautifully integrated. Everything looks right in speed skating and football with the flags on the ice and the skaters going past but there’s no way you can do that on mobiles right now. The difference is that sport is a fixed environment controlled and adjusted in real time by a bunch of guys sitting in a van or a studio. We can’t even track a phone properly, so we don’t know where it is and what to line up.”

All the same, the limitations of AR once you step out of a controlled environment haven’t stopped the likes of some very big players getting rather interested in this emerging space and what the AEL has made of it.

“CNN very is excited. We explained to them how it could work for events like the Times Square bombing. You could turn up with your phone and see it all in AR with how it happened or what might have happened and all with just a bit of coding in the web design.”

So, is that to be the extent of AR’s usefulness in society? What other applications does MacIntyre feel are going to be important other than virtual tours and gaming?

“The bigger question is what are the big things that AR will be useful for. Two years ago I might have said one thing or another but now, I have no idea. If you went back and asked Tim Berners-Lee and all those other guys who made the Internet what it was going to be used for, they’d probably have talked about the sharing of information. It never would have occurred to them that there would be YouTube and Flickr and Facebook and everything else.

“In the same way, we’re letting go of what AR might be used for as well, and part of our work is focused on admitting that we don’t know what the analogues to Facebook, YouTube and Flickr are. We’re making accessible tools for an Internet of AR where other people can figure out the applications.

“Not a lot is going to happen until we have head worn displays. It’ll be more compelling when it’s there. The day I have a pair of funky sunglasses and walk around with non-obnoxious advertising, news, social networking, totally immersed in the world, then AR will really have arrived.

“It has to have all the little things presented in a way that doesn’t cut me off. It’s not a Wall-e future; not virtual reality with people going around too absorbed in screens to know about others or the real world. It should enhancing our existence in the physical world; bring the digital world into the physical one.

“But before that, there are so many huge questions to answer such as if we can get all of the technology we need onto a mobile device; can we make it usable and not invasive, not a distopian existence and can we use it to build a postive future?

“Will AR like that arrive in my lifetime? Well, perhaps something of it, but not quite so well as we see now done in sports.”

