I just started doing some Ubuntu stuff in the late 2000s, and I've been working deeply or directly with Debian for about six months now. So definitely a newbie in all of this. But we have a purpose and a reason for doing this, and it's to support HP's latest cloud product called HP OpenStack Helion. That's the formal marketing name for it, and there's a couple of different flavors of that. This slide is supposed to show the product delivery ecosystem. And we need to talk about the cloud product just a little bit to understand why Linux has become so important to us. The Helion product is delivered as two separate additions. One is kind of a do-it-yourself, download free, no support. Make those kind of experiments and give it a try and a run on its own. And then we have a for-sale version that comes with full support, installation, all that kind of thing. The intent is to deliver an entire stack from the bottom where we would have hardware, our servers, our networking, our storage, that kind of thing, all the way up through all the solutions, the consulting, the installation, all that kind of help. So that's what this slide is supposed to be showing in the vertical direction, from all the way at the bottom of the hardware, up to the top with guest operating systems. Now, we have lots of cloud developers. We have lots of people playing in the open stack space. And that's where they like to concentrate. They want to, their idea of low level is Python. They don't want to go below that. They want to stay up in the cloud and the VMs and the virtual networking and all that stuff. And that's good. We need people like that. That's where we have to have those people take the direction that we want to go with this. However, everyone in this room knows for sure that somewhere there's Linux running. Somewhere there's real hardware. Somewhere there's electrons and spinning disks and things like that. Linux matters. Our group was formed to take care of the fact that Linux really does matter. So hopefully that's the last we'll talk a lot about open stack. If you've looked through the diagram there a little bit, you can see sort of towards the bottom just above the hardware is this thing called HLinux, which is going to advance or not. There we go. We came up with an implementation that we were calling HLinux. We're in this for a long time. One of the tenets of the cloud is for a large company like HP, we can never turn this off. Once we've started taking on data and taking on customers and doing things, we can't change our minds in three years and say, just kidding, we didn't really want to do this after all. We're in this for the long run and we have to make decisions based on supporting those long run choices. So we've gone with our own Linux and HP Linux for two major reasons. One is the business side of the equation. Why have our own Linux? Well, if you remember back to that stack picture, there's a couple of major segments to it. We have a lot of influence on open stack. We have a lot of the project team leaders, half a dozen project team leaders and several dozen core reviewers and lots of influence in the open stack space. We've made our commitment there. We believe that's the way to go. Our current public cloud is based heavily on open stack. So we've got the history to understand that that's a good decision for us to make. We control the HP value add. It's not just open stack and healing on. There's a lot of HP customized things that allow cloud operators, whether they're private cloud, public cloud or hybrid, to use our product to their advantage. All the way at the bottom, we control the hardware. We've got the servers, the networking stuff, the switches, the storage, all the things that it takes to actually support the cloud because there is the physical side. But one part where we do not have equivalent control over our own destiny is in the actual operating system, the real operating system that's there. We are currently using another Debian derivative that is more of a co-op petitive. Co-op petition rings pretty good. Co-op petitive is hard. Arrangement with them, they have their own cloud aspirations and the things that they want to do. We're in this for the long run. We're not sure that we can guarantee or control the level of cooperation we can get from these people over that long run. Nothing evil, nothing bad. But we want a similar level of control that we have in the other three areas. That's why we want our own Linux distribution. It restores that piece of control that we need over the entire equation and now we can go forward with confidence. The other side of it is technical drivers. Why did we choose Debian? There are plenty of choices out there for a base to do a distribution from. We chose Debian for a couple of reasons. One is directives from on high that our cloud solution will be developed in open source. The final solution will be open source except for the HP value add stuff at the very top. But the foundation, the under layers, everything talking to the hardware is going to be open source. Our current HP public cloud is based on a Debian derivative. So we've got lots of experience in the mostly Debian space. HP does have a long and comfortable and good record of working with Debian and the Debian community. We want to leverage off of that. And actually we want to improve where we've been for the last couple of years and restore some of the things that were very, very true say 10 years ago. And Debian is open enough. It was there, it's available. There's things that we can do to it if we need to for our needs. And then there's ways we can give that back to the community in a very comfortable and supportive fashion. So that's where we're headed is our own Debian based distro. Then there's time to market. A lot of these plans were put into place quite a while ago. As I said, I've only been diving into Debian for about six months. That's when our group was formed. But there were already schedules set up to achieve these type of things with our own distro. And they were pretty short. There was a story recently about a train operators back in the 1960s in New York state that were looking for ways to speed up their schedules, get more trains on the tracks, get more passengers to destinations. And the big rigs asked, how fast can our trains go? We need to make our trains go faster. What's the maximum speed? By the way, you only have a month to figure it out. And this is what they came up with. This is exactly what they did. They strapped some jet engines onto the train and ran it up the speed. And it still holds the world's record for the fastest self-propelled train. That's engineering I like. The only thing I don't approve of is that they wore helmets during the test. What was that for? Not going to help.  The helmets won't make a difference. So along those lines, we've charged ahead blindly, boldly, not knowing any better. And this is what we've come up with. We call it H Linux. And there's a secret to that, and I'll let you in on it later. It's composed of three main parts. One is Debian.org. The user space comes from Debian.org. We chose testing. So we drive everything from testing. We update it frequently. We're doing two architectures. AMD64, obviously. Why? I386, it feels like a smart thing to be doing from the get-go. We might change our mind on that, but it seemed like it would be easier to take it out later than add it later. So we started with that now. That ends up being about 65,000 packages in our full repo. And HP has this internal process called the open source review board where we have to examine the licenses and the contents of every source package to make sure that we're not in violation of some GPL or an Athena license or the Afaro license, things like that. So we have a dedicated legal group that inspects all this stuff. When we start from Debian as the source, that job is greatly simplified. It's very clear and very open. The disclosure is there. We don't use the kernel that we find in Debian testing. We wanted something much later. So we go straight to kernel.org and get essentially a month or two ago's kernel and track that continuously through time. Obviously, we have our own custom.config that we're tweaking as we go along. The idea being to best support OpenStack and Helion on HP hardware, but not exclusively that. Finally, there's foreign packages. There are some Debian packages and non-Debian sets of things out there that OpenStack needs to run. We carry those as a one-off support effort on our part. Our goal would be to take that and work with the upstream owners or maintainers such as they are and help them get it into Debian testing so that it's not a one-off for us anymore. It's driven by the community. But those things that we mash up call HLinux. We roll that. And that's available. All 65,000 packages are available for the development group to use. Turns out they only need about 600 of them. We're not putting a graphic environment on this. This is mostly server-type code. So those 600 packages are the ones that we're really, really concerned with. For Helion, we're using TripleO from OpenStack. OpenStack on OpenStack for the deployment and development process. That is, the manufacturing is centered around something called Disk Image Builder. It's a name of a program or Dib. It's essentially a smart wrapper around Deb Bootstrap. So Deb Bootstrap does what it does and then Disk Image Builder pulls down extra things, does some other integration, manages the configuration, does all that kind of stuff. And what they end up with then is something that they can run down through more integration. Helion does not use the OpenStack bits in Debian. They're using them directly from OpenStack.org. So they're much tighter to upstream, kind of like we're doing with testing and kernel.org. They also pull things from PyPy and some other sources out on the net. There's some HP value-added code, or lots of HP value-added code, and then configuring all of that together, that's where they end up with that 600 or so packages that they're actually using from the HLinux repo. We get that information back as to what they're using so that we can do lots of repo manipulations calculations, resolve some issues. One of the things we can do very easily is this OSRB and security review of the packages that you're using, what's the license in each one, or which one of them have some kind of security notification active against them. And we can advise them, change this package, try something different. Did you know this has a security hole in it? We can start doing product comparison and see what we're using and how we're using it. Where should we focus our efforts? Where should we invest in the open source community? What packages should we be worried about? Can we try to predict the stability or the bugability? I think I heard the word the other day. Where can we go with this? And then finally, feed that back into the repo that we deliver. How often do we do this? What pieces do we deliver? What else can we do? So it's a very active loop. We transform it a little bit along the way. We're trying to keep it simple so we don't have a full offering of everything. None of the other flavors are there. We're just working with testing. And then we do some branding or rebranding. We were a little bit concerned about some of these packages escaping into the wild, if you will, that a DEB package would end up on somebody else's system that really came from us and that we had touched and maybe done something to. So we changed the branding to make sure that it's blatantly obvious. No, this particular package came from HP. We don't know where we got it, but it came from HP. We're not going to blame Debian for any problems we have with this. So we hope that's what will happen. We've done a couple of reference ISOs for people to play with and get some practical feel for what's in HLinux. It's not really part of our original design intent or product deliverables, but they're starting to gain attraction. And as has been discussed in some other forms here, the installer takes a little bit of hand holding and getting things to work. It's absolutely getting better. We've got a guy who's working on that and will be delivering some custom ISOs to different internal customers. So this is kind of a rough flow of exactly how we do it. As I said, we've got three major sources. Go from Debian testing from an internal mirror that we maintain inside of HP. Kernel.org, we take the N-1 kernel or one from a couple of months ago. And then these foreign packages from different sources whether they're from Debian unstable, we consider that to be a foreign package because it's outside of Debian testing. So anything outside of Debian testing is a foreign package to us. There's some very one-off packages, some tar balls, things like that. We would love to drive those to zero, but that's not practical right this minute. We take all three of those and kind of mash them up into a giant pool or a prototype pool and do rebranding in place on that pool. Push that over to the next chunk of the pipeline and that's where we take the pool and run all the standard Debian tools on it to create the disks, the reference files, do the GPG signing, all those type of things. Got two big explosions here on validation and testing. I'll talk about that in just a minute. Once we believe we have a good repo that looks and smells like a Debian network repo, supplies the packages we need for the use cases that we have, which is HP Helion, then we'll make it available internally only on different sites and we encourage other sites within HP to cache that or mirror it themselves because we've got kind of a worldwide effort going on this. The other thing that we host here is archives and I'll talk about that in a minute, long-term archives of every one of these repo roles and we have the ability to do one-off repos or special repos or custom lists of packages for people who need it and that's all hosted on the tail end here. One of the goals was to chase CI, CD, Agile, DevOps style repo as best we could. We don't really have a version, a solid version. We are starting to establish a regular release cadence but for us, we look at the top of the tree and how can we copy the latest changes out of testing, kernel.org and these other packages and keep them fresh and available and ready to go. We did change the nomenclature a little bit. We don't call it testing to avoid confusion. We call the release that we have cattle prod and you'll see some of us running around with cattle prod hats or if you're wearing a cattle prod hat, let's raise your hand here. We've got some more HP people in here and I'll introduce them in a minute. It was mainly to be not testing because number one, it's an overloaded word. Number two, it's too much of an implication that if there's a problem with this, you go to the Debian community. No, if there's a problem, you go to us and we will learn how to work with the Debian community to resolve those problems. So we needed that different nomenclature. We do these regular roles. We always save the old packages, create these archives, generate new indices and just kind of roll our top of tree through time and keep the roll points and make them available to the development teams. When you start looking at managers and schedules and paying customers, their excitement about continuous integration and continuous deployment starts falling off. They want to see the June 2014 release. They want a frozen point in time. So we've come up with a mechanism to bridge the gap between CICD, which developers love, and frozen points in time, which customers love. And I'll talk about that in just a little bit also. But these frozen points also allow us to tell the different development teams, choose what you want, choose what works for you. We do this every two weeks or so. You can find one in there that will make you happy and will work with you on how to upgrade individual packages as needed and keep that flow for you as a legacy archive. Mention testing a couple of times. Two of the gentlemen here, Bill and Cameron, who are here all week and catch their act at the Marriott City Center, are working on our testing framework. The idea is that we have a framework which is independent of any other product that really supports remote execution. Because internally, I have no idea what that said. I hope it wasn't kick me. Internally, we have a lot of hardware partners that want to do qualification testing on HLinux. But we don't want to have 30 pallets of equipment showing up to the Fort Collins, Colorado site all the time and run all that. So we want to be able to drive these remote tests with our help. And we can test hardware remotely to do qualification cycles so that we can fulfill the stack. So what TARDIS does is support that remote execution. And we're in the early days of being able to do things with it. The remote execution works. The framework works. It can install and deploy on bare metal virtual machines, Linux containers, any type of abstracted hardware that's targeted to run HLinux in a production setting. Do tests, record tests, conditional tests, all the type of stuff you'd expect from a test framework, but in a very controlled fashion with remote execution. We're looking long term at how do we integrate the things we find in HLinux back with Debian and the bug tracking and the systems there. So long term, we have lots of goals for this. Short term, we're just trying to get it running just like we did with the repo. Finally, there's policies because now that we have this up and running and people are using it, they say, well, when you're going to roll it and what kernel's in it and how do I get this in and where do I go with this bug and I've got this favorite package that I've got to have in and all the types of things that come along with owning a repo. We have the same headaches, hopefully on a much smaller scale but I'm preaching to acquire here. Sure, we would love to have consensus and have everyone arrive at a nice comfortable decision. We don't have the time for a consensus to develop all the time. Sometimes somebody just has to make a damn decision. We make the decision and for the most part, everyone gets along with it pretty well in a very short amount of time. I know this is starting to eat into the official release time for dinner. I've got a few more things on a little deeper details but are there any questions, clarification questions on what I've gone over so far? Because I'm ready to keep going if we're all good here. Just wanted to make sure, yes. The question is, what are we using to manage all of our repos at the different stages through that pipeline? We're using apt mirror to pull in from the original Debian testing mirror that we have on site and from there on, it's custom code, special sauce that does a lot of the control flow like the rebranding, opening up the packages and doing things. That's stuff we've used but mostly it's gluing together standard Debian tools like when we finally get to generating the indices in the archive and all that's FTP archive and scan packages and all those standard tools. So the true grunt work that produces the Debian compatible stuff is Debian tool sets but the orchestration is custom. Any other questions right now? Yes. As an outsider, I have the sense that it would be helpful I'm making a proposal that a group of folks could meet with you, Rocky, to talk about what giving back to Debian could look like in terms of a workflow. We would love to have many meetings like that, yes. So just on that, on Saturday, there's the derivatives on Saturday, there's the derivatives on Saturday, there's the derivatives buff. I would encourage you to come to that and meet the other derivatives and Debian people working on derivatives stuff. I think we have someone who's planning to attend that. All right, are we good? Excuse me. I mentioned that we maintained archives of every release role that we do. So over here on the left, I don't know if you can read it exactly or not, our time stamped top level views of a repo with the disks in a pool and two weeks later, another disk in a pool and two weeks later, another disk in a pool, that's about 120 gigabytes of pop and it adds up fast. So we really do want to keep this stuff forever. Ten years from now, we do want you to be able to get the March 2014 release. We don't know why. It seems like a good idea though and we make disks. Still, we don't make that many disks. If we had done this all the way up from March until now, we'd be well over two terabytes if we did that. So we came up with a way to do data deduplication. Essentially, we copy all the pools into an ocean. Common directory, because when revisions change, file names change. So that original one that was laid down in March can stay there forever and if that was the only package that changed then that's the only file that's going to get added to our ocean. The rate we do the repo rolls, we get about a thousand packages changing or added, essentially. Each time we do this. So we have this giant ocean of all packages ever seen organized the normal way and this idea of a package management database is more of a concept than an actual database at the moment. We're in the process of building out and flushing it out, but its basic function is to map file names and revisions to the actual file and where it's located in the ocean. Once we have that information I've got my ocean over here like I had on the other page and this mapping database. I can take something called a playlist which is just a list of packages and their revisions. It's a very, very simple idea. It's a text file. It can exist in several different forms, but I can take that and generate a custom repo. We call them hot rods. Hot repos on demand. Because knowing what the files and revisions are that you want and where they are in the ocean, I can quickly construct a pool directory that's made out of sim links and not the actual files. Once I have that pool directory I can run the standard index generation tools and now I have this repo representing whatever the playlist was to start with. For the archives it's the full 65,000 packages. For our group that does storage testing it's the 100 or so packages that they need to bring up a Linux system that will execute against the storage arrays. And literally have nothing to do with OpenStack, just enough drivers and smarts to generate that. We can archive these 600 packages into custom repos every time the Helion group turns its thing. Doing it this way, that two terabytes for all those things is now down to about 220 gigabytes. We only add a gigabyte or two every time we roll the repo. That's something that's manageable over the course of decades. That's one of the things that we can do with this playlist technology. We've got lots of benefits that we can do with playlists once we start representing entire repos as our unit of granularity or the thing that we're working with. We can do playlist manipulations, playlist math, playlist comparisons, playlist aging, playlist reductions and generate an awful lot of these benefits that you see here. I'm not going to go through all of them in this presentation but we have some of them going. What this really boils down to is we take the 65,000 packages we're getting about a thousand updates a week coming through Debian testing. We're doing a couple repo roles a month do the math, carry the three, whatever you want. We have a lot of data. We have some big data opportunities if we start thinking of the repo at the repo level. That's our minimum unit of granularity. This is the way that we're looking at a repo. These are the topics, areas, I can't remember the exact word, that packages are classified within Debian. Networking databases, there's about 20 of these. For all practical purposes we have added another virtual group to this collection called Helion. Those 600 packages that we care about for delivering our Helion product yes it draws from these other packages. Networking and databases obviously we don't pull anything from graphics I imagine but you get the idea for all practical purposes we've added another group to this giant thing called Debian repo. I want to turn it on its side a little bit and look at edge on and then look at more of them across time. This is a three dimensional representation I hope you see 3D in there where the major axis that we care about is time. Yes there's other axes and maybe there is some way to assign meaning to those that's a little over my head for data science right now. But when we have all of this data and we have this playlist concept and databases to support it we can start doing calculations and tracking predictions who knows what. This is a big data opportunity that we haven't fully explored yet. We think it's going to be very very useful to us in understanding the supportability and the maintainability of Helion based on H Linux. And that's where we're planning to go with this. We've already started doing a couple of things like that in terms of how we track stuff for example we took one custom repo I think this was one of those storage group sub repos of about 100 packages to to drive storage unit and look at the package volatility over time because we have all this data that we've snapshot it and stored. We can tell things like this there's useful information in there. Another way to look at it is if somebody gave us an arbitrary random list of 50 packages where is the closest repo that we have in history that would reproduce that set of packages from a unified repo. Now given the playlist we can spread it out over repos because it's just a playlist it's just a custom repo we don't really care where it comes from or where the packages are resolved. We can start playing games like this right now which we think opens up an awful lot of opportunities. Kind of summarize all of that H Linux is stuck in the middle between two very different rates of progress innovation. Velocities of product release if you will. OpenStack is moving at a very very fast pace a very very quick clip for how they do point releases it's almost weekly where they do a major point release of the complete thing. However the hardware on the bottom does not operate on a weekly schedule it operates more on a semi-annual schedule. We're caught in the middle we've got to balance those different velocities in the things that we release in a repo. Once we have all that data we can do forensics and analysis and predictions and regressions and all kinds of fun stuff that we really haven't even thought of yet we're still developing the data collection and the design of experiments phase. One obvious thing is to provide these legacy repos we don't know how long we'll keep them going but we are planning to be able to keep them going for decades, yes. Ah, I have to step in you know that most of your problems are already solved in Debian Are you aware of Snapshot Debianhawk? We are aware of those things we had some trouble getting a few of the things to run with some of the other constraints that we had doing but with that schedule that we were given but that doesn't mean that we need to keep doing it the way we started and collaboration with the community could actually mean throwing out some of the stuff we've done and taking what's already there if we can figure out how to get it to work for our needs. You should talk to Peter. He's always happy to help with Snapshot related tasks and nearly all of your problems are solved in Snapshot. That's good to know. That's one of the reasons we're here because we know we proceeded a little bit blindly or naively with good intent and we're hoping to find better ways to do things and give back and maybe do it at the same time. So thank you. The big data thing I've talked about a little bit and explained so I don't think I need to go into that a little bit. We really are here to generate a new level of collaboration and work with the Debian community. We need it. We need it for our long term. I say the phrase with a straight face, we really are here to help. We're here to play. We're here to be for the long term. We can't turn off the cloud. Sure, we could reverse our decision on choosing Debian testing but it's turning out to be a really, really good decision so far and over the course of the next year it's going to get set in stone and we're going to be chasing this for a long, long time. So we're not here to second guess ourselves. We want to hear feedback. We want to hear criticisms. We want to hear, did you think of this or why didn't you do that? We need the experts. The things we're learning, the things we're trying, the things we want to do, we think will not only help us but we think it's going to be of general interest to the Debian community. We want to understand how we can give that back. Is it just the results? Is it our techniques? Is it our code? We don't really know where that's going to go. We are looking for help and specialization in a couple of different areas. One is the general performance characterization and capture and tuning of a Linux system especially based on Debian. Repel management, obviously that's going to be our problem, our bugaboo for long after I've retired so I want to leave it in good hands for somebody else. How do we deal with bugs? How do we track them? How do we generate them? How do we integrate them? How do we help fix them? How do we foster Debian maintainers and developers within HP? These new areas, data scientists and up and coming area, we need some. We could use some. The collaboration opportunities are all over the map including maybe we sponsor interns to help take care of some of the things that we need directly for H-Linux or that Debian needs in general to be more sustainable and more viable in the future. It's in our interests to make sure Debian thrives. If you're already working for somebody else and you're happy with it, great. If you're looking for some other arrangement, great. We're here all week. And we know you've got ideas that we've got to hear. So introduce Bill and Cameron already. Also here is our director Steve Gehry, who is in charge of all things Linux at HP. Can that be true for a day or two? Okay, it's at least true for a day or two. It'll absolutely be true at least through tomorrow night where we're hosting a happy hour, kind of a late happy hour after the final evening session down at the Marriott City Center second floor, they've got the Den, dining entertainment and networking. So come down, have a little more late food, have some drinks. Talk to us about anything because we want to talk to you. We've got a lot to learn and we think we have a lot to offer. Are there any other questions? So you're rolling a new release every few weeks from importing packages from testing. Presumably because you want fresh packages. What do you do when Debian freezes testing? We'll let you know as soon as you do that while we're doing this. I can say we've been doing this since March April with real intent, I guess you might say. So yes, there's lots of pitfalls and potholes waiting for us and we'll cross them when we come up to them. Well now we know who to ask. Get his name. I was wondering if you have a similar question about the use of testing. Why did you use testing in the stable plus some specific backpots? Stable was not new enough in the opinion of the open stack developers. They wanted us to use unstable and I wasn't quite that brave. There was a little too much flux going on in that especially in some of the packages that looked like they were intended for open stack. I'm quite surprised about that because I thought that open stack upstream was using the Ubuntu LTS releases as a recommended platform. Open stack upstream is not a multi-billion dollar company trying to do good customer legacy support. We definitely want to support them but that was one of the trade-offs that we made. I was wondering about the integration testing and integration system. Do you intend to publish some code or at least some documentation about how you do it? I'm going to turn that over to the testing guys. So basically what we have in place right now is an alpha release of our test framework and it really allows us to independently drive almost any kind of test or any type of localized or external to that test framework ability to test outside of that. I think long-term we'll see that come back into the open-source community. We haven't had those discussions yet but I'll call it the edict within the company is that I shall contribute everything back to open-source. If you want to say something about that, Steve. Yeah, so long-term could be very short-term. Right, so again we've only been around a few months and the idea is indeed to put everything out in open-source and at this point in time by the end of the week if you can chat with Bill and tell them why that's a good idea. Oh, okay. Yeah, right there. So chat with Bill tell them why that's a good idea or not and how it might be used. We're certainly open to it. Long-term we would like to pick this entire machine up out of HP and put it all as open-source. We don't have the edict, we don't have quite the permission for it. We're trying not to do anything now that would prevent that in the long or short run, as Steve says. You did mention that you are doing copyright reviews of the packages. Are you getting feedback from lawyers or just from technical people? Yes, we have an entire... He seems motivated to go after that, so... No. So the OSRB process that Rocky referred to actually has across discipline review board. So there's different R&D technical people and there is a legal community that's looking at them as well. So we get plenty of feedback and we'll be more than happy to share that feedback with you as part of what we give back if there's interest. Okay, there's interest. Yes, there is. And some of that, the phosology effort which examines source is all open-source itself and we can find out more information on that for you if you're interested. Another question, how do you handle security support? Or do you have... I mean, do you have any specific support or special needs for security? Yes. Regarding software vulnerabilities. We don't break out a separate security repo. Some of the problems with... I'm trying to figure out where to find the answer to this question. Building with triple O and disk image builder, it is limited to a single source repo. There can't be multiple sources.list entries or alternate places to get packages. It all has to come from one fat repo. So that's why we've gone after this mashup concept where it comes from different places and we present it as one repo. So are you looking for how we address security issues or as a discipline? Many security issues. Many security issues. Along with the OSRB review we are doing a security review. Especially once we know these 600 core packages that are of critical interest to HPE Leon. Just like we have OSRB experts looking at the legal side of packages and licenses, we have security expert on the team who works with other people within the company to track all of the security notifications and the alerts that come from the rest of the world and compare those against the packages that we're pushing through. But are you actively trying to let's say you find out that you are vulnerable or a given package that you are using is vulnerable. Are you actively working on fixing them? And if so do you have some kind of involvement in the community? That's what we're trying to foster. At HPE we are the first point of contact for support, for bugs, for security violations or alerts. It starts with us. Over time we will develop more and more expertise to be able to handle more of the critical problems ourselves. However we can't do everything. We're not ever going to be that big. We are going to rely heavily on the community to do it. I think a really good example is Heartbleed. We picked up all the notifications on that and told all of our internal partners, but by the time we got that kind of announcement out you guys had already proposed patches and fixes and the chain of events was there so that we managed to offer a two-day turnaround inside HPE for that fix. We had it posted on one of our new repos before half of our development group even knew there was a problem. So we'll fix what we can. We'll contribute where we can. We rely on community involvement to figure out how to help you guys do that. Speaking as a member of the security team I'd like to we would like to know and have a greater collaboration because we know that there are several companies that use Debian and we think they provide security support like in this case but we don't know and we've never heard about them. I mean directly. We've never received an email and for some cases like in SqueezeLTS it is until everything is set up that we receive one email like yeah, we might be interested but that's about it. So please if you're interested in getting in touch with us at least we would like to collaborate. Absolutely. We have people in place, we have a plan in place they're skilled and experienced in this and we would love to have you tell you just how smart we think we really are. All right, we've reached a collective oh, one more, gotta have one more here. This guy's what's between you and dinner, right? So for software that doesn't come from Debian like Kernel and your own OpenStack branch to use Debian packages as a format to distribute software or Yes. One more invite for tomorrow night 8.30 at the Marriott City Center it's about 10 blocks straight down Broadway, 8 to 12 depending on who you ask very short walk, there's transit come by and see us, come by and let us say thank you and spend some time come by and tell us what we need to be focusing on and what we're missing. That's exactly, exactly and bring I don't know markers or something so that we walk out of there with good information. Just to follow up to Luca's question, are you making Debian source packages as well or Debian binary packages? Yes, we're still getting good at Kernel source getting those packages in after we've put our custom configurations in but we're providing source packages for all of the Debian packages where it exists, some of the foreign packages source does not build or comes from other companies that we're working on the Precona database stuff if any of you are familiar with that they take and muck with MySQL and do some stuff to it and push it on and that's what OpenStack needs that one's being a little bit sticky but we're working through it. And do you plan to publish the repository at some point? Yes, with the focus on at some point. But those 600 packages that we're using, that's we can completely share that. Essentially it is shared when you do an installation of Healy on an OpenStack, the packages are there so the mental hurdle against opening those up in a pure repo has really been breached. There just are some of the logistical and legal things about setting it up that we've got to work through. If you guys have got ideas about how we can be more open and share and be more transparent about what we're doing we'd love to hear about it and then we'll wiggle through whatever we have to from our side. Great, thank you very much for your attendance, really appreciate it.


The first type of interview questions at TikTok is behavioral interview questions. For example, why TikTok? Why TikTok Live? Why TikTok Shopping? Question number two, what's your strengths and weakness? Question number three, tell me about yourself. This is a mass-asked interview questions in all type of companies out there. Question number four, what's your past experience in ABCD company? Question number five, tell me a time when you failed. Those are typical behavioral and leadership interview questions, but lots of people, if you're not able to pass a first round interview at TikTok, it's likely you failed those behavioral interview questions.


At Audiology Associates at Prestonsburg, you can live your life the way you want and find the freedom of better hearing. You'll experience patient care that is specific to you with exceptional follow-up care that ensures your hearing and balance needs are being met. Audiology Associates at 1428 Northlake Drive in Prestonsburg. The Appalachian Center for the Arts will be hosting their first round of acting classes beginning February 16th. Classes will be available for ages 12 and up, and beginner classes will be offered. We have a couple of improv classes coming up, improv for adults, improv for teens. We've got an audition class and an acting class just for preparation for people who are wanting to audition further, wanting to learn how to audition, wanting to learn how to be a part of the process. And then what happens when you're in the room? We have some acting classes for that as well. Once we drum up enough support, we're going to have some master classes throughout this year just to kind of get people interested, get people involved. Acting, improv, audition and combat classes will be offered and classes will meet two to three times a week for ten weeks. We at the App really want to be a part of the community and we want to help people in the community. As important as the children are, the adults in the community are also still a part of the community and they might have desires to act, to express themselves in different ways and we want to offer a place where they can do that. Tuition for the ten week program is $200 and includes a free ticket to a show at the center. For more information, visit the Appalachian Center for the Arts on Facebook. Reporting for Mountain Top News, I'm Brianna Robinson.


The Ministry of Tourism and the St. Usher Tourism Authority, the SLTA, have launched Collezion d'Opérpite, the brand that will embody and fortify the marketing capacity of St. Usher's small and boutique property sub-sector in the regional and international source markets. The bespoke brand was unveiled on March 16, 2022, during a special ceremony under the auspices of Tourism Minister Honourable Dr Ernest Helaire in the presence of stakeholders and partners of the tourism community. The name Collezion d'Opérpite originates from the island's French heritage and means a collection of small pieces or nuggets of gold, or something rather special to you. The community tourism is about finding the right solution, authentic experience. It's really going to focus on how we can sell to the world authentic solution experiences in the various communities, the basket weaver, the bar, the restaurant, the accommodation. Whatever it is that speaks of an authentic solution experience, it will fall under the Community Tourism Bill. And therefore, when you add this, you can now understand the range of options St. Usher will be presenting. The CEO of the St. Usher Tourism Authority, Lorraine Charles St. Jules, noted that the St. Usher Tourism Authority is enthusiastic about this market-ready product that will further appetise travellers from the island's source markets. As we go post-COVID into the pandemic, the clients from all global markets are looking for something different, something special, something out of the ordinary. So this collection, rebranded and coming together as 400 unique properties, offers someone who's looking for luxury, you're looking for wellness, you're looking for chill relaxation. And I believe that products that we are launching today captures all these elements for the global market. In embracing the opportunity and value in the partnership, the commemorative plaque was presented to the President of the Vacation Rental Association, Thomas Leos. We have a city Oasis in Arundel Hill in Marsha, a super host, which means they have met top quality standards consistently for several months. That's Arundel Hill in Marsha, maybe surprising to some of you. We have Michael's Place in Balata, we have Sanipam Villa in Labry, which was here tonight. So the collection does not also only speak to the luxury properties, but quite a range spread out over the entire island. Admittedly, the concentration of properties is in Grosjele, Souffre and Labry. But I think with the advent of Commodity Tourism, which the Minister spoke to earlier on, we would like to see the spread of the accommodation sector throughout the length and breadth of St. Lucia. All participating properties have been accredited by the Ministry of Tourism and the St. Lucia Tourism Authority to provide an added layer of reassurance. Senior Tourism Officer in the Ministry, Deepa Gadari, explains. Our standards for this new brand were developed through the revision of entry requirements developed under the Caribbean Experiences brand, which was part of an OAS Step program and the CrossQ approved national minimum standards for accommodation. To participate in this new program, small properties, meaning those with under 35 rooms, will be required to be COVID-19 compliant in accordance with the COVID-19 Act No. 74 of 2020 and successfully complete an on-board survey which assesses the prospective applicant against the general entry requirements outlined in the St. Lucia National Standards No. 22 of 2012, Part 1, Specification for Tourist Accommodation. For more information on collection de propit, visit stlucia.org. From the Government Information Service, Lisa Joseph reporting.


My name is Ferdinand Trover and I'm a researcher, security researcher to be more precise. I used to work in the game industry and I took a look at the general state of security in the game industry and that's the presentation you're gonna see today. It's pretty scary what's out there. I'll take questions at the end, so if you've got anything that's pressing, please write it down, remember it at the end. Okay, let's get started. I want to give you a quick idea of how this talk is gonna be structured. First of all, I'm gonna talk about the history of games which explains kind of the mess we are in. Then I'm gonna give you a quick idea of what the gamers out there are thinking and if you're a gamer yourself, you probably have a slight idea of what's going on there. Just out of curiosity, who plays games in here? That's what I expected. Yeah, who doesn't? Anybody? Then I'm gonna talk a bit about the developers that actually make games because it's interesting to go a bit into their mindset and look at the specific challenges that they actually have compared to a regular developer. It's not too vastly different, but it's interesting to look at. The next step is what is crystallizing to become game engines. Think of it like the backbone of the game that in the future will be shared between multiple games for mostly financial reasons. Then obviously, once you see how bad this is, the question is what do we actually get from it? Profit. And then I'm gonna take a quick look at the malware that's out there which is quite interesting and very surprising.  You'll see what that is in a bit. And then I'm gonna go into an exercise in exploits. The thing to note about here, I kind of like neutered this because I didn't want to go into anything that you could really replicate out there because it will just get me in trouble. Sorry. Okay, the clarifications before I go into the talk is I'll be talking about PC games. So if you have been thinking about your console, sorry, I tried to scope this down because there's so much to talk about and I cannot touch everything. In the spirit of this, I'm also gonna look at non-casual games mostly. This is like the games that you actually buy at EB that you pre-order that you pay premium price for. Think of your call of duty, your crisis, something along that line. I'll also not be talking specifically about web-based games. There is a lot of things in common here and they're kind of merging together but so far I'm gonna stay with the client view on the PC. I also won't be genre specific so I will look into shooters, MMOs, what have you. For the MMOs themselves, I'll be looking at the client only for now. There's a whole slew of problems that you get with the server if you actually just think about your own client that you attack the server with. Since this is not directly exposed to us, it's more interesting to think around on the client for now. Here you see the historical development of games in a very visual aspect between the late 1990s and 2008. Does anybody have an idea of what the left game is? Anybody? Yes? And the right game? Perfect. So historically, games in the late 1990s have not been mainstream. Nowadays they actually are. If you have been paying attention to the news coverage for GTA IV, you'll actually note that the first review for GTA IV was in the New York Times. This is pretty indicative of where the whole industry is moving. In the 1990s, you used to have nerds play their own niche video games, which I kind of miss now. And now it's just this mass market phenomenon. This also explains why the costs of games or to produce games have exploded. You used to usually get a game between like two to three million dollars and I'm talking about the really popular games out there that are selling a lot. And nowadays you have to almost tenfold that. I mean, you're talking about 15, 20, sometimes even 40 million dollars. The development time on the other hand has not increased significantly. Gamers still want their newest iteration of their game to be released roughly in a one to three year cycle. And this is actually one of the problems already because you see there's more pressure on the development team. You need to reuse more code. And in general, you just don't have so much time to pay to engineering and proper security practices, for example. This also ties in with the publishing that you have and the publishers for the people that don't know. That's usually a company that takes the games from the developers, pays them to develop the games, will give them a cut of the profit and does everything like the marketing and so on. Nowadays, you pretty much have three major publishers and the biggest one is EA. You, because of those three publishers, you have a lot of homogenization, homogenization. Street give out. Which pretty much tries to toss everything into one bucket and you'll see more of that later on. Earlier you had a lot of custom graphic solutions. Every game came pretty much with their own engine and the only thing you shared was the low level graphic stuff that you still have to share now. Nowadays, you are pretty much limited to a set number of engines. Either, you usually have two choices. You'll either reuse the engine that you've used in the previous game because most of the games nowadays have sequels. So for example, this would be, if I use a name, Splinter Cell. Splinter Cell has used the same engine for all installments. They've upgraded it, but usually they've used the same engine or derivatives of the same engine. The other engine type that you have is you just have an engine that is something that is shared between multiple games. So for example, you have the Unreal Engine that is shared between multiple games that do not have something in common. The number of those games, the number of those engines is actually dropping significantly and that's due to the fact that it is very expensive to build an engine and people want to share it. The problem with this is obviously, one exploit on one game binary, more specifically on one engine, will cover multiple games. This is just the fact of life if you use the same engine. Middleware and for the people that don't know who, what middleware is specifically, it's usually something that you drop on top of your engine or in general on top of your game that provides extra functionality that you can't create yourself. It's typically AI, physics or something along those lines. Used to be not that prevalent back in the 90s. Nowadays, you have to have it in every game and that's partially also due to the gamer. We have like an increased sensibility about we want to have neat physics in games, we want to have awesome AI and it's the game developers simply don't have time to do this. Yet again, if you use the same middleware across multiple games, if you identify the binaries, you can just attack multiple games with one simple exploit. Now looking at the distribution of games. Back in the 90s, you used the physical media pretty much. That's like 99%. Nowadays, you see the advent of digital distribution of full games. It's a bit tricky because they're big, so downloads are long, but it's mostly due to piracy, which I'll talk to about in a bit. You have two big delivery mechanisms right now that are pushing really for the market and that's games for Windows Live and Steam. Steam being significantly bigger at this point.  But they also allow you to go into something that's bigger than the game itself like a community and that's what I call games 2.0, sort of like web 2.0. The thing that I don't like about this is you have content protection on top of those platforms which is an artificial restrictions, which gets you into all kinds of trouble. And you have automatic patching, which is awesome for the noobs out there, but if you actually realize what automatic patching is, then that can get you into big trouble once you actually figure out some way of exploiting automatic patching because then you have got an awesome delivery mechanism to just push your exploits out. Hand in hand with this, you have back in the 90s offline games.  You had your Altima online, your first step in that direction.  Nowadays, online is default. Pretty much every game ships with an online aspect. If you don't have online, it's not a real game anymore. And MMOs are huge, hands down. Like for example, wow. The problem with this is if you're online all the time, everybody can track what you're doing. You just need to have a packet sniffer. You can see when they're connecting to, for example, the game platform. You can see when they're connecting to the Vow server. You can see when they're just doing anything over the network. You don't even need to break the encryption on it or the protection on it. You can just develop usage patterns out of sniffing the network. Looking at custom content, which is also one of the problems that it's good and it's bad. So I don't wanna speak against custom content here, but the problem with custom content is that it opens up this whole new attack vector. Back in the 90s, you had a very limited group of people that constructed custom content and it was for editors that were put on top of the game as a pure bonus. You needed to have a significant expertise, actually getting those to even run and work correctly. While on the other hand, nowadays games pride themselves on having an editor. They're games out there that are virtually just a sandbox that allow you to just create anything and share it with the community. I won't say any names here because then people will get offended. But in general, the problem here is that you take content, you integrate it into the game and you have this awesome delivery mechanism that if you figure out how to exploit content, you can just push it out for free. You just put it onto your game world and it will be pushed to all other people. It's also interesting to note that you have XMLification of content, which means that everything that is bad or good about XML comes with that. And since game developers love to have the quickest algorithms out there, they will be probably tempted to write their own XML passes, which we know is not necessarily a good thing. But in general, the biggest problem here is the automatic polling. If you give people the option to download it manually, which is an inconvenience, then you have at least some level of protection there.  The same applies for community. Back in the 90s, you had a community that was built purely around web. You had a couple of portals opening like Battle.net, for example, but it was mostly web-based stuff. Nowadays, you have this gaming platforms and they pride themselves on giving you community. Actually, for some of them, it was more like an afterthought because they originally were created to protect the content. And here, it's very obvious. You don't even need to play a game to log on to this.  But underneath the hood, the game service actually connects up. And for example, for Steam, if you haven't been online for a week, you're locked out of all your games because it needs to verify it again, it needs to pull down the keys and so on. And again, I can see what you play in detail even. So what did not change over time? Game developers are still spending a lot of time to making it pretty, pretty sells. I mean, that's just the fact of life.  Security is not a major concern because my favorite quote here is like, it's just a game, right? What should we worry about security? And because of all the pressure that you have, you also need to use significantly more middleware in production of these games. The other problem here is also that games are not really stable, which lowers the bar kind of a bit for attacks because if you're a test manager crashes or your internet browser just goes down, you might be kind of suspicious. But to be honest, how many people have seen games crash constantly? It's not really that far fetched to have just a crash because it was poorly coded. Patching is because of this also common, either for stability or for new features. Usually most of the patches are for both, but it's not for security in most cases. Hacks, cracks and trainers are also still available pretty much throughout the network.  You type in a game and say no CD patch and you will usually find it. It's very hard not to find a game that doesn't have these. But while this might be something awesome that saves you from always putting the disk in, assuming that you have legally purchased it, do you really trust the guy that actually wrote this? Do you think the guy in Russia that provided you that kind of executable or that patch didn't put anything in there? Most gamers will just download it and be happy and never worry about the consequences. And this ties in with piracy. A lot of those online platforms like I talked about have been built not because there are new features, there's something cool for the gamer, but because companies wanted to protect their investment and wanted to have something where they can prevent you from actually going ahead and just ripping this from a disk, dropping a new disk patch on it and go. So a lot of those platforms were just built for the wrong reason. It's not pretty. Now let's take a look at the gamer and that wasn't a gamer by the way. Generally PC gamers are significantly more hardcore than your average console gamers. That's why I kind of like this market more than the console gamers. You actually need to have significant expertise to get a game running. This is decreasing over time and developers try to make it easier, but it's still true that you need to have knowledge of your operating system, of your configuration and so on. You also usually have significantly higher hardware than your base market out there because you need to have a decent graphics card, you need to have a decent hard drive speed, memory, you usually need to tinker with your OS. In general, the system is beefier. Gamers also tend to, or they really hardcore market I should say, tends to tinker around with the settings on their OS quite a lot. They will disable pretty much anything to get more frame rate. And this is usually all the security features and everything else on top of that. They will run it in the mode that will get them the highest performance, which in general is administrative mode. And I'll get to this in a bit, which is obviously not good. The other thing that is interesting about PCs from a security standpoint is, and that hasn't changed over time, it's only increased is that it's a multi-purpose system.  I mean, if you trust your PC enough, but that's what people out there do. You do it for shopping. There is a wealth of personal information up on your PC. And so it's not like a console where you just pop the disk in and if you own the console, you have like a box sitting out there. You have all that information that's stored there and you might even be able to siphon more information out of it. Speaking also to what I said earlier about the crashes, the gamers are generally not paranoid, especially if they tinker with their system, if they overclock their hardware. They're just used to having stuff blow up on them because they overclocked it because they disabled some service that the operating system desperately needs. And they're also used to frequent patching. They will usually drop custom content on there as long as it's pretty, like I've already said. And they really think what harm can it do because it's not executable, right? I'll just drop stuff on there. It won't harm my system. It's just a texture. Well, we'll get to that. Games also need to be run by default with the highest privileges. Luckily, I must say, this is changing. Back in the day, all games were just admin only and if you didn't run it as admin, they would break badly. And so once you own a game, you've got admin rights, everything is over. Nowadays, this is actually shifting and I must say thanks to Vista because you don't really want to play a game that halfway through pops a ULA dialog up and asks you, UAC, excuse me, dialog up and asks you for do you really want to do this? So it usually runs in limited mode which actually helps you for security quite a bit. Most gamers also spend more time online which just makes them a bigger attack surface because you are even forced to be online with those platforms. You don't even have a choice and that's the thing I personally don't like because you're stuck with the system if you like it or not. If you don't have an internet connection, you might be able to play your game for a week but in the worst case, you might not be able to play it at all. The other thing that's also true about gamers, if they have the choice between cool hardware or an increase in performance, they will take that over security. They will take something pretty and fancy. On the other side of the fence, you've got the game developers. They're pretty much like any other developer that means they still make mistakes like all humans do except if they would be cyborgs but yeah, we are not yet. They're also under severe time pressure. They have hard deadlines. They need to make Christmas most of the time or some other date that the game publisher usually has said or they have said because then they'll just run out of money. This is true for other software as well but here it's critical and it makes or breaks a game company to be able to do that. Most games still run late which makes you kind of worry because you have games that are under severe time pressure they can't do a lot of like in depth engineering and then they still run late. This also talks to the usage of canned code because the more canned code you use, the faster you will be able to push out a game but you can't really review that canned code. The game, excuse me, the developers also love the latest and greatest and I've doubted this the shiny complex and I've seen this over and over again. It's kind of amusing. You have the developer, they home in on something that they think is really awesome because it's this newest feature. They might even home on it like three weeks before they want to ship because they just saw this awesome thing that adds like sparkles to the main character that looks awesome and gamers will just drool over it and while it seems like a cool idea to like get that out there and push it to the gamer, it causes a slew of problems. It does not help with the schedule and the testing which it does doesn't make it more secure. Quick design usually means that you have quick exploits and quick coding usually means you have significantly more bugs. Moving on to the engine. In 2007, there were roughly 810 PC games out there and this is based on Wikipedia numbers so this is roughly accurate. Hopefully, if you trust Wikipedia. 42 of those games are considered major selling games and to give you an idea of what this means this would be like crisis like we've seen earlier, your call of duty, something along those lines. The results are actually interesting. Down on the right you have the graphics engines and 30 out of those 42 major selling games were reusing some kind of previous engines. In these 30, there are the three major engines that I talked about that are also used across games at the moment it hasn't crystallized out yet but you see the push more and more towards having a reduced number of engines there because it's not economically feasible to create your own engine. Only 12 games had a custom build of whatever engine that was used for one game only. Most likely if they get into a second iteration they'll reuse it. On the left, on the right side you've got the distribution platforms and this is kind of counter what I said before in terms of games being more and more digital but you need to understand here that about a year or two years ago this would have been 99% only media like disks. Nowadays that little chunk that is like 130 or 138 games of digital media is just exploding. I expect this to be by next year probably twice the number and the year after that even more. It's just huge and big and coming because both the developers and the publishers have realized that that's the best way to curb stomp piracy. And as you can also see here the biggest platform at the moment is Steam. Game engines usually look a bit like this. You've got the usual stuff that you would expect for game engines to have here like graphics, animations, physics, audio and so on. While you note here that physics usually is a third party it might also come with the engine. The other thing that's interesting here is the scripting language. You've got a scripting language in most games that allows you to do specific stuff and this might even be a C derivative or something that's very, very close to C underneath the hood that runs in a sandbox. Well, you might already guess the problem. No sandbox is perfect and this can cause huge issues if you manage to exploit this or just get full disk access with this. The things that are new here is in-game ad systems. For example, in-game ads are up and coming. It's still a question if they will stick around because if you run through the magic forest and suddenly you get an advertisement for Coke on a billboard it probably isn't that good.  So there's still some shake out to do but this is probably gonna stay around because it's just another revenue stream. The other thing that is definitely gonna stay around and is new is the online platform. And this is so wide at the bottom because this goes across the whole engine. This might even in the future be integrated into the engine. At the moment this is still third party so they add it onto a game but in general it's just something that really is gonna stick around. Another thing that's interesting about engines is that fixes by users and users in this case are the game developers. They are not shared because you usually patch games not engines. And you might have fixes that are even custom to the developer or the game. So even though they might have found and exploit in game A that they release, game B might not be able to fix it with the same patch and it's just too much effort. The other thing about game engines that's interesting is that the binaries are not hard to trace. You can just go ahead, go into your favorite game directory, go down, try to figure out where the physics engine is once you have got a name for it, click on the properties and you will have the file version there. This file version is usually indicative of very detailed information that you can get out of. So for example, if you have a file version 1.5 there and the next game has a file version of 1.5.1 then it's very likely that the exploit you just found in 1.5 is also applicable to 1.5.1. And sometimes game developers will just drop this onto their games so you don't even have to worry about different changes in the binary although they have the same version number. Like I've already said, shared engines provide easy exploits, middleware and so on. The thing that I want to add on here is that customization of the game engines that developers actually do for performance reasons might actually help you here because they could obfuscate one of some of the problems that you encounter. So the question you might ask is, can we patch the engine? Can we have something where we patch the underlying fundamental problem and then just be done with it? Historically you patch games, not engines. It might not be a bad idea in the future to look into some kind of mechanism to patch engines but that would require developers not to modify engines and at the moment I think they couldn't make the games as pretty or fast if they would just stick with a generic engine that works for everything out there. In some cases the engine developer or the game developer might be the same development studio so you have an advantage there but in general this is not rolled back into the overall business. So if you look at an engine that's used by two different game developers and published by an engine developer then a fix from one of them will usually not be pushed over to the second one of them. At the best the engine developer will actually get some kind of hint that there is something wrong and fix it in the next version. And again automatic packaging can be spoofed too. Well the question now is where's the profit? It's just a game, there's nothing to gain, right? Yes. Profit here is two-fold. You've got the game side and you've got the system side. On the game side the first two points might be pretty obvious depending on the game you're playing on. Griefing and cheating, that's usually not something that I don't really worry that much about.  But that is not that much of a problem in comparison to your payment information, your full personal information. All your virtual assets that you spend hours countless of hours creating in the game or even that you spend real money on buying in the game. Think of something like Second Life. On the system side you have pretty much everything you can gain from hacking a regular system. And like I already mentioned a bit earlier is systems of gamers are usually significantly beefier. You have better hardware, they are better staging system. You have a broadband network connection. It's just a cool place to start out from and do more malicious stuff from. Virtual economies are the other thing that you can get huge profit on. At the moment there are about 14 plus million MMO gamers out there which is huge. And to give you an aspect here, I think 10 million of those actually involve the for craft. It's quite insane. The user base is a huge target for malware. I mean there's just tons of stuff out there. And those virtual economies obviously also create a significant stream of revenue. On one hand you have the stream of revenue that is directly to the game publishers and the game developers which is the monthly fees. But you also create a lot of money through game assets. I mean you had this whole discussion about one and a half years ago about taxing in game property. That's because there's so much value there. You spend time, you create wealth in the game that you can actually sell on eBay for example. And the other thing that you might spend money on is services that are rendered to you which is gold farming, auto leveling and stuff like that. And this revenue is in real money. It's not something that you can cash out on. This is especially true for in-game microdance actions. There are a lot of games in Korea actually. It's really popular there that you get free games. And then you actually pay for pretty much everything that you have in game with microdance actions. So you might be spending without noticing hundreds of dollars on a little game with fluffy pink balls, I don't know. There's some interesting stuff in Korea and MMOs if you're interested. Obviously this means that in virtual economies every exploit can get you quick gains. Stealing assets, selling assets that you have just acquired, leveraging players accounts, just creating games, creating assets in the games that you will sell later on. And obviously they're all important payment information. That's why usually developers of those games are really easy to react if there's anything wrong with their server or their service in general. But they usually are slower to react to actually fix something like exploits on the client that will just own your information. And this brings me directly to the current malware. The top of the line malware that is out there at the moment and the biggest chunk clearly is the account stealers. These are small little malware pieces that go ahead and will just try to grab the account information for MMOs and they usually span multiple MMOs. There's a lot of different stuff out there but there are eight big families. And the scary part here is that according to the Microsoft Malware Protection Center and if you don't know who those guys are those are the guys from Windows Defender that little thing that chips with Windows by default have identified in June alone 2.5 million systems out there which is 18% of the user base of MMO players that had been infected with account stealers. It is unknown how many accounts have been stolen from those because sometimes these accounts are just leveraged to post something to a message board or something like this. But this is a huge number and there are indications that for July this number might be twice as much and that's just the stuff we know about. That's not the stuff that we haven't noticed yet because it's not covered in those eight families or it's very well hidden. The propagation of this type of malware is actually not that complex and not that tricky. It's mostly done through the community, through community sites, through social engineering, through what you usually would expect from malware. There's nothing that pushes it automatically to your system in most cases. It's pretty straightforward. This is pretty much the state we are in right now with malware. Now moving on to games 2.0 which is something that's very dear to me because I like privacy. Games 2.0 to repeat are the games that are fully integrated with online platforms for example MMOs or games that run on anything like Steam or games for Windows Live. The privacy concerns here are mostly that the system knows when the game is run and it is required to unlock the game which is purely something artificial that is done because there are too many people out there that will just pirate the game otherwise. The online status of a game is usually published and you might be able to deactivate this but underneath the hood the system always knows if you're online because otherwise you can play it. It might be even underneemers underneath the hood but you can still sniff the network and find out when people connect to the service and voila, you've got a full nice user profile. The other game, the other part of this that's interesting is that all of those systems or most of the systems actually keep some kind of track of what you have done in the game like achievements or something like this. This allows you to create a nice user profile sometimes even with time slices. So if you have discovered that little kid that just pranked you earlier, you could look into his user profile, figure out when he's not at home and torch his house or something like that. I don't recommend you do that. Also a list of friends is very easy to obtain. That might not be real life friends but you might be able to with this list of friends relatively easy socially engineer somebody and drop even more malware on top of him. Basically you get everything from these platforms that you would expect to get from a social platform like Facebook and you get significantly more too. It's hard to get or almost impossible to get around this. You have to do this and that's the thing that bugs me. If you want to play those games, you have to be part of this platform. You cannot opt out of it unless you say I'm not playing this game. Well for a social platform you always have the option to opt out. Now to something that should be pretty interesting. Exercise and exploits. I've got three examples here that are not necessarily something that you could do right away but it should give you a rough idea of how the biggest exploits are in the games right now. And I'll start off with the little nude patch that could then go to the post and then give you an ad from hell. Well this is an engine exploit or a scenario for an engine exploit. So the idea here is that you've got Alice who plays this game and she's not happy with the scene on the right for purely academic reasons. And she goes up to nudepatchworld.com which by the way it doesn't exist right now so if somebody in the audience wants to start a business go ahead and finds a patch that has been provided by Bob. She looks because she actually knows about threats that could come through patches like this. She knows how to disassemble them and looks through them and she actually finds that there is a character file and a modification to her game settings that will reference that character file. And the character file only seems to contain a texture so no big deal, right? She drops it in there but actually as it turns out Bob just pwned her machine and Bob just pwned her machine and is enjoying her credit limit. So what happened? Looking deeper into this you had the character file but the interesting thing here is that there was script code in there because as I mentioned earlier engines also have script engines in there. And this script code is actually the problem. It's executed in the game with the game permissions and that's usually administrator even on new systems and it also is crafted so it gets executed before the model is loaded. And ideally if you want to be sneaky about this executed once and once only. Well, script code is run in the sandbox, right? So this shouldn't be a problem. Not necessarily. Because the script engines are highly complex and can cause severe issues if they're not engineered properly. It is just a matter of time until you find something that allows you to get access to something deeper down. So grab your favorite fuzzer, run it on the scripting engine and you will find something. Ideally this will just allow you to create a one-time hack but you might even be able to do something significantly more sinister. And once you're there, you own the box because you're usually already administrator. The next one is social engineering which is the biggest one out there like you've already seen. The scenario is that Alice is playing an MMO with roughly nine million users, no names here. And she goes to the message boards and browsers them quite a lot for information. There's a little flash tag in one of the posts that she is looking at and there's nothing specific going on. She responds to that post and suddenly Bob actually has all her account information and everything. And he also uses that account to make more posts. The problem here is very, very straightforward. Bob just exploited a flash vulnerability and this is actually critical because even if your game is secure, your website might not be. Or even if you have patched your game to be the safest thing on earth that nobody can hack even if they are totally lead and have awesome skills. You could still have a user out there that runs the crappiest browser on earth and gets pwned immediately if you just sneeze at the browser. And so, securing games is not everything. This is actually to some extent outside of the hand of the developer and that can be a very big problem to as some MMOs have experienced. And the last thing here, and I'm blazing through this in the sense of time, is a middleware exploit. The scenario is that Alice actually has her favorite game and it just got in-game ads, like you see in the screenshot. This is provided by middleware. Well, Bob uploads a custom image to the middleware and he might do some sneaky stuff so sneak it into the system, he might just purely upload it. But the interesting thing is that whenever Alice views the picture, she crashes and what happens underneath the hood is that Bob just pawned her machine. And by the way, this is true for everybody that watches that image. Well, what happened? And this is what you might have already expected. There's an image rendering flaw in the ad system and one flaw covers all gamers because it's an automatic delivery system. And the user can only prevent the attack by not playing the game. The biggest problem here is that one exploit can put the whole cloud at risk. You've got this automated delivery system and you also have to make sure on the back end side that nobody is able to either spoof this or upload stuff. And this is sometimes really critical because developers will not necessarily think immediately about their bad implications because if I've got enough money, I could just make up a bad company, submit something to them and pawn the whole cloud. By the time they figure it out, I'll be on the Cayman Islands and enjoying my new accounts. So in-game advertisement engines also have additional code in there that will tell you when, where, how and who watched pictures and that just increases their tech surface. This is pretty much the end of my talk so I'll open the floor to questions. Anybody? Go ahead. I haven't seen anything specific so the question was if anybody has done an analysis on the EA's ad system. I haven't seen anything specific for this but it would be very interesting to take a look at that and see how they're actually filtering their content which is to some extent not exposed to the regular use out there because you're just on the receiving end. Anybody else? Cool, thank you.