Today is Strobak, origin of the Burom people. In the backdrop of the sadistic and terroristic carnage that was recently perpetrated or committed on the plateau, where almost 200 human lives were brutally cut down, it is imperative to shine a synoptic historical touch on the plateau. The Burom are mainly found in the present day plateau state, where they are the majority tribe and part of southern Cardinal, they speak the Burom language and are part of the Niger-Kungo linguistic family. The Burom have been linked to the Nox civilization, a civilization that dates back between 200 BC to 1000 AD. They are believed to have migrated from as far away as Ethiopia, moving to Sudan, Chad, then to Niger, where they settled for a while before moving down to just plateau. They were recorded to be the first to settle on the just plateau. The majority of Buroms are Christians and most of them are farmers and hunters. The Burom have been a major activity and names in Burom reflect their origin heritage, names such as bot, meaning fog, top, meaning fish, suck, meaning toad. The Buroms were historically ruled in their villages by the Council of Elders, who were led by a chief priest called Gomkwit. The Council of Elders settled marital issues, the greatest concerning land and inheritance amongst other disputes. Around 1916, this tool of bong-gong jaws was established by the colonial office in conformity with the cost-saving, divide-and-rule British colonial administrative method, often used to monitor and control the activities of colonized people. Gola oba.


I'm looking at how bacteria can adapt and change over time to cause disease in people. Bacterial diseases are a cause of one in four deaths worldwide. This is in part due because they are capable of adapting, evolving and changing so much faster than what we can. So what I'm trying to look at is a model of bacterial evolution in the lab. I'm using a new model system, a new method of trying to figure out how bacteria can adapt and change. So I'm looking at a bacteria that is very similar to E. coli in people. It's a mouse version of E. coli and causes a very similar type of disease in mice as it does in people. I watch the spread of this mouse version of E. coli through mice. So they infect each other in a similar manner as what people infect other people. And so I'm watching this in a very controlled setting to find out how the bacteria are changing as they go through different animals. Then I'm trying to figure out how the bacteria has then changed over time. So if it's become more dangerous, less dangerous, is it better able to infect more people or is it less able to infect? There's all these sorts of questions that we really don't know the answer to yet. If we can figure out what's going to happen before it happens then hopefully we can figure out some methods of trying to stop it from happening, trying to stop these diseases from getting worse.


And I'm welcoming now the first speaker for our Java sessions, Mary Grigelski. So welcome, Mary, and show us what you prepared for us, the good stuff. Hello, Anna. Hello, everybody. Welcome to my talk. And thank you again to Red Hat, to everybody on your developer advocate team, Anna, Etzin, Burr, everybody. So thanks a lot. So I'm all excited today. So really, really thankful. So thank you. See you. The stage is yours. OK, thanks. Yeah, I'm going to share my screen now, folks. Thank you for your patience with me. Let me kind of, let me share my screen. I just got brought on, so things, OK. So I'm assuming everybody can see my screen now. So if you can't, please let me know. Let me kind of go to the chat too, just to make sure. I'm not sure which chat, but I guess maybe, yeah. OK, then there's also a stage. OK, great. Yeah, awesome. Thank you. Yeah, I think I'm switched to the correct chat now. So thank you, everybody. And I know I don't have like a tremendous amount of time to be talking on a topic. In fact, that's actually can be quite involving, right? It's not very simple if you have worked with it. So here we are. So my title is about exploring stateful microservices in the cloud-native world. So everybody, when you look at the word microservices, looking at the word cloud-native, it's always about stateless, it seems like. So I'm kind of going against the current, so to speak. It's talking about stateful. So I'm going to be a bit doing some introductory, like conceptual stuff too. So bear with me if you're already familiar with the subject matter, but I need to be aware of maybe not everybody come in on the same page. So I'm going to talk a bit too about some basic concepts. So yeah. So thank you, thank you. And again, I'm a developer advocate at IBM and very happy to be here today. So who am I? Again, I'm actually from the WebSphere Liberty team. And I'm actually more on the open source side. And in fact, primarily doing open liberty, micro-profile, Jakarta EE, all of these Java kind of wonderful open source Java tools that allow you to build like complex enterprise Java applications. So, and I've been with IBM for three years and I joined the WebSphere Liberty team about six months ago. And prior to that, I was with the, we have our developer ecosystem group that was focusing just on Java. So in fact, I was doing mostly reactive stuff. And today I'll touch a little bit on reactive because every time when we talk about microservices and all of these things and stateful and when it comes into play, then we kind of talk about reactive. That's dear to my heart. So I'll be talking a little bit about that too, just very little bit. So, and okay, so I have myself too, I have over 25 years of software development, engineering experience prior to becoming an advocate. I'm also the, and just to give you an idea too, it's just so you know, I also work for another company that actually was, I was on the app server development team at Sybase before two for seven years. So, and then I was then with various companies in Chicago that are more on the application level and doing like production delivery. So I understand very well, so to speak, right? Sometimes the pain and the triumphs of working in IT shop. So that's what it is. And then I'm also a very active community builder. I'm also the president of the Chicago Java user script. So I like to also invite you to join my junk, you know, if you can, you know, from anywhere, right now we are all digital. So, okay, so that's me. And so let me then start. So the topic is about stateful microservices. So you're wondering, well microservices, okay? Isn't it all about stateless, especially in the cloud native world, right? So that's the thing. But let me first start too, you know, if you're new to this topic because it's a bit less talked about I'd say, but yet it is actually such an important topic. And let me use a kind of fun kind of illustration, finding Nemo. So I think if you have little children, especially, or if you're into cartoons and animations, that's a great, you know, story too. So why am I using this? Because I want to illustrate stateful person stateless, what it means, you know, in computing. So that's kind of like to get into the cartoon part first, the fun part. So as you know, right, the story goes is that Nemo was lost, this little fish. And then his dad went about, you know, the bigger orange fish, Marlin went about looking for him. And then in meantime, Marlin ran into another fish, the blue one, Dory. Now Dory is a very forgetful fish, but yet she is kind of happy, go lucky, you know, and like that. So maybe you're getting the point now. Like when Dory sees as a school of fish, you know, or jellyfish coming in, she was like, oh, so pretty, let me jump on them. And Marlin said, wait, wait, wait, don't jellyfish can sting. So, as you can see, right, Dory represents the stateless computing side while Marlin is the staple side. And as you can also see, you know, there's advantages to both, but yet we live in a very, you know, kind of a stateful world, you know, so let's take a look. So computing life used to be simpler. Stateless computing, what is it, right? So essentially too, it's a communication protocol that doesn't retain any session information. So that means, you know, data travel, right, in your application run time, but the state of the data doesn't get recorded between transactions as it travels through because it's just more for, you know, doing something and I don't care, you know, all these contexts around it. I just want to get something done, kind of like that. So, and as a result too, the architecture design implementation too, it's much simpler. If you want to scale the systems, it's actually relatively straightforward. It's just a matter of spinning up another component or in a cloud native kind of environment that you have a pod or a working node or something, you just spin it up.  So, so as a result, right, it's a lot faster, scaling it is easier. Recoverable, recoverability too, also is easier. There's any failure, you need to spin up another component. It's just a matter of spinning it up. So it's kind of like a dory fish, you know, happy go lucky getting things done quicker. But realistically, we live in a stateful world. I mean, imagine the fact that if you forget the birthday of your loved ones, right? Especially if you're a significant other, if you forget it, what would they, you know, he and she respond, right? That's not a good thing. So as such, we live, you know, in a world that we need, you know, to keep track of things. We need to maintain that context. So stateful computing, then it's about, you know, communication protocol that actually would retain all session information. State of the data gets recorded at every step across all transactions along the way. And then as a result, the architecture, the implementation design, everything's more complex. Because you need to scale the system, it's harder. You know, you need to like add another component to it because you need to serve a small request. And you want to make sure that that component will get initialized to the same state as the rest of them, you know, of your components in the cluster. So it takes more time. So performance wise, right? It's also kind of not as efficient as a stateless way of doing things. And recoverability likewise, too. You need to recover from system failure. You need to make sure everything needs to get initialized to the same state. So everything's just take longer. But yet, you know, we can't avoid it. It is very important. So how do we do it in, you know, a cloud-naked world? But let's first take a look to statefulness in the cloudless states, meaning that before cloud, you know, how is state being done? So let's take a look. You know, in the client-server systems, right? And let's not go back to the very, very, very early days. But let's go back to, say, the 90s, right? Kind of like client-server kind of coming up. So stateful database systems on the server side. So as you can see, too. Database-style transactions are common, right?  You can do any, you know, any language, obviously. But the thing is that you let the database handle the transaction. If you do some kind of connection to the database, if you don't explicitly say it, then your database engine, you're letting it control your transaction, you know.  So if something, everything goes right, then essentially, too, the database will take care of committing your results. However, if there's problem, then it will roll back, right? So that's how, you know, the database can be done.    You can, we can have an IBM DB2, we can have Oracle, we can have SideBase, actually that's where I used to work for. So let's say, you know, there are situations in which that type of ecosystem happens, you can have multiple distributed database, and what do we do? Then there's also two-phase commit, you know, that's a protocol that's being used, and it is still being used today. And in fact, it's quite a robust type of protocol. So quickly, so what is two-phase commit? So essentially, it represents, you know, a commit protocol that will take two phases. So, and then it also requires a coordinator is called the two-PC coordinator. And so the coordinator, what he does, what about this coordinator does, what is the coordinator for the two-PC? Then let's say I have, using the same example, I have DB2, we have Oracle with SideBase, and so first phase, you know, it's like a voting thing. So then my coordinator will need to ask, you know, DB2, are you done? You know, what is your status?  Then it's basically tell the coordinator, good, I'm good. So now the coordinator get the result from DB2 and then move on to the Oracle. It's the same thing, right, ask, are you done? And if it's all good, collect the vote. Go into SideBase and then it's all good, collect the vote. Now the coordinator looks at all three, it's all good. So then now I kind of go back and say, you know, DB2 commit, Oracle commit, SideBase commit. Now that's actually obviously would work. And also it seems like it should, you know, be pretty good, you know, you're making use of kind of an orchestrator, a kind of thing, right, kind of not so much orchestrator as it is more like a coordinator, making sure things are good. However, that's the problem is that this 2PC protocol is that it's pretty, you know, kind of inefficient, right? Now the thing is also is that using 2PC, we're making some assumption that isn't correct, which is the network, right? Network, we are all assuming in distributed systems that the network is reliable, but in reality, network is very unreliable as we know, right? There can be latency, right? There's traffic issue or it's completely some problem. You can't even connect. So in that case, you know, the 2PC protocol coordinator could be hanging too, like all of a sudden I'm doing this, you know, I'm on another computer or another system and I'm asking, you know, DB2 asking all these things. If my network is down, then I can't, my 2PC basically... Mary, sorry to interrupt you. There seems to be some echo on your transmission. Can you check if you don't have another tab open with hopping just in case you're hearing yourself? You didn't see myself? Okay. I have it open but not connected to the audio. So is it better? Let me, I'm so sorry. Let me kind of take a look too. I thought I had already kind of... Yeah, there's some notable echo. So we're trying to grow from where it's coming from. Sure, I'm so sorry. Okay, so let me take a look. So over here, I don't have anything that's open on my other browser. In fact, I restarted my browser before I come in. You have some headphones or something? I have headphones too. Yes, I can do headphones too. That will be better. Is it in bad, all of a sudden? No, I think now it's bad, but it didn't have in the beginning. So it's growing. Okay, okay. There's some reason you said that. Okay. Okay, I'm so sorry. Joris, that's why we do it live. Yeah, sure. Okay, sorry about that. I never had this happen, but can you hear me okay now? Now it's even worse. Oh, really? Let me do this one thing. I'm so sorry. Okay. Is this better? Yes. Oh, okay. No echo. I think I know why because I'm doing aggregate device rather than using my short. I don't know what is going on. Maybe the, you know, the system may be having problem when I aggregate it. Okay, but that's good. That's perfect. Sorry about that folks. But thank you for your patience with me. Okay, let me then resume. Okay. Thank you. Thank you. I'm sorry. I wasn't paying attention to all the chat too. I saw something, but okay. Thanks. Thanks a lot. Okay. All right. Then let me kind of go back to two PC. So, okay. Two face commit. Let me be a little bit fast, but I think you get the idea that to two PC, there's a, there's a disadvantage to it because, you know, if everything is good, you know, your network is reliable, everything. It will be efficient. Maybe. But the thing is too, you can have some problems. So it's a very synchronous type of protocol. As you can see, my coordinator needs to ask this database, then get back to result, go to the next one. So in the meantime, the rest of your databases are waiting for instruction. So it can't do anything until the coordinator tells you what to do. So in that kind of a scenario is not very efficient, but the thing is too, it works, you know, and it is just something we need to be aware of. So that's two, two PC. And it is still being used today. Don't get me wrong. But we'll also examine into another approach of doing this type of distributed transaction kind of work. So in my couple of slides, I'll be talking about it. Okay. So I'll be a little quick. Okay. Then now let's look at Java, right? So how does Java handles it? Java, we have enterprise Java. Right now is Jakarta EE used to be Java Enterprise Edition or JEE. And then even before that was J2 EE, right? So, okay. JEE or Jakarta EE has EJB. So enterprise Java beans. And obviously there is session beans and then also entity beans. There's also stateless EJB. So we won't talk about that now. Then there's also session EJB. So session, what we'll do is that during a session, the data will be kept, you know, kind of the state will be kept only as long as the session is in place. However, if the session goes away, then basically you will lose your data. So the way to kind of handle it is basically, if you use entity beans, which actually, you know, kind of complicate things, obviously, then you have to kind of connect with your database or some sort of external storage, right? That kind of persists your data. So there will be entity beans. Now for HTTP connection, then there's also for surflets, there's HTTP session. So that's kind of responsible for keeping state of the data in your HTTP session. And then let's take a look then on client side. You know, client side, we have like caching of server responses. So there's also like cookie-based authentication. Cookie-based too, as we know, right? The cookie will carry itself, you know, the session ID to the server, basically present to the server and say, hey, this is my session ID. So server does all the validating. If it's all good, authenticate you. Then basically the cookie, right? The client will then go back to, you know, to your client side and then fetch the actual payload, you know, all of the stuff that needs to go through. Whereas then there's also the newer token base, authentication such as JSON web token. So that's more efficient because then in my, carry in my payload, it's also, I include the session ID and the payload. So you don't need to make extra trip. So that's kind of just example of how state, you know, statefulness is being preserved, you know, in a living system. So all right, let me, I kind of spend enough time on this. And then I had problems with the other thing. So let's kind of now take a look into stateful microservices in cloud native environments.  Now I won't go into all the details, but let's kind of touch it quickly, right? Cloud native, what is it? Right? It's an overarching approach. It's basically a term that came up, you know, if we go back to some history that came, came into place, basically coined by Netflix, because they were trying to, they were early adopters of all of these cloud native thing. So they call it cloud native because they want to leverage on the cloud infrastructure to do systems that can be highly available, very scalable and performant. Now all these three kind of goals to, is actually bring something dear to my heart, which is reactive. Now reactive is slightly different reactive, but which I'll talk about in a little bit, but anyway. So this is the goals of like in doing things, you know, the cloud native way, we need to be lightweight, a very efficient performance and all these things. So, and let's take a look too. There's also cloud native too is guided by the 12 factor application kind of principle, which is a methodology that was drafted by developers at Heroku. So essentially it's a guide set of guidelines and best practices for portable and resilient applications that are well suited to be in the cloud environments. And one of the factors indicates the need for self-contained services, which are to be deployed as stateless processes. And microservices architecture so far, right? It's the one that can satisfy this kind of a requirement. The nice thing about this is that it's such a set of guide, set of guidelines and it doesn't enforce the tools and the libraries that the applications must use, but it provides kind of the solid concepts that the applications must follow. So here just a kind of a quick listing that I kind of took it, you know, kind of summarizes all of these 12 factors. So I won't go into like exactly all of the details, but just wanted to point out number six is the processes. So we want to execute the app as one or more stateless processes. So that's kind of a microservice kind of approach. Now there's also another number nine to its disposability. We want to maximize the robustness with fast startup and graceful shutdown. So really this is the microservices kind of a space. So how do we preserve state across session, transactions and network boundaries? Let's kind of visit a couple of kind of areas of this. So techniques, mechanisms, we're still using this today, right? Caching, you can be using caching just for example, like a J cache specifications in Java. There are providers that you can use. For example, like Hazelcast. I actually have a quick example that I can show you about session persistence too, shortly. And so caching, then there's also database style transactions that are also still being used in place. Then there are also cookies, right? These are all still being used sessions, right? Then we make use of HTTP session to kind of hold, you know, carry with it the state of the data. And then there are tokens like JWT, things like that. So these are kind of more like the techniques and mechanisms. And we can kind of look into that later. And then let's move a bit too then into cloud native infrastructure. We can't avoid but talking about Kubernetes on, of course, there's OpenShift too, that's actually, you know, bring Kubernetes up another step, right? So, okay. So let's kind of quickly take a look at the features in Kubernetes OpenShift, right? So there's a concept too. In any distributed systems, you need to have some algorithms in there that helps with doing leader selection. Because if you think of it, right? We have all these clusters replicated, you know, kind of parts and working nodes all running together. You need to have somebody to make decisions. So kind of do election. So you need to select a leader. So the thing is too, you can use Apache Zookeeper, for example, but the thing is too, is that Kubernetes has a built in leader election algorithms. That's already in at CD. So you can kind of rely on that. It's not like something that you can do in extra. And so you can use that without using any external leader election algorithms. Okay. And then there's also Kubernetes has stateful sets. So as such, right? The name already implies that stateful sets, it's kind of a natural Kubernetes feature that will help you to manage the statefulness of your data sets. So as such too, Kubernetes is an infrastructure level kind of capability, you know, more like that rather than the application, but it helps with keeping statefulness of all of your data too on the infrastructure level.  So all of your components get upgraded and kind of, you know, kind of going through all of the life cycle. So it will manage that for you. This is the stateful sets. So. And then there's also persistent volume, which is then persistent volume to works also, like with persistent volume claims, which is like request to kind of request the storage is actually gets into quite complex too, which I won't go into details. And of course the red hat, you know, open shift folks are very familiar with this. So somebody will be explaining that in much deeper detail, but at least to, we know that Kubernetes has the persistent volume, persistent volume claim to kind of request storage and also the storage class of these things. So it has its own way of managing all of these kind of volumes and map it to the actual, you know, all of these complicated kind of settings in your whole cluster too. And then there's also in Kubernetes, there's also this concept of a session affinity, sticky session. So I'm using some sticky cookie picture in here because the thing is the idea is that you, because the thing is there are so many, you know, kind of cluster in your, in your cluster, you have different notes that can do the job. But the thing is the best way to do it is you want to kind of keep, you know, that same kind of component that's servicing your client to service the same client again. So then you avoid any kind of like initialization need, right? So there's this concept of the sticky session that will kind of help you sort of like a cookie kind of in a cookie kind of way of handling of the session, but it's on the Kubernetes kind of infrastructure level. So, okay, so that's that. And then let's get into a bit more of the programming side of things, which is on Java level. So what can we do, right? There are programming design patterns that you can utilize. So first and foremost, the most famous is the saga pattern. Saga as such, you know, it's like a long story, right? That that's what it, what it means, you know, outside of the computing context. So essentially too, you can be doing transaction that can span, you know, like, you know, kind of a long time to, or span multiple components in a highly distributed kind of manner. So that's saga pattern. So quickly too is that saga is interesting. It's not the same as traditional database. So what it has is that if there's any problem that goes on, you don't actually call it a rollback. If there's, there's no such concept as rollback, although the effect of such can be kind of like a rollback because you're trying to restore the state, you know, of your component. Let's say, you know, before the transaction happens kind of like that, but we don't use the word kind of transaction in saga.  If anything goes wrong, then you kind of call upon your compensation module to compensate for what's kind of been done and you need to restore it. So that's the idea. So that's a kind of quick nutshell about what saga is. And there are two ways of doing coordination saga too. There's choreography, which is event driven way. And so in the, in that kind of model, you don't make use of any orchestrator or like a middleman to kind of direct traffic. Whereas you're kind of more like a self initiating event driven way of doing things. Now the other way would be orchestration. Now let me kind of quickly kind of do this. I know I've been talking a lot. And by the way, this is Chicago. So hello from Chicago. Anyway, so quickly. So I'm actually working on, I come up with this design that I will be implementing. Hopefully, you know, very soon when I can find time. It's actually a choreography event driven saga. So here I am, right? Using an order processor to illustrate this, you know, how we can do it. Now the thing is, so I'm working with open liberty and micro profile. So my micro profile has this new feature that came out long running action LRA that came out just in May. So I also have resources for you if you're interested. So as you can see, I can have, you know, some order that comes in, let's say using Kafka event driven, right? There's order coming in, then it's trigger, you know, the order micro process, microservice to get to get busy doing this thing. And then it was sent essentially to it doesn't really send until, you know, inventory check is subscribed to my order generated event. So is credit check. So then I'll, you know, these two will get to work once I get the order created event. And then they too go into work. And let's say everything is all good. Then it's kind of say is checked. It's all good. Then I can move on. But in the meantime too, you can see too there are also this self feeding kind of thing back to the order too. So then the order can keep updating his status as well. Now. So let's say if it's all good, then the inventory is all checked. You know, I have the product in place that's being ordered and the credit is checked this, the subscriber, the consumer is all good. So then I would just say, you know, payment processor, you can get to work now. So payment processor in the meantime would then go off and kind of work with the payment authorizer, you know, third party, all of these things. When it's all good, then basically it will have status that would be changed. So once the event, you know, kind of gets updated, the event gets generated, then it's basically trigger if order paid and kind of go back to the order to kind of indicate his pay, then I can then trigger the shipment processor to go forward. Now, this is kind of like a simple way of illustrating a usage example. As you can see, it gets, it can get very, very complicated. Now the thing is too, is that in using this kind of approach, I do need to also write compensator. It's basically components that will go hand in hand. And that's actually a bit of the controversy about using compensator. It kind of means that I'm going to tightly couple my compensator to certain action, you know, that goes along the way. So that actually will be left for another talk that I'll be talking about this compensator approach. Okay, but that's kind of like giving you a quick example of how you can do, you know, with a saga pattern. So, okay, then I will get back to my here. So, okay, that was a very quick thing. And I realized my time is running out. So, okay, so the quick thing is I mentioned about long running action, saga interaction pattern. So how can you do it? So again, micro profile, we have LRA. So I have information for you to kind of check into. Now, what about reactive quickly? So reactive systems, right? I kind of point out about the responsiveness, right? These are the four tenets of reactive, responsive, elastic and resilient, essentially kind of satisfy the cloud native goals too. But then as such too, there's differences because reactive systems, we're dealing with data as streams, you know, versus, you know, traditional way, we're not necessarily all streams, you're an imperative, except I think if you get into project loom, then it will be different. It will change the whole course. That will be interesting to see. But I just wanted to point out that perhaps using a reactive approach in your micro services and also in an event driven way is actually they can all go hand in hand too. So, okay, so I think I may not have as much time to do the code example, which I was hoping to, and I'm sorry again for my audio. But the thing is too, just wanted to point out to you, you can go to our interactive lab session for open liberty. I was going to show too. Let's go back here, open liberty. Everybody you can visit open liberty. It's always good. So let me kind of quickly kind of, this is our open liberty link. Like I said, it's just been exciting actually for me to kind of start working on this because it really is very cloud native ready, kind of very flexible library and framework and runtime. And also the quickly kind of point out is that the runtime is actually operates, you know, underneath the hood is open J9. And that's the Eclipse open J9, which actually J9 use was a, you know, IBM kind of clean room implementation of the JVMs. All open J9 has been donated to the Eclipse Foundation. It's already and right now it's packaged up into like no cost. No, you know, kind of any extra baggage of IBM summer room runtime. So I suggest you to look into it, but open liberty is already working with that. And then also open liberty works very well too with open, with micro profile, which is all Jakarta EE compliant. So all of these in micro profile, you can find all of the features that you need, for example, Kafka, GRPC, GraphQL, all of these things. So LRA is what we're talking about. So I suggest that you can kind of take a look into it. And then I know I only have two minutes. So for take home, you can go to the open liberty guides. And if you look into the guides, look into persistence. So the persistence is what I was going to show is caching HTTP session data using J cache and Hazelcast. So then you can also kind of get your hands on and after this, you can follow this and we'll explain to you, you know, how, how you can leverage on Hazelcast in this case that will implement the J cache for session persistence. So, okay, so I realized I only have one minute. So let me then quickly kind of go back to my slides. So this is like a J cache and Hazelcast example. And then there's also another example that I have is basically using a stateful open liberty application in Kubernetes. So if you want to visit that, then I'll also share the slides with the organizers here and some resources and links. These are all the resources in here. Please visit that. And specifically is saga design pattern, micro profile, LRA 1.0 and also a nice block that was written up earlier this year. And these are links to our open liberty micro profile and Jakarta EE. And these are other IBM resources that you can visit. There's also a programming with Java and IBM cloud that will explain to you how you can get started with cloud native in Java. And the other thing and then also I'm also live streaming on Twitch every Wednesday and in fact this afternoon I'll be doing that. Today's session I'll be talking about Kafka. So there are other talks too. And then there's also our expert TV and meetups too. And also my Chicago Java user script if you'd like to join. And also this is free IBM cloud account and we run support open shift. And with that comes to a close. I think I'm right on time. So thank you very much. If you'd like to stay in touch, please connect me on Discord as well. That's my link to Discord. I'd like to continue the conversation there and follow me on Twitter and everywhere else. So share with me what you have. Thank you so much. Thank you.


So I'm Dennis Brown. I'm here today to talk about zombies and how hackers kill them. Unfortunately, this isn't the kind of zombies that you kill with shotguns, which is fun or the ones that involve people loving Twinkies, although I do. It's about the badge game we had at a conference called Kohakon, based in Rhode Island earlier this year, where we had a zombie game on our badges for the conference, the electronic badges. Let me pull one up here, like this one here. If anyone can see that, probably not. And we had a lot of good fun with it. We had a lot of good data as a result. I'm here to talk about that today, how we did this, what the data we collected was, what people did with the badges, once they got their hands on them, the havoc they created, and other kinds of good stuff. So let's get into it. So what is Kohakon? If you watch Family Guy, you probably have heard of the city of Kohak, Rhode Island. It isn't a real city in Rhode Island, unfortunately, but it's like every city in Rhode Island, if you know the place. It's also the state clam. Yeah, we're strange there. But it was a new conference this year. Regional one, we had a great turnout. It was a lot of fun. We focused on having a lot of info set tracks and having a lot of macro culture stuff, too. We had a wonderful hardware hacking area. The tool guys came to have a lockpick village. It was a blast. It was a lot like a really, really, really small version of DEF CON in some way. So we had a great time. And I'm Dennis Brown. I help organize Kohakon. I also help run the DC 401 chapter. Anyone here from DC 401? Yeah, thanks for coming. That's all of you, I think. And my day job is I am a security researcher at a tenable network security. I do a lot of malware research, or if I talk to anybody who knows tenable at all, I write nasals because that's all anyone thinks. So Nessus plug into us all. So this is kind of out of my element for something, a project to work on. I haven't done a lot of things dealing with hardware. I've never written firmware ever before. So this is a fun project to get involved with. We had a great small team of people working on these badges diligently to really make them fun and usable. And it was a great project to work on. And our goal when we started working on this was have a badge that was hackable, something that you could just sit down, have fun right away, no barrier to entry or anything. And then something we wanted to use after the con was over, so you could take it home, keep hacking on it, and maybe get something useful done with it. And there's actually some great useful things people have done with it after the con, which is awfully nice. We want to include wireless connectivity on the badges because having an LED flash or something that is kind of passive is fun. But we want to have a little more interactivity with the badges because that's just cool, right? So we wanted to also put a game in there, too, because it's fun to have a badge that will like a TV be gone badge. That's fun. It's fun to cause trouble when you're in a bar with guys wanting to watch sports and you turn off on them. But we want to have people have something active to do at the con and interact with everybody else there. So definitely that was one of our main design goals when we started. We definitely wanted to have an open source development environment because it's no fun if you have to go and use proprietary tools or ID that you're not familiar with to get going on it. We ended up using GCC, so that worked out really well for us. And we wanted to make it easy to write custom firmware for. So I'd say out of the four points here, we only hit about three of them. The ease of development or getting people working on at the con was a little bit of a misfire in our part. And I'll talk about that later when we go over the mistakes we made. So the badge design itself was really great. It actually was done for us from the start. We worked with a company called Redwire LLC based out of Massachusetts. They had this product. They called it the Redbee Econo Tag. That's the picture of it there on the projector. It's a really nice device. It uses the pre-scale MC13224 microcontroller, which is an ARM7 microcontroller. It runs at something like 20 something megahertz. I forget the exact number of megahertz. But it's a really nice chip. And the great thing about it, it has a lot of nice features. It has a nice washdog timer to make so you can keep it from crashing when some guy's sweaty shirt touches it and makes the whole badge freak out which happened a few times. It has an AES right on the chip so you can have all kinds of nice encryption with it which we totally did not use. But the feature we used it for was it had ZigBee onboard us. 802.15.4. This was excellent. This gave us pretty much everything we wanted in one package. We started working with the Redbee guys. They helped us expand upon it a bit to really make it what would be great for our conference. Another nice thing it had that we chose to keep was it had a USB connector on it which made this a breeze to flash and write custom firmware for and just expand upon it however you wanted to. This worked out wonderfully for us. The picture on here is the end product we had. And if you notice the middle of the badge is pretty much what we had with the Econotech from the previous slide. And I apologize for missing a few things on the slide there. Hopefully all the good data is there. We took the Econotech design. We changed a few things about it. We ended up with the same interface they had with two buttons and a reset button. The reset buttons on the left. We added five red LEDs on the left side. Like a little bank of five and a RGY red green yellow RGB on the right side. LEDs on the right side. And we made it use triple A batteries which we thought was a good call because if people wanted to do something after the fact they could just go to any pharmacy, any store and pick up some triple A's pop them in the badge and they're rolling once again with whatever they're doing with it. Ultimately we worked with a local manufacturer J&J Technologies out of Foxboro, Massachusetts which we loved working with local fabricators for these. We ended up with about 30 bucks for badge which we rolled into the cost of admission which made this very affordable for us. But the overall admission was $100 so this was right in our budget. It was great. We just rolled it back to the attendees and everybody seemed to enjoy that part of it at least. So the badge was really easy to code for. Like I said earlier it's just used as an arm cross compiler. If you go to the MC1322 development site run by the Red Wire guys they have all the tools in there to build the tool chain, get it all working just properly. Very easy to work with. And there's a lot of fun things you can do with it even after the con when you're not playing with your fellow con mates anymore. Josh Wright at Kohakon used the Killer B tools he developed in order to attack other badges there and really make them freak out quite a bit which was fun. I gave him a few tips on things he should try and he tried all of them and really made people freak out a bit at the conference. There's Kentucky Support. Kentucky is essentially a full system environment for the badge itself which helps you, which abstracts away a lot of the things. It gives you IPv4 access, IPv6 using 6 low pan, a lot of other great utilities for it. Essentially it's like running a full operating system on the badge in some ways. There's a lot of sample code that worked really well that was distributed with the development tools for the badge which people were using to right away write packet sniffers, packet injectors with a lot of great tools. And just the other day Dragorn implemented Kismet support for the badge itself, which I'm going to demonstrate here and I hope it still works. It's a very alpha stage. But what this does is it turns the badge into a ZigBee sniffer. So you could use this or if you purchase the Kono tag from the RedBee guys, go to your next pen test with a KON badge and try to find, maybe if you're at a skate episode they use a ZigBee find vulnerabilities there walking around to your conference badge. I thought that was pretty cool. So I want to demonstrate that here to show the kind of stuff we see. One thing I asked for before this conference, if anybody was reading on Twitter or on the DC4 mailing list or on the Kohakon site to bring your badges so now would be a good time to get them out and generate some traffic for me to actually see with this. Just one second, I'll set this up. So I'll apologize if it isn't too easy to see, but okay so let's see if we can see anything here. Sorry I hit some buttons here. If not I'll fire something else up. So this is the Kismet server we're running right now. The ZigBee, the D15.4 module is a new one in Kismet SBN. It was checked in last night so I'd advise checking it out if you want to see it. Nothing happening I'm guessing. Live demo failure I guess. Oh well. But it was in there if you have something like this or any other device that uses ZigBee, that uses serial ZigBee drivers available for this chipset and some other ones as well. I was talking with Dragarn who unfortunately couldn't be here this year, but he said he believes it'll work with the ninja badges as well this year because they use the same microprocessor so hopefully get some good stuff with this. Is this thing just failing? Absolutely. So supposedly these things have a 600-foot line of sight rates. Let's try it out. Oh hold on, let me try that. Okay so this is absolutely failing. Sorry it's really really early alpha code. But if you have one you want to check it out, just check it out of the Kismet SBN repository. It uses the plugin architecture which is pretty new in recent versions of Kismet so if you haven't used it, it's really just other modules you compile in. You can add them to your home directory, Kismet directory Kismet slash plugins and it loads right up and works fine except not right now when I want it to work of course. So let's get away from that.  So like I said earlier with the firmware we wanted to have a game for it and we wanted to be something that attendees could do while they're sitting in talks, while they're chatting with people, just to kind of pass the time, have some fun with and then of course attack in any way they can see fit. So we came up with a bunch of, we had multiple design ideas. The one we got working first and actually had fully implemented in the firmware was a Tamagotchi game which would have been incredibly lame. This was about four weeks before the conference so we were very worried that this wasn't going to pan out and we were going to be stuck with a incredibly lame firmware on the badge. Fortunately that didn't happen. We went through a few iterations and then as we were working on the design for it we landed on the concept of a zombie versus humans game where some people would be humans, some people would be zombies depending on the firmware they got when they registered for the conference and they would have instructions on how to attack the other side or maybe come another side or switch over to the other side. We implemented this in about three weeks which is an awfully short amount of time to implement something for this scale but it was hacked together really poorly. It came out to be about 4,000 lines of code total for the various firmware versions we had and the code is absolutely atrocious but it will be available on the KohoCon website if you want to check it out and criticize and laugh. So that is certainly something you're welcome to do. Next time we do this we like to have more time to implement this. I'd recommend anybody who tries to do a project like this themselves give yourself more time than a few weeks to do it. So the way the game works like I said before humans like to kill zombies which is how things work in nature. They had multiple attack modes so they could do stronger attacks but they could take more damage if a zombie hits them or they could be more defensive and not get hit as hard and not hit as hard.  And the speakers and vendors we decided to give them something special in the firmware without having a different badge for it by making them a cleric or a healer where they would heal humans and try to convert zombies back into humans. So this was just fun because we figured the speakers probably were just going to do their talk and meander about maybe not sit around too much to play with it and the vendors are probably too busy doing other things. So we gave them pretty powerful capabilities for these badges here. And then the security people we called them the muscle there and after another type of clam or shellfish where they would be able to attack anyone but they had the weakest attacks possible because they were supposed to be working the whole time and we didn't want them having actual fun. We wanted them to actually work so we put them very weak. So I'm going to do a live demo here with some badges I could use five volunteers actually to come and grab one and give it a shot just come up just come up I don't care over here over here one two three four five that's five that's good wow that's really creepy thank you so grab on the badges here you have a whole kit with some AAA batteries please put the batteries in the right way there's little marks on the inside of the battery thing and there's land use if you want to hook it up and hold it somewhere so what a pleasing design I know the green circuit board design was called how we do this very cheaply so it worked out well for us in that regard so this is the exciting part watch people put in batteries so once they have it in it'll boot up right away and load up into a mode hold yours up there this would be a human badge the one with the white or the whole column of red LEDs lit there that means they're not zombies and they can use the bottom button to change the mode they're in and the top button will then depending on the setting they have it in will send an attack packet out and if another zombie in the area reaches sees it they're hurting and they're not so happy at that point so what we have can you hold your badges around to the front for humans and if we all have humans this is epic fail on my part is that batteries in the wrong way they won't catch on fire so to make this a little more compelling I'm going to fire up one of our schools for this so we collected a lot of packets through the whole weekend we had two packet sniffers running in the various parts of the conference and we captured a whole lot of data here probably have to reset that so I'm going to run this here while people hit the button so you can see the actual Zigbee magic in the air okay so we see on the screen here I hope you can read it coming up there the QZ entries are all various packets going through the one denotes a human attack, two is a zombie, three and four are cleric attacks or heels I guess they are technically so this is what we saw all weekend long just traffic like this flowing in and out the rest of the columns here QZ just denotes the type of packet it is, the type of attack, the third power is the strength of that attack and then the rest of it is actually random data so that's just garbage if you notice one thing here where all the threes and fours always have a 20 after it I'll be talking about that shortly that was a terrible oversight in our part but anyway so his badge is flashing in and out that means he's a dead zombie so you've lost I'm sorry you've lost but you can button mash and then come back to life so zombies had a very nice feature in that regard so this is fun I have this demo with everybody having fun seeing random texts come up on the screen I'm sure okay well that's good you can just keep playing this whole time have fun kill other people yes I like how you're button mashing we'll be talking about that briefly so that's good thank you very much for helping you guys keep your badges you can have a little souvenir on the top thank you very much you're welcome yeah just give yours away because you're a loser okay thank you for the volunteers so that wasn't too uncompelling but that's pretty much what we saw all weekend that was exactly what it was like being there not really so like I said the attacks for the humans and zombies did one to five damage depending on the mode depending on things actually went up to double that if they had certain other conditions the humans are out with 500 health so they have to take hit 500 times and they also had to have defense they had other ways to defend themselves against it too so it took a while to kill somebody zombies had less health but as I mentioned if you button mashed on it you could come back to life so their health kind of didn't matter after the clerics like I said they healed 120 health at a time which was terrible this is an influence of me having played a lot of World of Warcraft back in the day where nobody wants to heal so I figured nobody would want to heal at a conference either there you go that's my priest over there so this was kind of unfortunate we should have made this much lower because there were actually people in the vendor area tweeting saying hey low on health come over to our vendor booth and we'll heal you up which is great advertisement it's a way to get people to your booth but kind of impacted the game where a lot of people didn't die like we wanted them to because I was definitely on the side of the zombies for this so that was pretty interesting and there's also another badge which I don't have a demo of here but it ran in God mode where I could turn people into any mode they wanted me and one other person to have the God mode badge because you didn't want that getting out too much but in the end of the day it ended up being a prize if you're able to hack your badge properly and get to this mode this was really fun to play with actually because once I noticed the healers were having a little too much fun healing and advertising on Twitter and getting really upset if people were trying to attack them where they would be physically running away from people although that doesn't matter guys unless you're really really fast I had one point where I was intent on getting Dan Kaminski who was very happy to be a healer and very upset if people were coming after him following him around trying to turn him into anything but a healer but unfortunately didn't work and there's reasons why so when we roll all this out we made a few predictions okay what are we going to see people doing with this what's the kind of stuff they're going to want to do to really mess with us well the first thing I figured well all the packets we didn't those were unencrypted but all our packets were encrypted and we chose intentionally bad quote-unquote encryption so people would actually crack this and not really have the whole weekend spent saying oh I don't know what's going on or just doing a simple replay attack or something like that but we did expect that kind of stuff we didn't expect replay attacks especially if they cracked the encryption we expected them to actually totally own the game which a few people did we expected hardware hacks people putting like a 555 timer hooked up to their attacking button and just constantly auto attacking the whole weekend long we didn't really want that to happen but it was certainly a possibility to a point and then we knew people would do things we never expected and boy did we miss the mark on some of them so on April 23rd we opened up the con at 5 p.m. start distributing the badges we wanted the about 5% human badges attributed 30% zombie the rest for everybody else and the first day went pretty much as planned the humans there was just so many of them they didn't really care what was going on they hit buttons they wouldn't see anything happen so they just said oh this is great I'm going to go enjoy have a drink and see what's going on the zombies saw things happening and they were very excited because there were way more humans than there were zombies so they went on the attack and really try to get a foothold of more zombies in place than humans so I was certainly encouraging this to telling everybody who could attack as a zombie to definitely attack as a zombie because the humans needed to die so they did so the first day Friday night really went as planned it was a lot of fun we'll see the data in just a moment here but on the second day as soon as people woke up and had some time to play with things really strange things started happening in the data now this graph here I apologize if you can't quite see the top of it is what we saw on the very left hand side you see I apologize if it's hard to read but the very left hand side the big spike you see there are the zombies and those are the people like I was saying get everybody all the zombies need to attack the humans were attacking a little bit they were there but they didn't attack anywhere near the volume of the zombies that they did the next day however if you notice all four colors on this graph spiked all the way to the top and they did a little bit later on in the day that evening too so I looked at this data when I first saw it and I was like this is really strange why would we have all these attacks like that so we had to go through the data dig through and investigate and we came up with a lot of interesting results which I'll get to in a few slides here so one of the things we did see were hardware hacks we saw a lot of neat people trying to do things with a badge to affect the game directly which I thought was great unfortunately outside of predicting that people were going to use like timers to automate their attacks we didn't really provide any good ways to use the hardware to use the game directly and we actually wrote into the firmware a time out so they wouldn't actually be able to automate their attacks because we didn't want people just flooding the network full of packets all weekend long that was no good that would really be detrimental to the quality of service so to speak so we rate limited all the attacks nobody was able to do anything about that and some people tried other people this also prevented people from mashing the buttons like we saw one of the volunteers doing while he was up here where it really didn't matter you could only attack once every five seconds and so people had a lot of sore thumbs as a result for no good reason the buttons were not very friendly on the thumbs but this was great because it stopped many of the automated attacks we saw and everybody just we didn't tell anyone so we had a lot of fun laughing at people the whole time so there were some moderately successful attacks depending on how you define success probably the most impactful event that happened was one guy had a fuzzer he just took some of the sample code we had said well I'm just going to throw random junk out there I know how big these packets are I'm going to throw random data and see what happens and it wasn't entirely predicted by us and we didn't really think someone was going to actually flood the network as much as they could the entire weekend long to essentially shut down the game in a few ways which was quite unfortunate he made people's badges were changing modes really strange things as a result things we didn't really predict and it essentially just turned into a giant denial of service attack on the conference for the entire weekend the badges just did not know how to handle this we didn't do anything smart like check something on the badges so that was really a poor call in our park because every packet was essentially valid and this really confused the attendees and myself for quite a bit so we collected about 30 megs of logs which was the data you just saw on the screen there so 30 megs of just those 10 or so characters is a lot of data and the attacks we saw we saw 39,000 in change successful attacks and this includes a fuzzed attacks and other automatically replayed attacks that were of a valid type but we didn't really differentiate them we couldn't differentiate them from what was actually fuzzed and then we saw one and a half million fuzzed attacks so the people who are doing this kind of stuff they were just firing out packets as fast as possible and really really messing things up you can see afterwards in the data though what we have in the screenshot there is an example of what they were doing the third column there is the actual attack type the valid types were 1 through 4 and 99 for the god mode none of these were valid all the data here was clearly just being randomly generated and this had just such a devastating effect on the badges it was fun to a point so fuzzing aside people did similar things with packet replay which was very successful there were at least two people I know that were at the conference who actually figured everything out started replaying packets and just started impacting the game however they wanted this was great if you want to have an automated attack you just have to make a custom firmware which was very easy to do with the samples start replaying a captured packet that you saw that you knew was a valid attack and just go to town on it and it worked really well for them this got around the rate limiting code we put in place because at this point it's your code you don't really have to apply to ours at this point and some people actually did this to replaying just generating some random data in there but using the proper format and inferring what was going on by watching other packets and they actually accessed the god mode packet these ways but they didn't really understand what was going on they just were replaying what they saw and replaying it a bunch of times and seeing everybody's batch freak out so the way that this got a lot better was when they cracked encryption now we use this very very high tech thing called XOR if we did this intentionally because we wanted to make it fun if the one line there is just a sample of what a packet looked like raw the first byte there actually the second byte in this line is the XOR key itself so every packet that went out we had the key in it so this made it pretty easy to figure out what was going on and then the second and third bytes there were the packet type and the power of the attack or just look into the other parameters that passed through for the function and the rest of it like I said earlier was junk once people started cracking this the whole game opened up everybody just started saying okay what can I send through now what will work I know I can send valid packets of any type I want let's see what happens and this is where people started having a whole lot of fun another way people didn't have some really good luck was by brute forcing as well especially once the encryption was broken we were seeing people just going through trying everything they could to figure out what was going on and they essentially mapped out the entire protocol this way especially if they were in closed quarters without a lot of other people around direct actions of what they were doing on other people's badges so that was a lot of fun we had a excuse me so we had a few obvious attempts of what was going on here I'll show that right here this was clearly one person or maybe multiple people I'm not exactly sure because we just collected the data randomly collecting or just trying anything they could they started replaying the same numbers through the different power fields here this may have had an effect if they weren't using zero as the packet type that was an unused value so this was pretty interesting where people are just firing away at crazy trying to see what they could make it do and then someone smartened up here and said it's probably awfully hard to read but they just went through and iterated through every kind of packet there was to see what would happen trying at first in the packet type column which they didn't know at the time of this they just tried one column at a time then again at the packet power column this was really smart this actually was pretty close to actually breaking the entire protocol for the game and figuring out what was going on and except for the part that having zero as the other value would essentially make all the attacks be of zero power which obviously won't hurt anything oh and then like I said in the slide previous the attack type of zero so that didn't do anything either so whoever this was was just a hair away from cracking everything that was going on and but just didn't quite have it so unfortunate for him but this is definitely the way that people were going through and thinking about how to attack this trying to crack what was going on in these packets figure out what was going on and ultimately people were figuring this out people were generating any kind of packet they wanted out of their badge and including the God mode stuff which was letting people walk around in just nuke entire rooms and turn them into any kind of thing they want even make them all dead which was a lot of fun for everybody so that was definitely the grand prize for the badge hacking that was going on so we learned a lot of things from this here one thing we learned is when you're playing around with something especially something small scale like this in a confined area with people who know what they're doing you really have to prepare for denial of service attacks they really ruined us for the game essentially every time that one guy would walk into the room one guy knew who was doing it walked into the room room pretty much shut down the game was in action for a bit if I think I lost a slide here but go back to this so the big spikes we have here the ones where all the colors are going up the same that was when this one guy would walk into a room with his badge that was constantly fuzzing because of that when everything shut down things were just going nuts and you can tell when he went out for the night to check out Providence when he came back to the hotel around midnight so this guy was a walking denial of service attack so that really sucked we got to make sure we do something about that next year the one thing we really need to do is put checks on the packets maybe we're using a lot of just really fake packets they weren't really valid ZigBee packets or anything at all but it was really just what we got to work in the three weeks we worked on it it would be a little nice if we made something so check for a valid packet make sure replay attacks don't work and make sure fuzzing doesn't work in the future but it's nice it's kind of fun to track people at the same time so that worked out pretty cool sorry about that the one thing that was cool about XR encryption was it was almost good enough for the conference to last about 24 hours from when the first badges went out so people would have to work on it off and on for a day and then actually have something that they could play with based on the data we saw people started really doing strange things and really started doing what looked like brute force attacks after about 18 hours so I don't know I guess that's if you take a sample of 250 people someone will crack XR encryption in 18 hours so I want to quote me on that metric but I thought that was pretty cool one thing we need to do definitely next year is incorporate more hardware hacks into the game this was probably one of the more complicated parts of it sharing down the 5.5.5 timer attack approach was really unfortunate especially since somebody tried it because only one person tried it where maybe nice to have some way to influence it that way that's definitely something we thought would make the game a lot better for next year and really improve upon things quite a bit so but at the same time a lot of people had fun with the hardware it's a really fun platform to work with add things onto it has a lot of extra GPIO pins so you can start putting things on it like you wouldn't anything else really fun I would really recommend checking out this whole hardware platform and doing what you can so I have a little conclusion here but not the finale so to speak in the end the humans did win mostly because the zombies couldn't do anything about it so that was unfortunate but that was the reality of what we saw and we love it just the same because people had a lot more fun with it that way having Zigbee involved in badges like this is a whole lot of fun if anybody is trying to come up with an idea for their own conference their own whatever functionally to have some sort of ID I'd really advise checking it out this chip here is in about the $15 range for the chip itself so you might be able to whip up something pretty low price we wouldn't have anything really fancy here an integrated antenna onto the board and a few other components so you could probably do this even cheaper if you really tried but it was a whole lot of fun and people really really love messing with their fellow attendees they love making problems for other people watching people actually just saying I'm attacking just so I can attack you was a lot of fun and people really got to kick out of the badges they're really functional they really worked out well and we were happy that we had something that people have gone on to use after the conference so that was pretty great this picture pretty much sums up how it all went for us we set up a cardboard box near the exit saying recycle your badge here and all we got was a napkin that said sorry way too cool so that was a nice validation that everything went well at the end of the day and unfortunately we didn't get any badges back so so I have a bit of a spoiler here that we're going to be doing this again next year in Providence, Rhode Island the end of April and then April 29th through May 1st at the same places last year the Hotel Providence which was very gracious to us and we're trying to come up with a badge of sign for next year now we definitely want to do it earlier to give ourselves more than three weeks to code the thing but we're thinking about going to this chip here the ATMega128RFA1 which is based on the Arduino platform one thing we found from doing this contest was when people were trying to do the badge hacking most more people had experienced with the Arduino than they did with the GCC or any other processors out there so we want to enable that and let people help figure that out and start being able to do things while they're just at the con attending and having fun the picture you have there is from a that's from the Dresden electronic company in Germany I forget the exact model number of that chip there but they saw that as about 21 euros right now I think for the price of it so it's about 20 to 25 bucks I guess US but it's pretty cheap the chip itself just for the ATMega chip is actually much cheaper so hopefully we'll be able to do this better and cheaper next year but just like we have with the chip this year hasn't embedded ZigBee components in it we might try to do an LCD but I think the Ninja Networks guys beat us to that this year so that's unfortunate but anyways it's also subject to change so if we find that this is a bit out of our range or things get a little crazy we'll probably just go with the same thing we had this year and do something similar but we hope to see everybody there it's going to be a really great time and it'll be really fun so that's about all I had is some special thanks I want to thank John Duckstuff for being the guy who took care of everything last year we're dealing together Dragorn for his edition with Kismet where if you get your hands on either one of these badges or the red be a Kono tag it's a really great enhancement especially if you're doing anything involving ZigBee at all it's now in SVN so go check it out the redware guys were excellent working with us helping us out getting the entire badge design essentially people have play testing like me and the kind attendees which really made this into a fun experiment for us to figure out what was going on see what kind of attacks people would do and go on from there and we had a lot of fun doing it and everybody seemed to have a good time so I'll take some Q&A now I have a few, I'm a little early here I think the demo didn't go exactly as planned but if anybody has questions we have to answer them now yes over there how many badges did we end up making was the question we made about 250 I just think it was exactly 250 we got them all produced locally in Foxboro, Massachusetts which worked out great for us they cut us a nice deal for helping them letting themselves advertise during the con we were happy to go to the local provider too when we did the initial batch we received about 25% of them were broken or had some sort of flaw with them about a week before the cons that made things really hairy towards the end there we had a lot of emergency soldering and emergency shipping and stuff to do which was a lot of fun but in the end it all worked out pretty well so we had enough for everybody any other questions okay I think that's it thank you oh one back there actually oh thanks we have plenty of time so come on up and yell it and it'll work better oh thank you I'm sorry actually we don't we had very few coming here and we saved them for the demo so it was very very tight at the end everybody kept theirs nobody wanted to share theirs after the fact I think that's it go