Today we have, on behalf of or from Utilisec, Mr. Justin Cyril, who has spoke at a number of different conferences, including Black Hat and Def Con. Let's give him a big hand and welcome to today. All right. How's everybody doing today? All right. Good to hear. Good to hear. Okay, so at Black Hat, I did a four-hour presentation, a four-hour workshop on this material. You guys get the condensed version of it, but I still have a lot of slides. So we are going to go through the slides. We're not going to hit all the points on the slides, but we are going to have them uploaded to the Def Con site so you do have availability to all the content. Okay, so first off, I am an managing partner for an organization called Utilisec. We specialize in security services for electric utility companies. We provide penetration testing services, secure architecture design, and other types of services, including trying to represent the utilities and their interest in a lot of the standards that are out there, trying to build in security in some of the standards. In fact, some of the groups that we've worked in, we currently lead and facilitate, one of our managing partners leads and facilitates the NERC CIPs. Let's do Joe Bucciero. We also lead many of the different groups that the electric utilities have put together to try to build security or try to generate industry awareness in security issues that we have inside of the smart grid. So the purpose of this talk, a lot of times we hear a lot of different talks, Black Hat, Def Con, Shmucon, Torcon, of different types of things in the smart grid. We have a lot of good research going on, a lot of good researchers that are doing a lot of good in this industry. We also have a lot of talks that are smart grid talks that are simply there to try to generate a lot of the hype. So the purpose of this talk is really to try to give everybody a very clear picture of what the smart grid is, give you an idea of some of the issues that we're dealing with, some of the different attacks that we're seeing, and some of the different attacks that we perform as penetration testers on the smart grid. And more importantly, show you that the smart grid is more than just SCADA. The smart grid is more than just smart meters. There's a lot more out there to do. And a lot of the people inside of this audience that are professional security professionals, your skill sets are very applicable in many areas of the smart grid. You just may not know where to look and where to try to generate that business. And ultimately, one of the ultimate goals of this talk is to really try to generate more awareness, more interest in the security community, to try to get more researchers and more people in this field. Because the smart grid is not perfect. Name a single vertical industry that is secure. We need to get more security. We need more people in this. We need more expertise. We need more technical expertise in particular. So really that's some of the goals. So the first half of the presentation we're going to be focusing on trying to let you guys understand a little bit more about the smart grid architecture. And then the second half we're going to be talking about some of the different penetration attacks and the defenses that we're recommending and working with vendors and utilities to try to address these issues and mitigate them. Okay, so first off, what is the smart grid? The smart grid, anytime you hear the term smart grid, this is something very similar to hearing the word internet. It is a marketing term. It can mean anything and everything you possibly ever want. Ultimately what the goal is of the smart grid is to try to take our existing infrastructure and add additional intelligence to it. Add capabilities where in the past we had to have people sitting in the control rooms looking at different sensors coming back and having them sit and toggle the different switches, toggle the different controls to cause reactions in the grid. With this smart grid we're trying to add more infrastructure to be able to give us a better view of what's really happening in the grid. A better view of what's happening at the homes of each one of you consuming power instead of being able to see once a month how much consumption you have be able to see within a 15-minute interval of how much energy you're consuming. And hopefully this is going to be something to benefit the rest of the community as well. I don't know about you, but I personally want to know exactly how much power I'm using in 15-minute intervals in my house because I can do a lot of really cool things for my own self for that. Of course, attackers can do a lot of cool things with that as well. But ultimately that's the goal. We try to do the same exact thing with the technologies out in the substations themselves. So if you look at this diagram, this diagram goes through and shows you the different elements or different major domains in the smart grid. So we see the ones across the top, we have the markets, we have the operations and the service providers. These are a lot of the organizations and the companies that are kind of the glue holding the different processes together. Then if you look on the very bottom, we see that dotted yellow line across those four clouds on the bottom. This is the dotted yellow line represents the energy that's flowing from the bulk generation plant to our homes. All the blue lines are communication lines. We have a lot of different types of communications between these different entities and these different domains. So this is basically the same exact diagram. This is showing you just more information and more of the devices. These are the actual systems that you're going to see inside of a lot of the utility companies that are out there in their back offices. So this is all color coded. If you look at the yellow part, this is the operations. This is what most of your electric utility companies are doing. Each one of those boxes are a lot of the main control centers. These are some of the major systems that they have to control your power and to monitor your power. If we look in the... I can only go back just one slide and show you one thing. So bulk generation on the very bottom, bulk generation, I think that's pretty obvious. These are going to be the power plants, the nuclear power plants, the coal plants that are generating the power for us. That power flows over to a group of organizations called transmission operators. These transmission operators are what take this power from the bulk generators down to the distribution operators, which are more of the companies that we think of as electric utility companies because they're the ones that we're buying our power from and then that power flows back into our home. So the transmission operators and the distribution operators, they're very distinct. There are several different organizations and utility companies that act as bulk. But when we look at this diagram, we see the transmission operators in the upper left-hand corner. These are the transmission field devices, the devices that are on those big giant steel poles with the big power lines that we see across some state borders. My wife and I are falconers and we always call these the steelies because when we're looking for falcons to trap, some of your bigger falcons like the Jeers and your peregrine falcons like to sit up there in the morning and catch the sun as it's rising. So that's how we find a lot of those birds. So those are the transmission operators and their field devices out in the field. The upper right-hand corner, that's the distribution field devices. These are the devices that are put out in the field to control the power that's ultimately flowing down into each one of our houses. That's primarily done through what we call substations. Those are those big things that have the big fences around them, the security cameras, all the barbed wire. So those are substations. The transmission operator will take this bulk power, they'll drop it down to the distribution operators, usually in one of these large substations, and then from there the power is distributed out to the smaller substations that are closer to our neighborhoods and each of our neighborhoods that are connected back to those substations to be able to pull the power. We have different types of devices like feeder switches that allow the utility company to control which substation one single neighborhood is connected to. So if you ever have a certain circumstance where your power goes out and the power is only out for five to seven seconds, feeder switches and relays are going to automatically be connecting it back over to a different power source, and so that's why you drop out just for a couple of seconds and you come right back on, because they've had some automated event switch you back over to a different power source to try to avoid the power issues that you're experiencing. Now if you look on the very bottom, the things that are probably more interesting to a lot of you are going to be the devices in our home and the lines and the communications between our home and the electric utility companies. That's the bottom right hand corner with all the green. So these are the smart meters that are inside of our homes and the other devices that we ourselves bring, some of our home automation devices, those that have electric vehicles. How many people have electric vehicles in this room? Out of curiosity. Now that we're actually going to start buying them in quantity, so we have one or two. So now we can start buying them. We're going to start seeing more of them. Now look at this diagram. This kind of gives you the overall of the different components. What I'm going to do is I'm going to remove the labels and remove the nice pretty fluffy clouds and show you the communication links between each one of these devices. This will be lovingly called the spaghetti diagram. This is a diagram that I created for, I should say Darren Highfield, one of my partners and myself. The two of us created this diagram for NIST in an interagency report that we released last summer. So if you go back and check out this NIST report, you can see the reference on the very bottom for those that are interested in seeing it, 7628. This document is about, I don't know, 700 pages in length. It comes up in three different volumes. There's a lot of good information. If you're interested in learning more about some of the issues, some of the more details, some of the concerns, some of the security architectures and security controls that we're recommending. At a high level, we realize that this is high level. This doesn't go into a great amount of detail. That's a great document to be able to get that information from. Now what I'm going to do is you see this overall architecture and the communication links. I'm going to point out in different areas that we hear all the buzzwords. Everybody has heard buzzwords like SCADA, right? Everybody has heard buzzwords smart grid, or excuse me, smart meters. I'll show you exactly where each one of those different areas are. So first off, the SCADA. When we talk about SCADA, these are the types of devices that allow us to read sensor information from the field and be able to then make decisions on sensor information and send control signals back out to the field to cause reactions and changes in our real world environment. So you can see those blue sections in the upper right-hand corner that are circled. Those are going to be the sensors in the different devices, IEDs and other types of devices. Yes, IEDs does have another meaning inside of the smart grid and are to use the devices that are controlling a lot of the devices out there in the central brains that usually collect a lot of the sensor information and send it back. We're using SCADA protocols to send those back to what we call our back-end SCADA systems. So here that one single yellow box that circled is our distribution SCADA. In the smart grid we have several different protocols that we use, some of the earlier protocols like Modbus serial communications across basic serial lines. Then as history went forward we started taking that serial communication packetizing it, throwing it on TCPIP streams or UDP streams, sending it back across higher bandwidth lines back to the organization. And some of the newer protocols, DNP3 is probably one of the most commonly used SCADA protocol in the electric sector here in the United States. A lot of these protocols have very, very limited security and it's something we're trying to address. We're trying to build new protocols to replace them. DNP3 being the most popular one it's only been about two years ago that we even had encryption capabilities inside of DNP3. And we are looking at replacing DNP3 with additional protocols that have much stronger security models. So that's distribution. Transmission SCADA is very similar as well. Of course they have their own field devices out there. There's usually their own SCADA, Transmission SCADA server and a lot of those signals. So you can see that there are some communication links between the distribution and the transmission devices. And then we have last but not least the bulk generation. Generation plants, nuclear power plants, all those guys, yes they have more infrastructure than what we show with this one single box. The reason why it's only one box is that was for the most part out of scope other than the lines of communication between those bulk generation and the utilities themselves. For the work that we were doing for NIST. So that's why you only see that one box. But realize that there's a lot more technology and a lot more devices and communication links inside of the bulk generation. The next buzzword we want to talk about is the electric vehicles or the PEVs. A lot of us are to bring these back to our house. Right now they're fairly simple in their communications model modules. For the most part when you get a PEV you are going to be talking to your electric utility company and they'll usually either issue you a separate electric meter for your house that's specific towards your electric vehicle. Because each one of these electric vehicles on average consume about the same amount of power to charge its batteries as your whole entire house uses. So there's a lot of power going on there. Each utilities are trying to find different ways and different strategies to try to deal with additional load inside of our infrastructure. So part of that with separate meters sometimes they'll have you just plug into your normal links inside of your house but then you usually end up getting charged more. Most of the utilities will give you a price break by having the separate meter in there. Right now like I said there's little to no communications. There's a lot of work going on with communications to be able to allow your PEV to communicate back to the electric utility vehicle. It'll be interesting to see whatever happens with this but part one of the initiatives and some of the vendors are trying to get to the point where the electric vehicle can self-identify itself and when you plug power in or plug your vehicle into either your employer or your neighbor's house or some family member's house a lot of people in the industry have this idea that they want to be able to charge track where your vehicle goes and it still gets paid on your bill no matter if you're plugging into your neighbor's house or your family's house or somewhere else. Of course this becomes a security nightmare trying to tie this together with the electric utility companies. For most of us in the states it's fairly easy to reach each city or each town tied to one single electric utility vendor it's not quite as easy a case with those people down in that little state called Texas. They're a little bit different beast for most of us in the electric sector. The next area, syncophasers. Syncophasers are another technology. Now for those people in the room who understands the difference between a digital multimeter and an oscilloscope okay so right now if you want to think about it the way that in general once again this is over simplification but in general the way that the electric utilities right now are controlling and measuring the power inside of the grid is more or less with a whole bunch of really smart digital multimeters that are in all the different substations that are taking readings on average about every two seconds to find out what's going on how much power is being used and all the other pertinent measurements that they need they realize that while that's necessary well that's good they have some good information it's not quite as fine at as they need and so they're trying to employ additional technologies to give them something that's more visibility like you would see with an oscilloscope instead of just getting a digital number telling you what your what the power is they're putting these devices out that will more or less let them recreate that sine wave or what the power is really doing they actually call these phasors and they do phase angles to try to figure that out these measurements, these synchrophasors are making readings of the power at minimal 30 times per second at maximum a lot of the vendors are doing up to about 120 and there are some discussions about pushing it all the way up to 240 times per second down the future in the future but right now I'd say probably 30 to 60 seconds or 30 to 60 times per second are what most of the utilities that are employing these synchrophasors are trying to get to to give them a little bit better idea on what the power looks like at one end at one state to the other state across their whole domain because in general that sine wave should be nearly identical across the whole entire grid the United States is actually broken into three separate grids we have the east, we have the west and we have Texas like I said they're kind of their own entity and their own beast and they have a little bit different issues to deal with as well when it comes to power so that's what the synchrophasors are the synchrophasors are primarily being used by your transmission operators the people that are actually pushing the power long distances this information is also being sent back up to what we call regional coordinators regional coordinators are entities in the United States that try to work with the transmission operators and help the transmission operators balance out and make sure that we have a nice stable grid so there's about 15 different regional coordinators inside of the United States that help try to manage this power the next the buzzword we usually hear about are the smart meters themselves while these are definitely fun devices to play around on this is something that is a relatively minor a minor issue that we have compared to a lot of the other issues that we're facing with the smart grid so down here you can see that the smart meter in that bottom right hand corner the one right in the middle that's going to be an interface that is usually deployed on the meter itself it's not necessarily its own device in most circumstances for residential people when we get into larger deployments for corporations and more so even in industrial there's going to be a separate interface to be able to control and manage all the different meter readings that are deployed out there so this is where the meter is there's lines of communications between the smart meters that are being deployed back to the electric utility company to what we call a head end that's in the back of the electric utility company with this infrastructure traditionally for most of the people inside of this room most of you do not have the new smart meters on the sides of your house if you're interested to find out what type of meters you have on the side of your house just simply go out there look to see what the manufacturer is look to see what the model is it's usually very visible right on the front of the meter once that into Google and you can get spec sheets on any of these meters to tell you information about what capabilities it supports a lot of you will be able to go to the meters and you'll find that information you'll also see this little acronym that's ERT on the front of the face of the meter what ERT is is this is a protocol that probably the majority of us inside of this room have inside of our meters this is not the new smart meter technology this is a one way broadcast it's a protocol that the meters will go through in protocol and about every two minutes what their data consumption is so one way there's no way to be able to use this communication protocol to talk back to the meter and it's broadcast out using a 900 megahertz protocol that does frequency hopping well doesn't even do frequency hopping it randomly chooses when it comes up to it's time it's allotment every two seconds it randomly chooses one of 40 channels to broadcast that information on so that way it tries to avoid stepping on top of anybody else inside of the neighborhood that's what we have with meter readers they used to have to come up and read the meter themselves now they can just do drive-bys because they're collecting these ERT signals that are being sent back out there are other protocols besides ERT there's also some that transmit and communicate over the power lines themselves but the ERT is probably the most commonly deployed precursor or semi-intelligent meter that are being deployed right now in fact we call these meters AMR meters these ones with the one-direction communications that are out there as opposed to AMI meters AMI meaning the more intelligent ones that have bi-directional communications the AMI meters, the new smart meters their deployment is relatively small depending on the research that you see you'll see that deployments in New York from 10 to 25% across the US once again there's a lot of debate on the exact number of those smart meters out there and exactly what's the difference between some of the smart meters and what are not some of the smart meters so to dig a little bit deeper into the smart meters themselves that generates a lot of interest in general if we take all the detail information out and abstract this to a generic architectural view this is true for most of the field devices in general that we deploy in the smart grid just differences of protocols in the exact terms of the devices but when we look at the AMI meters themselves we see on that far left hand side of this diagram the electric utility company themselves and all their back end systems that's reading this information the very first device that's part of the AMI network these servers that are purchased with the meters are going to be these head ends these head ends usually are protected by some type of a firewall or some type of perimeter that they set up before we get out to the field devices then of course we'll hit some routers and the routers will then put it back out for the most part to the ISPs and the the telco companies that are connecting to them most of the meter data is going across cellular connections so you can see that the different links we have there the good majority is through cellular connections out to these meters we do have some proprietary third party offered RF towers we also had certain circumstances especially in the case of industrial industrial customers will have leased lines out to each one of these communications for the meters go down then these control signals when they tell the shut the power off at your house we'll go down to what we call the drop point or the aggregator for the network these aggregators are devices that are deployed out in the field either as a pull top device up on top of a pole or as a meter kind of a more intelligent meter that's placed on the side of one house inside of your neighborhood you can tell when you have these take out points these aggregator meters when they're on the side of somebody's house because in general these will usually be sticking out a few inches further than all the other ones inside of the neighborhood because they have to make room for that additional communications namely they have to make room for the cellular communication module that's inside of the device beyond that in some deployments in some of the vendors that have the smart meter products that are being sold they will have some of them will be deployed with cellular modems in every single one of the meters but I would say that the vast majority of the vendors that are selling here in the United States instead will have a meshing technology set up so that all the meters inside of the neighborhood will set up a communications mesh to get that data and assemble that data back to the take out point or the aggregator to push that back up to the utility company itself of those devices that have this mesh network it's rather interesting every single one of the vendors are out there while they may be using meshing technology and almost every single one of them is using a 900 megahertz frequency to do that meshing technology every single one of the meshing technologies and frequency hopping patterns of those 900 megahertz spectrum communications is different generally what the vendors are doing is they're choosing chips from TI and the other different types of communication chips that are out there that are generic off the shelf commodity hard chips for their boards they're taking these they are taking the configuration file that dictates the frequency hopping they'll build their own frequency hopping algorithm and the frequency hopping patterns based on a number of different factors by changing the frequency hopping they can change the amount of bandwidth and the amount of distance they can get out of some of the devices they'll tweak that to get the magic numbers that that vendor is interested in and then on top of that they'll build the meshing protocol and every single meter vendor out there that has a mesh network has their own proprietary mesh network there's no shared technology between the meshing technologies above that meshing technology we'll see a combination of vendors I would say probably about 50% of them are using a standard protocol for meter communication called C1222 each one of them of those 50% that are using C1222 even their implementations are different while they meet the specification to the letter they don't necessarily meet it to the point where it's interoperable with anybody else because of a number of different factors the other 50% of the vendors out there for communications they're going to be using a TCP IP connection either IPv4 or IPv6 depending on the vendor and then pushing their own proprietary protocol across it either raw C1219 tables which is C1219 tables or how each of the meters actually store their data inside of the meters themselves or they are going to be using some standard protocols they might be using some web services across it many vendors out there doing web services or XML data streams and exchanges on those meters but there's a couple that are doing that here in the United States of this area of the mesh when we think mesh wireless mesh at 900MHz most of us automatically think ZigBee at least in the field as I wanted to clarify I did mention that every single vendor is using their own proprietary one that means that nobody is using mesh for this meter to meter network neighborhood every single one uses a proprietary and most of these devices are using ZigBee which most of them do have ZigBee modules in them are between the meter down to the devices inside of our house so that's where the ZigBee communications come into play and so if you can get these slides later you can go through and see some of the protocols and dig a little bit more into it if we look at the payloads that are being sent across to give you a high level idea of what happens in the type of communications that pass back and forth between these devices notice that the first block of payloads primarily are communications between the meters themselves and the head end the one below it, the three lines below it at the very bottom of the slide these are pass through communications so these are going to be the communications that come from the ZigBee network inside of our home area network the devices inside of our house passing back to the utility company these will usually either be tunneled across the connection in some situations they will usually be data inside of some of the C-1219 tables that are being passed with whatever communication protocols are there the head end will simply pull out that input from that table, pass it back on to whatever devices need it on the back end for the most part a lot of the utility companies aren't doing a lot of those pass through communications yet it's there for future use some of the goals that they have is with demand response programs this is where a utility would give to the power that you're paying in order to allow them to have some limited control of some of your high consumption devices in your house like your AC unit during times of peak flow so for instance I live in Salt Lake City, Utah a lot of people up there the electric utility company Rocky Mountain Power give this discount if you have this device called Cool Keeper put on your house between your AC unit and the control unit of your house and they'll lie when it's the highest temperatures when it becomes a threat to the power stability inside of Salt Lake because we're having a bit of a power issue over the last couple of years they can go through and they can power cycle for small intervals the AC units inside of the neighborhoods and they'll actually do this in a coordinated fashion they'll take approximately one fifth to one eighth of the AC units inside of the neighborhood they'll cycle them down for five minutes or seven minutes and then they'll go ahead and let them cycle down the next fifth for that amount of time so that way they're at least trying to decrease some of the load inside of the grid and they've had a lot of success with doing this and as customers we appreciate that to some degree except for on the hottest days when they actually start turning the cycling on but otherwise it's these are some of the different types of programs they have expect to see additional demand response demand response programs here in the future on the meter communications themselves we have some basics of course we have the consumption data that's coming back to the head ends themselves we have control signals to be able to turn off power at the house part of the reason why they have these communications in there is when people move out it's always been a big problem to have people going and squatting inside of the houses and getting free power so now when you call up and say hey I'm moving can you go ahead and remove from the bill they can immediately just shut down power at your house when you move in you can call up and say hey can you re-enable power they can do that while you're right on the phone with them that one piece of functionality that remote disconnect is probably the greatest threat from a security perspective of these meters I would say the other major threat from these meters themselves is realize that these meters are collecting information and sending them back to a controlling server on the back end and any time we have data passing input data from a meter back to a controlling server on the back end there isn't a chance for an attack there I've never seen any good proof of concept code to be able to attack any of these head ends there's always a concern in my book and something that I think personally deserves a little bit more research that are out there seeing if you can take the few input fields that those head ends are accepting from the meters and seeing if you can get a buffer overflow or some other type of attack to gain control of that head end because ultimately that would give you the most control over these meters themselves there are other attacks out there with vulnerabilities if you can find the right combination of vulnerabilities you might be able to get in and try to control some of the other meters remotely you might be able to communicate on these mesh networks or if you repurpose one of the meters and try to make it so it communicates on it if you can get the right combination of vulnerabilities you can do things and attack the meters remotely the good thing is is most of these meter manufacturers are on their third and fourth generation products right now if you go back and look at the first generation products it was a huge nightmare and very very much a possibility to be able to perform these types of attacks with the current models right now with these third and fourth generation devices that are currently being deployed it's a much harder attack surface to be able to find that right combination of vulnerabilities to attack that infrastructure most of them are doing a fairly decent job now with their security infrastructure and security architectures okay so attacks and defenses that gives you a little bit of idea of what the architecture looks like so let's talk a little bit more about some of the attacks that we perform as penetration testers and some of the defenses okay so this is an oversimplified chart realize this is not what a utility company really looks like this is highly oversimplified this just goes through to show that we have client side attacks, server side attacks network attacks just like any other industry we do have that additional vector of network attacks in the field devices and the hardware attacks on these devices as well so really fast client side attacks it is a threat honestly out of all the attacks are out there inside of the smart grid all the attacks that we can do to the meters on our homes all the attacks we can do to the big iron products in the transmission operator substations that are out there at the end of the day these are the things that personally keep me up at night the things that I'm most worried about because electric utility companies are just like any other organization out there they have internet links they have clients that are surfing the web and while they're for the most part they're control center technicians and control center operators they don't have direct internet access on their workstation they do have connectivity to services and different devices inside of the organization so if an attacker comes in through the front door and sees a presence inside of the organization it's only a matter of time before he can get his way and work his way to the right workstations and the right subnet to be able to gain access to some of these servers so that's what keeps me up at night I personally think the day when we see the biggest incident from attackers that affect us from the smart grid I personally think that the front doors that launch this attack will be coming through a client side attacks server side attacks nothing different here same vulnerabilities realize that all these control servers that I showed you the ones that are controlling all the communications for the SCADA networks and for the AMI meters this is commodity operating systems on commodity hardware so the things that everybody in this room does for penetration testing and devices your knowledge is directly applicable here for the server side attacks and just like any other industry we have a lot of the controlling interfaces are moving from that client type control interfaces and serial and terminal they're moving to web-based interfaces as well so things like cross-hidery press forgery immediately become problematic and for those of you that do web-pen testing you know cross-hidery press forgery is out there and probably 95-98% of every single web interface you ever touch so it's been a huge issue and this is something that's a very valid attack angle for us network attacks in the network attacks when we're looking at the field devices inside the organization it's just like anything else most of the protocols that you saw in the earlier architecture diagrams that have communications most of those are using web services and other types of very common protocols that we're used to dealing with when we get to the field devices we get a lot of proprietary one-off protocols especially when we get to the substations because for a lot of the devices that were deployed in the substations in the last 20 to 30 years a lot of them were custom built for each one of those utilities and so they're very, very customized proprietary languages even when we have standard languages like I told you C-1222 for the meters each of the vendors are doing customizations of each one of those protocols as well so when we look at network attacks for the most part when we do penetration tests if we have enough time and enough budget we will go through and try to reverse engineering the proprietary protocols that are out there but a lot of times just like it always is the case when we do pen testing we always want to at least do a very, very simple check of the network communication protocol for some basics some of the base that we're going to be checking for is going through and checking for the cryptography that they're using checking for encryption just trying to see if they actually have encryption enabled at all other thing we often see is for the wireless communications is a misconception that frequency hopping is one of the security controls frequency hopping is not a security control with the right equipment and enough time you can backtrack and trace what that frequency hopping algorithm is and bypass that it's more of a method for obscuring than anything and ultimately the reason why we do frequency hopping isn't for security it's ultimately be able to provide higher quality of communications we also see problems with communications with cryptography because a lot of the protocols that we're used to dealing with cryptographic protocols in normal IT simply are too heavy handed and the embedded devices that we're deploying don't have that capability so we have very limited capabilities for some of these devices so a lot of times we're going to be messing with symmetrical encryption because it's a lot easier to have these embedded devices use it we do have some asymmetrical protocols out there but there's very few asymmetrical options that we have at our disposal so when we're dealing with these protocols one of the first things we're interested in is how they do their key updates and distribution because that's one of the biggest weaknesses in Achilles heels to symmetrical encryption when we get a proprietary protocol that we've never messed with before never seen before the first thing we're going to do is if we're going to try to determine whether it's an encrypted protocol or not and if it is encrypted try to determine how encrypted it is so here's some charts just showing you different ways to determine you capture the packets, you strip out the header information, take just the payload we build a histogram checking the entropy of the data inside of the payloads themselves the graph on the chart the graph on top is going to be this is the prime model that we want to see when we want to see encryption something that's very evenly distributed out the one on the bottom is still encrypted but it's not as good as in this encryption mechanism because we don't have quite the even number of balance and if you ever see when you're doing this histogram in a comparison a large jump right in the very middle and it right back down usually means you've come across an ASCII-based protocol because you hit the big jump right in the middle of the ASCII characters and very very few representation anywhere outside of those common ASCII characters so another issue we have is in the web world we have products, vendors that are selling web-based products that say yes we're secure because we use SSL and TLS we have the same exact problem that we have inside of the embedded world we have vendors coming up and saying yes we're secure because we use 8 yes so one of the first questions we're always going to ask is what types of psychromotor are you using for your 8 yes and trying to start digging down into the architecture there of the hardware attacks the hardware attacks does represent not just the meters themselves this can be performed by any of the pull top devices like the aggregation points or the feeder switches this also takes effect of the RTUs and the different devices in the substations themselves of those hardware attacks most of these are susceptible to different types of physical attacks themselves either just getting in and trying to get information back out or making some modifications in the hardware themselves a lot of the vendors that are out there have minimal capabilities or minimal controls around trying to detect when people are tampering with these devices but for the most part they're fairly fairly weak when we are trying to attack each one of these hardware the purpose for us to perform a penetration test on one single piece of hardware isn't to compromise that hardware just like everybody in this room knows when you have physical access to the computer we assume everything on that computer is compromised with some minor exceptions and some edge cases same thing with the meters themselves and any of the hardware field the embedded hardware devices deployed to the field we get our hands on these devices not to show that we can attack these devices and get information off these devices and try to get information to enable us to attack the other devices inside of the infrastructure some of the things I'm going to be looking for are the cryptography keys the asymmetrical or the asymmetrical cryptography keys that are stored on these devices because with access to this information I can go through and I can attack the other devices inside of the infrastructure either directly let's say I can get the infrared password because all the meters have that little infrared interface it's an infrared interface so I can go through and just very quickly launch attacks on each one of the other meters inside of the neighborhood but what I'm more interested in is getting the keys for the wireless mesh communication protocols so if I can get the asymmetrical or the symmetrical keys for that and I can find a vulnerability in the way that they've been implemented I might be able to launch other types of attacks like impersonating another meter or impersonating the head end and sending control signals down to the other meters from my device the other thing we have is this is something we've had forever it's not a new thing ever since they've had power people have been stealing power from the meters themselves there's a lot of information any of these terms you'll be able to get some details information about how people have done this in the past and some of the attacks this is basic stuff simply trying to steal power that's not all that interesting to us but it is kind of fun to talk about especially when you see really cool pictures like this of people stealing power piggybacking multiple meters together ripping the meter out just getting in and actually hard wiring communications back to one single meter or usually outside of that one single meter so it's not being read at all we also have physical bypass a lot of these devices are being tried to protect with different controls like locked cases and lock boxes on them and fences and perimeters and security cameras and all these things of course locks we can pick the locks fences we can climb the fences cameras well number one if they have cameras number two if they have cameras if they do monitor them how long does it take to roll something out if you get a good attack and you can actually replicate this attack and make it very easy to go through and launch you can go to each of the different substations launch your little attack get back out within a 10-15 minute at least give you some remote presence and wireless remote presence inside of that substation to be able to go through and continue your attack down the road so these are things that we're trying to combat some of the things that we're trying to address inside of the smart grid when we're working with the hardware itself told you we're looking for keys in the hardware we want to go into these keys and try to extract these keys out two methods that we use for extracting these keys out is either going through on these hardware devices and identifying where the data is stored at rest so the eProms the RAM chips the flash memory inside the chips the onboard storage of the system on the chips and the microprocessors trying to gain access to those and dump the data back off of them another thing that we'll do is we'll go in and identify communications between the microprocessor and other keys like the wire or other chips on the board like the wireless board and we'll jump in and we'll sniff the bus communications off those devices trying to find those key exchanges this is something that we'll occasionally find is in order to set up the encryption if a lot of the encryption is being done by the microprocessor and the RF chip itself is just a very very dumb chip building the median for it to communicate on we might not be able to get any data off because the data is already coming encrypted but if that RF chip has more intelligence and is doing the encryption itself quite often the data is being passed plain text in a serial link across that bus to the chip and quite often that key is also being sent clear text over that chip as well because the key is usually stored on the microprocessor itself so if we can capture that key back off we can get access to the cryptography keys for the communications channel so these are just some pictures of going in showing how we jump in on some of the different devices with different wires to it using syringe is that one thing on the left hand side that's a big syringe that we use it makes it very easy to use a syringe to go in and get some of the small surface mounted pins that are on the devices themselves occasionally we have to go through and make some minor modifications to try to get chips out of the way from interfering with our communications and try to stop the attacks that we're doing we avoid this one possible it's not always possible here's just a little screen shot of us going through usually using either the I2C or the SPI communication buses on these chips themselves so basically bypass the microprocessors, get our own hardware and straight into that e-prom or the flash module and dump the contents out here's an example of bus snooping identify the bus, jump in between the two chips that we're trying to get to capture the information from and capture that information so we can go back and do analysis later once we capture that information we have to identify the information that we want out of the side of this dump because a lot of times these dumps are very very cryptic two different ways to do this if we have symmetrical encryption keys we do attacks very similar to the attacks that they did with the Blu-ray disks where we systematically go through and take the exact length of the key that we know that they're using and we systematically step through the memory dump until we find a key that successfully decrypts whatever traffic we're trying to decrypt if you see here in the very bottom this is a combination of Travis Goodfet's Goodfet tool and Josh Wright's Killer B protocol that we do this attack in when we're dealing with ZigBee so dump the memory contents off of a ZigBee chip or grab it straight out of RAM we then use that binary dump to step through it until we can successfully decrypt a ZigBee packet asymmetrical is even easier asymmetrical keys are randomly pseudo-randomly generated so all we do is we go through an entropy analysis of the dump we have and as long as the rest of the data on the device is not encrypted you can very easily decide exactly where that asymmetrical key is so you see that one spike right in the middle that was the asymmetrical key inside of this dump so they're fairly easy to identify once we go through and we have this information once again the goal is to try to leverage this for other purposes some of the defenses we're recommending for the utilities is try to use system on a chip as one of the best defenses they have as well as try to limit the cryptographic keys that they're deploying out to the other devices another item that we're going to be using or looking for besides the key itself is the firmware because with the keys while we can get the keys as all great fine and dandy if we can get the firmware itself the firmware gives us capability and greater insight to these devices and you can go through and use these or binary binary decompilation and go through and analyze the binary itself or do source code review on the device those are two different options if you have copy of the source code if you are going to be doing binary analysis on the flash it does become a little more problematic than most of you do in binary analysis on common everyday computers because each one of these embedded devices has different microprocessors with different instruction sets so you quite often will have to go through and build decompilers to try to simply gain access to the instructions in the first place so that's a huge obstacle that we come across and for those that like writing decompilers we could definitely use a lot more of them out there for the embedded chip space so great huge opportunity for you to get into other than that for conclusion yes the smart grid is out there the smart grid does have security issues out there no industry is perfect but at the end of the day I think we are moving forward we have a lot of people looking at these devices and we can always use more for those that are running security shops out there and providing security services realize that there is a huge potential inside of the smart grid working with the vendors of the smart grid as well as the utility companies that are buying these products your skills are directly applicable in many of the different areas and for those that want to learn new skills there is a lot of new skills out there you can pick up with some of the hardware hacking techniques and some of the proprietary networking protocols that are out there so a lot of good information out there if you want more information you can go through and check out a couple different resources throughout the slide deck I try to give resources wherever I possibly could if you want more information about some of the attacks specific on my previous employer in Guardians they do have on their website an attack methodology that goes a little more into detail about some of the hardware attacks that are out there that's a great source of course I did mention the NIST interagency report that was released last summer you can go through and grab that the URL is up there and then also if you go to smart grid at pedia it's a great place to get information and specifically on smart grid at pedia there is ASAPSG one of the work groups that I've been working on for the last two years all the products that we create and all the documents that we create to try to help the utility secure their infrastructure so that they can be publicly on this website you can gain access to any of these and get more information about any of the specific domains that are spoken about today other than that, thank you very much I will go ahead and take questions on Q&A but thank you, thank you


Minnesota legislators needed three days of overtime and nearly non-stop negotiations to finish a 46 billion dollar budget early this morning. Governor Mark Dayton said he wasn't sure if he would sign all the pieces, but conceded it might be the best deal he could get from Republicans. The overtime work was slow to get going at the capital where Dayton and leaders got hung up on final details of the broad budget agreement that would put 650 million dollars toward tax relief, free school offerings by 50 million dollars, and dedicate 300 million to fix roads and bridges. Governor Dayton said he was genuinely undecided and that he is weighing his concerns about a tax bill and the state's financial stability against the risk of a possible government shutdown if he vetoes the bill. Governor Dayton said he plans to make a decision by Tuesday. If you've enjoyed this segment of Lakeland News, please consider making a tax deductible contribution to Lakeland Public Television.


What's going on, everybody? DottoDoy here with another Dragon Ball Fighter's rank session video. As always, let's start by looking at where we are. Oh, cool, Super Saiyan God. We are number 890 at a Super Saiyan 2. Oh, I guess I'm not a Supreme Kai. We'll see what happens in the first match. Who knows, maybe I'll get Supreme Kai in that one. Let's see, Slothasaur Supreme Kai. Yeah, see it? It says I'm the Supreme Kai here, for some reason. Slothasaur. With 400, how much? 408,000? All right. I am supremely tired, by the way.  I have plumbers coming over. And for some reason, they flipped my thing that now it says cold when it's hot and hot when it's cold. And I did not know that when I was awake in the morning. It led to an extremely hot shower. Oh, let's go. So, I played a lot of games on the Xbox today. So, if my execution is a bit off, that's probably why. Okay, up we go. Hey, Yamcha's Assist is so good. Especially for converting on overheads. I don't think it gets any better than Yamcha's Assist for that. So easy to combo off of it. Do it, do it, do it. All right, very nice. Ah, Candy Beam. Goku Blue has such a good auto combo. Okay, with that. Jesus, what is this guy doing? I fell so slowly. All right. So, this is not the highest quality match so far. But hey, you know what? We got seven bars. We got TN. Let's get stylish with it. Oh, wait, hold on. Hold on. We got a Goku Black to deal with. Ah, very nice, very nice. Again, still terror. I should probably take a day off to like practice that.  I said that was the one thing I had going for me. I'm out. I'm out. I'm already out. I was not out. The fun fact was not out of that combo. In that explosion is a dying TN. Hey, but Goku Blue is back, everybody. You have to be extremely careful against the... Oh, but we went low this time. Let's go. Okay, and this time we're going to go for the overhead. Continue that with the Yamcha Assist. And this is game... Oh, guys, I messed up my execution. Guys, I ruined it. It's all ruined now. All right, we got a stall for time. Oh, man. All right, that down heavy has a crazy hitbox. We got to buy time here. Get off me. Oh, no. This is bad, actually. I'm just going to pop this right here. Let's buy time. Stall, stall. Oh, whoa, whoa, whoa, whoa. What are you doing? Reaction. Oh, no. No. This is why you don't try to re... Don't get cocky, kids. Don't touch me. Oh, that's some Dragon Balls. Wow. Okay, that got rough. So what did we learn? We learned I crack under pressure not to sleep on his Goku Black and, yeah, finish the job quickly. Oh, yeah. Don't mess up your combos. I could have ended that game. How did I mess that up? Very nice. Oh, no. Oh, I meant to go hit the medium button and I accidentally hit both the... Because my stick layout is weird here. Whatever. Sparking Blast, Marking Blast. Who needs it? Oh, I need it. I need it. Let's get him in the corner here. And by him, I mean TN, of course. Very nice. Oh, what? Thank you, Chatsu. See you in the locker room. Whew. What a game. Oh, what? You think I'm that predictable? You're right. Tie it up. Tie it up. Well, it's almost time. I think I'm still behind. He has four bars, too. Okay, very nice. What do you think? Empty jump low? Oh, I can't believe that connected. Let's go. The empty jump low worked. I just didn't convert off of it. Okay. Another one. Another one. I got to buy time. This is the best part about Yamcha. Just buy that time. Just nice and slow. Do it one more time. We're gonna buy that time. Yes, yes. Go ahead. Hit Yamcha. Yeah, so level three Sparking Blast. You just want to wait that stuff out, usually. Oh, that's bad. Nice reflect, Yamcha. Good looking at. Oh, that hitbox is so insane. Whatever. Oh, got myself in the corner. He has a lot of bars. You know what time it is. Oh, I don't have three bars yet. No, Yamcha. Yeah. What? Oh, no. Oh, no, he has six bars. I have to play so safe. Geez, this is bad. No, very nice. Very nice. Ah, damn. Shouldn't have vanished. I knew he was waiting for it. Okay. I feel like we went over this the first time. Don't get cocky. What did I do? Get cocky. And that cost me a whole 2,400 points right out of the credit card. Boom. That doesn't work. I should probably learn my combo off that. Oh, nice. Very nice. Let's try to take this game very seriously because I do not want to lose again because I get cocky. Oh, that hit. This guy is very... He knows that I'm a man that likes to vanish. Oh, he knew I wanted to switch up. I will take that all the way to the bank. Yeah, I'm killing this. I'm not going to let this one live. TN, the finisher. So good. Goku, thank you. Take a break. All right, let's try to... Oh, I am...  I should not have done that. I got to play so safe against this Goku Black. Ah, so bad at that. Not out, not out. So nice. I'm not going to vanish. Not going to vanish. Oh, Yamcha, no. Okay, he got out. Oh, yeah, TN's dead. That's fine. That's fine. That's fine. I think if I hit a nice sparking glass combo here, we should be fine. Oh, damn it. Okay. Realistically, I shouldn't have went for it, but I thought it would look stylish. But again, we have to remember the lesson of these three matches is don't get cocky. I got 3,000, so I got positive 1,000 points from this. I also really need to work on my not getting mixed up. Man, look at this. If this wasn't a rank series, I'd hop in here right now. Oh, pink square, let's go. I'm trying to cop a beating. Uh-oh. Here comes that beating I was talking about. 279-177 wins. Damn, I'm jealous of all these people with so many matches. I'm trying to hop in here, get these many. I'm trying to, what am I saying? I'm trying to get those matches. Uh, skip intros. Don't want that d-sync. Oh, go hunt. Wow, we have very similar teams. All right, you want to exchange TN beams? Going low. This guy's already seen all my tricks. How can I beat this guy? He has so much experience. Probably should have just stuck with my game plan there. Whoo, how was I not blocking that? I guess that was maybe an overhead. Maybe I was holding down back. Oh, I'm dead. Uh, all right. You got the level 1. I wasn't blocking that. Okay. Okay, that one? That one. That one I feel like I blocked. I didn't reflect that. All right. Nice, nice, nice. This is why they call them the pink square because he gets the job done. Let's go. Oh, very nice. Okay, just hit him with that. Hit him just enough so that he slides down. Um, ah, the back dash. I was feeling it. I thought, I thought it would work. I'm not going to lie. My yonche experience isn't the best. Oh, I tried jumping. Don't want to get hit during that. Oh, I recovered. Okay, yonche is dead. I mean, what?  Oh, man. Did it lag or did I just absolute, or did my vision go away? Oh, very cool. I think I'm ready for my rematch. Now, what am I saying? Tian got this. The 1v3 comeback is what Tian was made for. To make him ain't no fair. Oh, I wasn't guarding there. I was mashing a vanish.  Let me get, let me cop that rematch. Let me get that second beating. Oh, very nice. He ran under there very quickly. He was like, oh, let me get, I clicked the button. No, I wasn't blocking. I lost the character so early. Oh, no. I should have saved that bar to guard cancel. Oh, don't on ray. You can't cross up with assist on the screen like that. Okay, so that's not safe. Pretty, I think that was a mistake, maybe or not. Oh, whoa, whoa, whoa, whoa, what are you doing? I would have gotten hit by that reach of TN. Oh, no. Damn. I missed my guard switch. Guard cancel. Hey, TN versus the world again. Oh, I'm dead. Geez, that was nice. I want to run back if really badly. I lost Goku blue too early on both of those. Yes. The rematch. Space police pulled me over, took my points. Okay, the rematch. The match to end it all. Here we go. What did we learn is the test.  Okay. So far, it looks like we learned nothing. Okay, I'm going to guard low. We know that. Oh, that's the safe. Let's go. Oh, I always know that's coming, but I can never do that right. Well, I was guarding low. Oh, very nice. Ah, it was a little obvious. I can always feel it whenever I click a, wow, I didn't click a single button. Oh, I'm dead. I keep losing characters too early. I have five bars that I'm holding on to for, I don't know. Okay, I got to get him back out here. How's that a reach? Oh, please tell me I put my finger up. No, I did not. Ah, man. Okay, I just needed to get him off me. Okay, I got to start hitting those. Ah, I am so bad at that right now. Okay, let's see. Ah, okay, got to work on that, got to work on that. Punish that and those. I might have a chance. What are you doing, Yavcha? Okay, okay, okay, okay. Ah, that was risky. I'm dead. Yep. All right, all right, learned a little more. I ain't above learning. I can't lose a character too early. I got to use my meter and I have to, yeah, I think that, oh yeah, punish the, okay, now we're playing my game. Yes, all right, feeling myself. Oh, he blocked that. All right, I'm not, I'm not sure I'm a fan of this Yavcha versus Yavcha matchup. Okay, guard cancel. I thought I'd be able to punish that. Oh, if I could have punished that, that would have been very nice. Oh, I was hoping the volleyball fist would eat a hit. I did not want to do that. I did not want to do that. No, come on, I can't lose Tien now. No, he's going to kill me. Ah, man. No, six bars that would have been so perfect on Tien. And I used my sparking. Okay, we still have the lead, but I do not have my sparking, which would have been very, very nice to have. Oh, no, that was very bad.  Hey, let me live. No. Very nice. Sparking blast lets you dash on block. Oh, I forgot about that until just there.  Okay, we are, okay, we have a chance. We have a very good chance, but this was not a clean round. I'm not happy with my performance. It's bad. I'm going to get Wolffang Fist. No, I hit a button now, I'm dead for sure. No. Oh, so unsafe. Oh, so unsafe. Ah, no. Oh, he's a demon now. Am I even going to match up with that guy? Ah, man. Oh, I lost Tien. I played very unsafe with Goku Blue. There was a lot I could have done better. Oh, you know what, that is, that's the, you see, you learn more from your losses than your wins. And I learned a lot right there. What a beating. I copped a fine beating right there. I'm not doing the outro because I'm waiting to see if I get matched up with that guy again. Why do I do this to myself? Who is this? Well, I hope you enjoyed this video. Hopefully it showed that I have a lot to learn in this game and I'm still pretty scrubby. I hope it also showed that you learn a lot more from your losses than you do from your wins. As you can see by the copious amounts of losses I got in this video, whoa, whoa. Are you me from another future?


Flooding is one of the most devastating natural disasters, causing death, destruction of property, and damage to infrastructure. Flood modeling is an important tool used to mitigate the effects of floods by accurately predicting their course and magnitude. This paper summarizes the current state of flood modeling, its challenges, and potential future directions. It also discusses the use of advanced technologies and integrated models to improve flood risk management and reduce the impact of floods on society. This article was authored by Vigendra Kumar, Kul Vibhav Sharma, Tamaso Kaloyaro, and others.


I'm moving on to outside Africa. The Iraqi government has condemned the U.S. air raid against Iran-allied fighters along its border with Syria and said it will study all legal options to prevent such action from being repeated. The cabinet, headed by Prime Minister Mustafa al-Kadimi, held an emergency security meeting the day after the U.S. strikes, which is called the violation of Iraqi sovereignty. It was a rare criticism of U.S. action by the government of Kadimi, who is seen as friendly to the United States and has tried to check the power of Iran-allied militias. Iraqi officials say they want to avoid being dragged into a tit-for-tat escalation between Washington and Tehran.


First of all, thank you Hyperledger for inviting me today to speak here at the Hyperledger Global Forum. I'm Robin, founder of Distribute, which is a specialized IT consulting company that helps connected industries with the digital transformation journeys by utilizing distributed technologies. I got in touch with Hyperledger the first time when I was still working at Daimler where I built the mobility blockchain platform together with four European blockchain startups that was just spinned off out of Daimler by my ex-boss, Dr. Harry Barron. Today I work mostly on different activities in automotive pharmaceuticals and supply chains and I'm still in touch with Hyperledger as part of my activities at the Mobility Open Blockchain Initiative where I help as a technical lead to connect working group efforts with the deployment of what we so-called Mobinet, a joint service infrastructure for future mobility. If you're interested in that, you can also find a speech that I've done together with Jim Mason in the Hyperledger Public Special Interest Group. Before getting involved with Enterprise Blockchain, I was a development engineer at Daimler Mercedes-Benz where I mostly focused on the development of autonomous vehicle concepts for future transportation solutions. My first exposure was decentralized technologies and distributed systems was actually in 2014 when I was still a graduate researcher at National Taiwan University and working on payment mechanisms for intelligent power networks. But today we don't wanna talk that much about automotive or power networks. We will want to take a look on integrated supply chain ecosystems and how open and trusted data sharing can lead towards more integration in the industry. So, I mean, the corona pandemic has shown us to be not only a crisis of public safety, but also a crisis of collaborative data sharing. It's clear that the effect of the corona virus in society and the global economy was unprecedented with serious consequences for the entire industry. Similar to past crisis, again, it has not only questions of public supply systems, but also data sharing mechanisms across companies, institutions and economies. Taiwan, where I've studied, for example, has demonstrated how to protect the country against the outbreak of the virus by utilizing more transparent and more integrated data sharing mechanisms that can help economies to protect its industry during times of uncertainties and crisis. So, what is the problem that industries still face today with supply chains becoming more and more integrated? And I mean, it's clear throughout all the industries that we are living in a world that is becoming increasingly complex and interdependent, not only across companies, but also industries. And what the pandemic especially has shown here is that most companies still think siloed rather than understanding themselves and their supply chain processes as part of an entire value chain. In contrast to supply chain of a single company, though, a value chain describes all the activities that are needed within an entire ecosystem to create a product from its early raw material inputs to the delivery of the finished goods to the customer. So, there definitely is a tremendous potential for integrated supply chain ecosystems to power economic growth. However, industries still seem to struggle with sharing data across multiple companies as a key driver of those ecosystems. So, before we are talking about the struggle and the problems that I have seen working together with these companies, let me provide you with a concrete example from the automotive world that I came across or worked together with BMW as part of the supply chain working group at Moby. So, supply chain networks in the automotive industry deal with numerous participants and parts on multiple tiers that are spread across several geographical locations similar to the picture that we've just seen before. Due to their high complexity and more and more government focusing on data protection, automotive supply chains often lack transparency and responsiveness. However, today, legal authorities such as the Federal Motor Transport Authority of Germany or the National Highway Traffic Safety Administration, the NHTSA of USA, start to require more end-to-end traceability of parts in order to not run again into scandals like the diesel gate that we have experienced over the last years, right? So, this is particularly important for the identification of defective and then manipulative parts and recalls that involve multiple suppliers around the world. So, what you're seeing here basically is the part chain solution, which BMW proposed as a decentralized supply chain traceability application that enables the creation, monitoring, and sharing of a unique digital representation of all physical parts across a supply chain network using hyperlegia fabric, right? So, what you see in the bottom is the physical parts. Then you have an interface and what BMW proposed in their solution and demonstrated in the study is basically the digital backend of beckons or the enterprise blockchain solution where they have shown that enterprise blockchain technologies or hyperlegia can indeed help with trusted data processing and sharing of critical automotive parts across various supply chain participants in a reliable and immutable manner. So, integrated track and trace applications already exist, but how come they still struggle to being adopted, right? So, I mean, data sharing is not new or a new idea in supply chains. However, most of the past initiatives focused mainly on bilateral supply agreements and point-to-point data sharing. So, although we may have beckons or a business support system, a decentralized business support system for a supplier network, integration and interoperability by setting industry standards is still far away, right? So, it's not the technology itself so much, but companies that I have worked with are mostly concerned about protecting confidential data defining a common data standards, which basically are centered around data operability and data control. I mean, having talked to various clients, what I've seen over and over again is due to the changing dynamic and real-time data requirements of global supply chains. Participants really struggle to align on dynamic data ontologies and industry standards. And mostly, that makes them often go back to negotiating standards bilaterally that then don't help ecosystems to scale. Besides, I mean, we have, but I think especially in the enterprise blockchain world, we are getting better and better by this. And now we have understood that supply chain ecosystems by nature are more than decentralized. And the blockchain enterprise solutions, such as Hyperledger Fabric can definitely help with a consistent view of data and lineage across multiple organizations by utilizing blockchain logic. And this brings us actually to the opportunity that we have right now, I'm saying. We have understood that solving the integrated supply chain problem is not necessarily something that you will get done with a centralized platform, right? So advising different customers and industries on next generation IT technologies, I learned that the success of supply chain ecosystem lies in cooperatives, which depend on a clear definition of joint and mutually accepted business use cases that arise from these ecosystem activities. As industry consortia around the world share similar business and governance problems that you see here, they have together come up with best practices or best practice sharing within already established industry consortia. In Pharma, you see Pharma Ledger, in the mobility world, you see Moby, in the Telco world, you see Sieben, and in the German industry around German automotive makers, you also see the Katena X automotive network and Gaia X coming up right now. And I mean, those organizations by now already have helped global supply chain ecosystems to accelerate by the standards they have defined. What we need to do today is we need to leverage the work those organizations have done, encourage a best practice sharing between industry consortias that I see very important to develop globally accepted standards and mechanisms for cross industry data sharing in the future. But setting standards is only one part or only one side of the coin, right? So it's all good that standards have been developed but they only get being implemented and being adopted when they move from paper to code. So what I want to encourage the Hyperledger community that we have here, where we have incredible developers and so on is to work together with cooperatives to find a way to transfer the knowledge that has been defined through industry standards into code and to repositories where we can contribute together. This is what I wanted to share with you. So it's blockchain, enterprise blockchain is not a single, like a single sports as a team sports. So it's Hyperledger. So let us connect, let's come together, let's contribute code together, let's connect standards. And that's why I am at least here and I want to thank you for tuning in to my presentation. If you have any more questions or want to talk bilateral, feel free to shoot an email to ourpillingatdistribute.com and I'll see you later.


What is the ugliest shoe of all time? I don't even know if it's the ugliest, but it's definitely the new Air Force and everybody's wearing it, and I'll never wear them. Panda Dunk Lowe's. The Panda Dunk Lowe's? What is the ugliest sneaker of all time?   Like, if it's fuzzy, that's weird. Like, no.


Hi everybody, welcome back to another technical demo around the Quarkus application. In this demo, I'm going to showcase how to make this truly tracing integration with the OpenTelemetry and for serverless applications based on Knave and Quarkus. But first of all, before we get started with the demo, let me try to talk a little bit about why we need OpenTelemetry and how that matters. In the cloud-level architecture, you got a bunch of applications to deploy in Kubernetes or hybrid cloud, which makes sometimes a broad operational challenge. For example, how to solve availability and performance issue real quick. In order to solve those challenges, Telemetry is a key role for the providing observability. Traditionally, Telemetry data has been provided a lot of opposite projects, but also emotional vendors. This is causing another standard issue. Telemetry enables to have a single and vendor agnostic solution, and this project also brought industry support and adoption for cloud provider, vendors and users. And then you can also, in the end, for Java developer, Quarkus also integrates this capability OpenTelemetry with one of the extensions for your cloud application deployment. And then moving forward to serverless architecture with the advantage of the application flow. Let me enable one of the common popular projects to make your existing Microsoft application and serverless. There, this video teaches how to trace your district Microsoft application specifically serverless application with the Java framework. Let's get right into the demo, how it works. Okay, this is my sample application based on Quarkus. I'm going to use the Quarkus-based version. And here's Dean, the Quarkus actually provides OpenTelemetry Explorer extension. Just add it, you can add using a given command line or Quarkus CLI, or you can just select and download G-File from code.quarkus.io webpages. And then we're going to add the Quarkus OpenShift extension as well. In the end, we're going to deploy this application to OpenShift cluster for K-native service for serverless application. The server application just is a greeting resource to Java file to handle RESTful API. As you can see, there are a simple application hello, and then just bring a log file, and as well as return text, hello from REST Easy Reactive. And one of the beauty of the Quarkus, you can actually use Reactive programming by default based on REST Easy Reactive extension. Right, I'm going to add a few more RESTful API in this example to trace this application implication from RESTful API. So I'm going to add a new path, Ola, and return Ola Daniel, my name, and the same return text, to add a logo print out of the same. But let's try to add one more RESTful API here. Let's say the new path is a greeting, and then the method name, same thing is greeting. Oh, here's maybe typo, but just I'm going to greeting, and then here to return the hello, Quarkus OpenTelemetry, and welcome Quarkus OpenTelemetry to the same text result. So here's the application property, how to define the OpenTelemetry for tracing. So as you can see, the application name, I'm going to say my service, and then I'm going to enable OpenTelemetry feature and also here the explorer endpoint. So here's the OpenTelemetry gathers all metric data from your source, for example, Quarkus application, or even IoT Edge device, and then it will send it back to the tracing server, for example, Yeager. We're going to use Yeager in this example in local environment as well. So here is localhost 4.3.17, OpenTelemetry controller. We're going to run OpenTelemetry server as a Docker container locally in a minute. So here's the Docker composed by a how to stand up Yeager server and the collector based OpenTelemetry. As you can see here, Docker composed has two container specifications. First container Yeager tracing with the explorer ports here. And another one is OpenTelemetry. And it also exports a few ports here and one of the TPC receiver and the 4.3.17 port we just specified in the Quarkus application can also mount OpenTelemetry collector configuration, which you can specify receiver and the process explorer and so on. So let's take a look at how we define OpenTelemetry collector configuration here. So we can see we're going to use the receiver by default OATP protocol for the RPC. And the other one is HTTP protocol, something like that. It's a similar concept HTTP for OATP 2 protocol. And the last thing we're going to specify the services based on receiver processor explorer and so on. You can actually specify the receiver and extension and the services explorer based on your back-end aggregation server and tracing server. You can actually use a Zipkin server for the receiver, for example. Okay, so here's my local terminal window. Let's try to run Docker composed first to start Yeager server and OpenTelemetry controller. So I'm going to use the Docker composed command line and it will take some time to run up just like that. And you can see Yeager server and then OpenTelemetry controller just to make sure it's running. And then Docker PS showcase the OpenTelemetry controller as well as Yeager server running on my local. Next thing what I'm going to do is I'm going to try to access Yeager web console for my local. Then you go to Yeager UI. There's no service editor moment when you try to reload it. It instantly showed the one service which is just Yeager query by default. Now I'm going to run my corpus application using corpus CLI. You can also use Babel CLI as long as you want to deploy in grader application. Okay, so corpus demo is running. The press W from the corpus runtime terminal it automatically open landing page as you can see and visit the web UI just for fun. Then when you go to one of the extensions here, OpenTelemetry is still an experimental feature but it's worth showcasing this demo at this moment. And the click on configuration in one of the OpenTelemetry extensions you can find all the key and value you can specify your application side to define OpenTelemetry controller which is really good. And then I'm going to reload my Yeager console UI and you can see my application which is defined by application file. My service is automatic startup. So this application and my service and then you can operation all and then there's no operation or when you define in my RESTful API. So let's go back to terminal window and then try to access our RESTful API and then switch another terminal window and then try to HTTP pi tool and access the hello and it will return less easy reactive and go back to Yeager UI and press Yeager UI and then go to operation you can see new RESTful API automatically detected which means in behind the scene OpenTelemetry automatically detects the new trace telemetry data from the application and then it was already sent to tracing server which is a Yeager automatically tapping. So when you're looking into my service and then you can find OpenTelemetry actually aggregate this telemetry data from my application. So let me try to call the RESTful API and go back to Yeager UI what happened in the next and I'm going to try to ask the new RESTful API OLA and including and you can see OLA Daniel and the welcome by text we go and then when you reload the Yeager UI you combine another two RESTful API operation here and then the same one trace and one trace because I just invoked one time. Let me try to call one more time for the OLA application and then you can see back to the Yeager UI and reload the Yeager UI you can find immediately another trace just gathering into Yeager spaces which is really cool. So now I'm going to try to deploy this application to Kubernetes which is I'm going to use OpenShift cluster for 10. Then here is I already created a namespace which is a project in OpenShift as you can see QuarkStage OpenTelemetry and then one of the good things of OpenShift cluster is allows you to install OpenTelemetry controller using operator. I already installed operator and also I already installed history tracing platform which already include the Yeager server.  Here the discrete tracing platform which he allows me to install and create a Yeager server and then the other one is OpenShift distributed tracing data collection. So in order to deploy KNM services and we need to actually install KNM's Derby module in Edge Apollo OpenShift serverless operator. The one thing I needed to add configuration here I need to try to use a ZipKin server. In order to do that I'm going to try to tracing ZipKin server which automatically detects the tracing data when new KNM services is created. Okay looks good. Now I'm going to just from my ZipKin server which I'm going to create it again for OpenTelemetry.   One of them is the own KNM or scalar part and the webbook and the HPA and so on. Okay go back to our demo project then I'm going to try to create a Yeager server and I'm going to just sort of with all default configuration which is a really comfortable Java developer and it will get started in a minute. Let's try to add a new OpenTelemetry collector as well. So here is the OpenTelemetry collector a little bit different thing between Kubernetes version versus local one but as I mentioned earlier I'm going to try to use the KNM server for the receiver so that's why I bring the receiver as a zip in and then Yeager which is the same thing but only different thing is here I'm going to set it up the right service name rather than just local host and then in a local host you can actually skip the TLS termination even which is not insecure for the demo environment local which is a good enough for moving forward but in production I strongly recommend you have to set it up. TLS termination will secure your application that's why I put in the TLS option in the previously I set it up insecure but in production environment I'm going to set it up TLS as with the certificate file here and then I go back to OpenShift developer console and create an OpenTelemetry collector and create that bring on create button and then paste all your file in this demo view and everything is good here is once again the right service name with the name space and the receiver zip key and explorer Yeager and then once you create the button and then it will start in a second okay looks good so when you click on the cluster collector or OpenTelemetry you can see the right logs on inside the pod and now I'm going to add a few more operations here to deploy this application into OpenShift cluster as a K-never services so first of all I'm going to specify container image group which is the same name of my project name or its name and I'm going to push this containerized image to integrate OpenShift generation 3 and then the deploy through which means when you package and use the application using focus CLI or main command line it will automatically deploy to a remote Kubernetes cluster and which is Kubernetes is a native so there are three options for developer you can define the target for deployment one is K-native and for just normal application path based on Kubernetes and OpenShift and I'm going to make it available router to access to endpoint by external user and I'm going to use the separate software for X that HTTPS protocol okay so I'm going to start my demo and then I'm going to deploy this application including building using QuarkXa build I'm going to skip the test and it takes a minute to packaging application in the meantime let's try to ask the Jager server from my production environment from OpenShift when you click on the open URL it automatically integrates the single sign on from OpenShift user account so which is really awesome and when you go to Jager UI you can see there's no service at the very beginning just like a user in the environment then when you load this Jager UI it automatically shows the default one service Jager query just like the same thing in the local environment okay let's give you some moment and the back to the terminal window behind the scene in actually packaging fast star like a Java application artifact and after that it will package an application using OpenShift S2 processor you can actually use Docker build storage as well and in the end it will push its containerized image into integrated OpenShift containerized street you can also push it into external container you'll need and then in the end OpenShift Worker node pulls that image into available Worker node in happening well we did just one single QuarkXa data command line which is really good and go back to OpenShift Opology view you can see the new application just started and this is the QuarkXa application let's try to make this hot icon with the QuarkXa but first of all you can change the icon kname services to function which is more explicitly showcased in subless application and then you can add application runtime label to showcase this application is based on QuarkXa or NCB to do that and then when you go to QuarkXa runtime logs you can find the QuarkXa application running on JVM as a job file and here's the QuarkXa race project and then here is the OpenTelemetry extension and REST React or Reactive Program which is really good for support application as well okay just take a look at that this is a pod name so when you go back to Yego UI and just refresh and it automatically detects a new services just like a pod name so the reason why you can see two different pod because when you change the revision name it automatically restart another container so that's why you can see two pod name here which means the OpenTelemetry automatically gather that telemetry data from kname service when kname service is a startle and which is really good so you don't need to set it on some specific code in the application side the trace all telemetry they're already down pod so when you go back to here and then you can see the application pod and subless application automatically scaled down to zero just like a default subless behavior I'm going to copy around url and then go back to terminal and try to access endpoint for invoke the REST API and just make sure the relevant trace detected by OpenTelemetry and send it to the Yego server okay so first of all I'm going to try to access hello and then you can see the QuarkXa application automatically starts just like a QuarkXa strategy under sublation analogy and then try to access another REST API like Ola and then Qubiti and the return result is exactly the same value we can see in my local then go back to Yego UI and reload that thing and open operation and you can see three different operations you can see and then click on hello and you can see one trace because we just called one time and then here's the detailed tracing data and telemetry data and then go to Ola and so one thing so let's try to hold one more time Ola REST API and go back to Yego and find that you got two traces immediately you can aggregate the data from OpenTelemetry and back to the Yego server and one time and then you got three traces here so let's go to Gritty and you can see the code one more time Gritty and then just find trace you have a two trace which is happening simultaneously almost and you click on Gritty RESTful services you can find the more detailed tracing telemetry data which is already came from multiple deployment because GKNM services keep up and down based on demand traffic. Go find the more trace log into OpenTelemetry collector pod when you go back to develop console and then click on cluster collector pod and the view logs you can find a bunch of the logs aggregating from the OpenTelemetry thanks for watching have a good rest of the day