Single-back Camaro with Winston, pressure coming, Michael Walker drops Winston, Anna trying to get there. J-Messal go down from a third time and a half. You shall not pass. Hey, what's up, Falcon Spades? I am DJ Shockley. Thank you for joining me for Falcon's film review powered by Microsoft Surface. And I want to show you just how this Falcon Spades rush has improved from one year to the next. You're talking about last season only having 18 sacks this year. You already have four. I'm going to show you how they got home with just four in this particular play here. Now up front, watch Grady Jerry. Grady Jerry had a great day. Had one and a half sacks. He's going to beat his guy with an over swim move. But the best thing about this particular rush is you're only rushing one, two, three, four guys. You think about trying to get home with more than four guys sometimes, or trying to get to the quarterback. You need pressure with more than four. But this time the Falcon is getting home with more than just four. The pressure from out here by Lorenzo Carter on the outside and Adé on the inside is a huge help to Grady in the middle. Play gets started. Boom. Grady does a nice swim move. He's already beat his guy to the inside. And that's important because the quarterback always wants to climb up in the pocket. When he can't climb up in the pocket, he looks to retreat from one side to another. But one thing the guys do really good job with holding their edge does not lose containment. Now Grady wins up the middle. Now look at Lorenzo Carter. He has his edge secured. You got this edge secured. There's really nowhere to go. James Weston wants to retreat here first. Then he sees, oh, I have contain. Now he wants to retreat to the other side. And now here's Lorenzo Carter sitting for him. So now you got pressure from the middle. You got pressure from the edges. And now you got contain from this side. There is nowhere to go for the quarterback. And that's how you end up with a good sack. Nice job of four guys getting pressure and doing their individual job to get pressure on the quarterback comes up with a big time sack. All right, here's another example of the passers getting home with the second level defender, Michael Walker, right here. Now this particular player here is Michael Walker sitting here. He's looking like he wants to but he's sitting on the edge. But the best thing here is the Saints actually pushed their offensive line to this side. So everybody's going this way because they think more pressure off to the side. You got Grady over here in the one. So he's more favored towards this side. You got linebacker pushed over and it looks like you may have pressure from this side. But the Falcons are going to actually come from the other side as the play gets started. And you see now he steps up and look at the offensive line. Everybody's going this way thinking pressure is coming from this way. And because this guy was standing on the line, they thought he was most dangerous. But he actually dropped in the coverage and this is great disguise here. Now you got pressure off this side. Now you're going to have anybody to block Michael Walker coming straight through here when they thought they had it picked up. This is scheme wise the Falcons out scheme the Saints to get home. And you see Michael Walker get home. There's nowhere for him to go to football because on the back end the Falcons also play zone coverage. We got playing cover three. So everybody's eyes is looking at the quarterback. So now he has nowhere to go to the football and the speed of Michael Walker getting home is huge. And now you come up with another big time sack. Nice job a lot of scheming the Saints. Nice job of using covers on the back end as well to get the sack. This was a total team effort on the defense to get the sack on James West. One hour after jumping to another play when it's extreme pressure they go cover zero and all that blitz. Let's jump right into the play here. As you can see here I want you to see what's happening. Look at all the Falcons communicate. Look at all the conversation going on. Guys talking, guys communicating. This is a big third down here. And if I go cover zero, cover zero means there's no safety in the deep. And you got more guys coming than they can actually block. But I want to focus on back here. Look at AJ Turin. Eyes are inside. He's looking inside at the quarterback. So now he's going inside leverage. He will not give up anything to the inside because that's the worst place to get beat versus cover zero. You got eyes on him. You got all these guys with eyes on that particular defender and they do a good job of coming home. Now you got cover zero. You're bringing all guys one, two, three, four, five, six, seven guys.  And because the pressure gets home, you use guys that are really fast. Now you get home. Boom. Look at the speed of these guys coming through the middle. Coming off the edge. Now you got contain on the outside as well. Now you're looking on the outside. Where's he going to go with this ball? Nobody is open. Everybody's got eyes on their particular receiver. Now he wants to throw the football. But because of the tight coverage, nowhere to go. Now you got speed coming through. And I love what you call this third down here by D&P. Getting home, forcing the ball out of the hands of the quarterback really quickly. And that's an ideal pressure. Balls out of the hands. Now you have a fourth down. So the nice job of bringing pressure for cover zero and third down. Pushing the offense back. All right. This next big time pressure is the one that I love the most by D&P's and his defense. Let's jump right into what I'm talking about here. This looks similar to a play we already talked about. Look at all these guys at the line of scrimmage. Who's coming? Who's not coming? No idea if you're the same or you're the quarterback. What ends up happening, the Falcons only end up blitzing four guys and are able to get home and get pressure on the quarterback. So here, I'm going to show you who's actually going to bail. You got both linebackers going to bail. Now you have Michael Walker. He's actually going to bail as well. But here's the pressure. The pressure's coming off this side. And you would never think he's coming because he has three guys over here that you're worried about. But he ends up coming right off this edge here and getting the pressure on the snap of the ball. Now watch all these guys bail. Out, out, he's gone.  Now what's so unique about this, the Falcons started this particular play where it looks like cover two. And now you rotate down and it looks like it's going to be single high. But he's going to go to cover two. This corner's going to go all the way back to cover two. This outside linebacker is going to play a cover two here. So they changed our whole coverage and now because the Saints thought that more guys are coming, they blocked it wrong. So up front as you start to play here, boom. This guard should have been here, but he saw him late just because he thought one of these linebackers were coming. So a great disguise on the back end. Now you get the pressure and he has to climb up in the pocket. Nowhere for him to go. Third down, you get off the field and he has to put the football in the hands of the Falcons. This is a great job with disguise. Great job with guys being in the place where they're supposed to and disguising what it looks like on the back end. All 11 guys on one accord getting pressure on the quarterback. And this is what a Falcons were able to get four sacks against the New Orleans Saints.


So first of all, as opposed to what Dick Winchelbaum said, I actually did follow the academic model and got everything done much, much too late, and did not have an opportunity to have a prolonged discussion with the panel at all. I did respond or react to Huckum directly about some of his material. There was primarily specific questions I wanted to ask. And now that I've been listening to the session here and participating today, as well as listening to Huckum's suspicions, I'd like to just go through a couple particular points and then have the John and Jeff and Cynthia respond. So I wasn't quite sure what was meant by different paths for phenotyping, but I think what I understand about that now is that just that there's adult phenotypes to pediatric phenotypes and that these are being done in different places by people of expertise, either in pediatric or adult phenotypes. I really don't see this as an obstacle at all. I think it's a strength in general. Pediatric research has been underrepresented in genomics, and so I think it's perfectly fine for there to be particular phenotypes that are enriched in pediatric patients and are not seen in adults, and that special attention will be put to studying them. I mean, this project, as I understand it from listening to Huckum, is that it is primarily a discovery project. This is not an implementation project yet. And so discovering genotype and phenotype correlations in pediatric patients makes a lot of sense to me. I had a question about the informed consents, and I know we discussed that a little earlier today. I had a particular question which has to do with what happens to pediatric data once the subjects become adults. I know there are some recommendations and some thoughts that keeping this data without re-consenting these individuals as adults is a problem. I see that re-consent process of tens of thousands of people as being a huge problem, and I'd like the panel and the people participating in this meeting to perhaps comment on that issue. I have some thoughts about it. I'd like to hear what people have to say. The next issue is, okay, is the TPMT genotype imputation. Has any of this been linked down to patient response? What fraction of these tens of thousands of patients have been genotype have actually been exposed to this drug, and do we have follow-up data and do we know what happened to them? I mean, you know, I think the impact from Mary Relling's work and other people's work on TPMT genotyping to guide a six-fire guanine or is a fire-cream therapy is well published and understood. I was wondering whether in this particular project we have any such information. I thought that the emphasis on the CNVs was also very welcome. I think they have been, they're obviously very important in clinical medicine and it's great to have them more involved in this discovery project. I just wanted to mention, it's something I mentioned to Hakon, that the DGV is a valuable resource, but there is ESCA, which has now become what we call the ICCG International Consortium for Clinical Genomics, and there's a major attempt made through a grant that's been funded by the NHGRI to try to get much more structural variant and copy number variant information linked to phenotype into ClinVar, into the public domain. And I think that having the eMERGE pediatric component work together with the ESCA or ICCG and to try to get as much information into the public domain as possible would be a great benefit for everybody. This is a small point about the concordance frame between sequencing and imputed haplotypes and it's not a big deal. It has to do with whether they're common variants or not. It's a rare variant that people are having more problems with imputing and of course that makes perfect sense, but for the more common SNFs, common variants, the imputation is far higher concordance than this and so I think that this is not really a major issue at this point. So I think to summarize my reaction is that I think many of the things that Hakon described as being obstacles I actually think are strengths and that's a good thing. I would like some comment made about the removal of pediatric data once these individuals have reached adulthood. I'd like to hear more discussion about going beyond just simply how well you are doing at imputing genotypes for TPMT, but instead is it linked to the EMR? What do you know about outcomes, phenotypes? How many people have been exposed to the drug? And I'll stop right there and ask the other members of the panel to comment. So why don't we'll pass the baton to Jeff Bodkin, who can perhaps as well as his job as moderator, moderate the answers to the questions that Bob has posed.


So now it's time to speak about C-Pass again with my colleague Mathieu Dupré and Florian Califromerti, and it will be a bit more technical how to use C-Pass to deploy low latency virtual machines with redundancy. So let's go guys. Thank you. Hi everyone. I hope you were there this morning for the C-Pass presentation. So this won't be a presentation or explanation about what C-Pass is. We are going to show you how to install it, configure it and use it hopefully in 40 minutes. So I'm Florent. I'm with RTE. Of course the slide doesn't work. Okay. Let's try this way. So RTE is the TSO, the electricity transmission system operator in France. And Mathieu here is from Salafa Inux. We're both working on the C-Pass project with the LF Energy. So C-Pass is the configuration project to build the virtualization platform for a latency sensitive application, I should say. Brief reminder of the architecture. This will be useful to have that in mind for the demonstrations. Usually we think that C-Pass has at least three nodes since it's a cluster and that number of machines. At least two hyperverses which you can have three, you can have five. And inside each hypervita we have loads of technologies that are used to deploy the VM and manage the VMs. So basically KVM, everything running on real-time Linux. And for networking purposes open the switch. That would be interesting to understand that we use open the switch to extend the network from one host to another so that VMs can talk to each other even if they are on different hosts. And also that if they have to move, if they have to migrate from one host to another, for them it's the same. Their network doesn't change. Yeah, C-Pass uses both Debian and Yachto. It can be run on top of Debian or on a custom Yachto distribution. So C-Pass uses Defer in the image creation. For Debian we use a full automatic installation to create the Debian iso machine which will be flash on the disk later. And on Yachto you use of course Yachto to create the image. The setup is done with on C-Ball or on C-Pass. And the on C-Ball setup is similar for both versions. But actually there is two branches, Debian main branch and main branch which is actually the Yachto branch. We plan to merge the both branches in the future. So the setup will be the same for on C-Ball and Debian. But for the moment it differs. For the users it's similar and uses both C-Pass version use the same tools and can use the same on C-Ball playbook. So yeah, so of course all of the following demo will be except for the image creation part will be can be used for both Debian and Yachto version. So to use C-Pass you need of course machine. It is important to have a node number of machine because in C-Pass you have a cluster and we need to have a quorum to vote and a quorum occur on a node number. You also need of course a flash image. You can use a flash image in the USB drive. And all the setup is done by on C-Ball. We made the on C-Ball playbook to do the setup but the inventory need to be customized by each installation. I will present you the media creation. I will begin with the Yachto version. For the Yachto version it is a regular Yachto installation. You have to fetch the source. We use repo to do that so you have to use the repo manifest and use repo sync to fetch the source. In C-Pass you have to put your SSH key during the image question step. So the SSH key will be used for the setup in the setup step by on C-Ball. So you have to put it in the keys directory. You can also if you want customize some little aspect using the C-Pass.com file. You can base on the C-Pass.com sample to do that. You can change for instance the key map and little small thing. After you do that you can build your Yachto distribution using big bake or we made a custom tool named CQD. We can help to do that. It's just two commands. CQD it will initiate and create the curve. All you need to build a Yachto distribution and C-Pass and after you are just to one CQD run to build the image. We do not have a demo for the C-Pass version because it's a very long building. But I can show you what we do for the devian base image. I'll try to do the demonstration live as much as possible. Of course many times it takes some time so we have also videos that will help to show things that take more time. But for building a devian version it's as simple as going to any Linux machine with basically docker on it. You find a repo. You clone the repository. Before I should say that everything is documented here. So if you start your C-Pass journey at the C-Pass architecture repo it will explain that there is a Yachto way of doing things or a devian way of doing things you have to choose. If you choose a Yachto then go to this repo and that will help you use big bake to build the image. If you choose devian you go to this repo and that will help you also create the installation media. It's as simple as running one command. Just before you need to do some mandatory customization that I will show you. So you clone the repo. You go to the folder. I hope it's big enough. Then it will tell you that you need to copy the default variable to the actual variable file. So you copy the default variable here to the actual variable file and you edit it with your own settings. You can change the key map, the time zone. You can change the ashes of the default password and the default username. You can set and it's better to set the SSH key that you will use to connect to the machine. That way you won't even have to put a keyboard or a console to the server. You can directly access the deployed server with networking. For that you have to set up some networking with a default interface, some IP address. And everything will be built in the image. Once you deploy the server it will already have networking. It will already have credentials and this is what I will show you right now. You go to the folder you want to build the image in. You run the command that's written in the read me. And basically it will create the Docker container for the environment. It will download all the package. It will build the image. And it will provide you with an ISO file. It takes about five to ten minutes. So I won't show you this. And like I said I have a video for the next step that we'll show you. Let's go at the end of the video. So at the end of the video you have your ISO file. To test it we're going to create a VM. Of course in the real world you will use real hardware. You will get the ISO file. You will set it as a virtual disk. And then you will run the machine. And what will happen is FAI is going to boot. You don't have to touch anything. It will deploy Debian version with the CPAS specs which is basically a list of packages. There is no configuration done beside having a list of the needed software. And like I said some networking and some default credentials so that you can access the machine easily after deployment. So if I go to the end of the event once again it will take like five to six minutes to deploy everything automatically. Download the package. And in the end it will reboot the machine. I'm going to show you. If you go to the end of the process it will reboot the machine. Boot the new Debian system which is once again completely empty. It only has a list of installed packages. And once the machine has been booted since you have configured your network and your credentials it will be as easy as you can check in the console that the IP has been set as you wanted. And then you will go to your favorite SSH client and connect directly to the server. Of course you will have to do that three times for each of your server. Like it's written in the documentation there. Once you have done that you connect, you interconnect your nodes. There is a documentation on how you should do that with some STP configuration and so on. We won't go into details here. Once your cluster network is created then you can go to the appropriate Ansible branch. So if you're using Debian it's this branch. And this documentation will help you configure the servers. That's what Mathieu is going to show you right now. Yes, now we have all the machine is fresh. You can set up Ansible. You can set up the closer with Ansible. So you can use a playbook in the Ansible repository. Playbook will perform action but you have to tell on which machine you have to apply the action, the task and you have to set parameter. With Ansible it is in the inventory. You can do that. There is several ways to write inventory in Ansible. We choose to do it in a YAML and to bring together machine and associated variable. So all the machine needs to be placed in the cluster machine group and hypervisor in the hypervisor group. Here it is an example which is kind of having the Ansible repository. We have three machines on it. Yes, node one, node two, node three. On each node we define the variable. For instance Ansible OS is a default Ansible variable in which Ansible will use to connect in SSH and connect to the machine and run the task. The other is C path custom variable for instance to define network IP or other parameter. Variable can be placed just in the machine, near the machine if it is machine specific. But it can also be a global if it applies for all machine to the cluster path you also need to place the machine and just scroll up a little. You have to set the Oath in the main group for all the machine will be in the cluster. And in the client group and the machine will store data need to be placed in the ODS group. OZ group. Can now show you a demo of the setup. It is an excellent Android time demo. To call Ansible you need to call Ansible playbook command. You have to pass the playbook you want. Here is a cluster path. And the inventory in which the action will be set up. It is your inventory you have just awaited. Yes. It is how Ansible works. You can see all the tasks which is performed and on which machine it's running. You have a state to if the task is keeping or if it is just run and strange looking. It is display in green and okay. If it changes something it is in yellow and it is white and change. And if there is an error it is in a way fail. At the end you have a little recap of all the steps. Yes. It is not very important. Yeah. And the tab. Good. Next step is yes network. Yes. So like I said before we are using OpenVswitch to extend the bridges from one host to another and to make the hosts communicate with each other in the most simple way. Of course once again we are using Ansible for that. So we are using Ansible to configure. Ansible to manage guest networking. We are also using Ansible for security and for basically everything. So I'm going to try to do this demo live. It's going to be interesting. So the idea is to go to an EPAVSR. So I have three EPAVSR. They are called Elabo 1, 2 and 3. I'm going to call them Node 1, Node 2 and Node 3. And they are not running VMs. Right now the VM are shut down. I'm going to show you how we're going to try to make a DBM, this VM, which is actually not running, communicate with a DBM 2. So we are with DBM, DBM 2. We're going to have DBM running on Node 1, DBM 2 running on Node 2. We can also have DBM 3 running on Node 3. And we're going to do what we need to make them communicate. So the first thing we need to show you is that if we run OBS control show, we can see all the configuration. And we can see that there is no DBM interface. There is no DEB interface. So we're going to create them, use Ansible to push them on the server and then configure the VMs so that they can connect to those interfaces and then run the VMs and see if they can communicate to each other. Let's do that. So for that we're going to go to our Ansible server because everything will be done not online, not on the server, but with infra as code. Let's go to Ansible. And there is an inventory for the cluster, which I showed you. There is also an inventory for the networking, for the guest networking. Let me find one, sorry. So we're going to use this one. And this inventory, this OBS inventory will be the topology, the open-v-switch topology that the cluster will emulate. So you can create bridges, you can extend bridges from one host to another and you can connect your bridge, you can create interfaces for your VM to connect to the bridge and you can connect your bridge to physical interfaces if you want to extend your bridge to the outside of the cluster. So what I'm going to show you is that I've prepared this. So we're going to create three interfaces in the bridge, which is called the IPO. And we are going to just add three interfaces on the topology. And once you've done that, we're going to run the playbook, which is called network. So once again, there is no, right now, no interface to that name. We're going to run the playbook and we'll take the configuration in the inventory and create this configuration on the clusters. So that's this step, basically. So once this step is done, okay, we should, yeah, go and see that the interfaces have been created and they have been created on all servers. So wherever the VM will be, it will have an interface that it can connect to. So the next step is to go to the VM configuration and tell it to connect to the interface. So we have custom tools to do that. For instance, this is our tool where we can edit all the metadata of the VM. And if we go to Debian 1, we can go to the XML of the VM. So this is everything, this is the VM configuration, basically. And if I, if I go there, I see I need to add this kind of XML that say basically create an interface on bus 0, PCI bus 0, slot 2, that will be important for later. It's not managed because it has been created by OpenVswitch and use Debian 1.1. And on the second machine, Debian 2, we do the same thing. Here, yeah, same thing. So use the PSI slot 2, Debian 2.1, not managed. Once we have done that, we just have to run the VM, basically. So let me show you that the VM is not running. I'm going to use a tool that Matthew will show you a little more in detail later. And we're going to run Debian. This is warning. Don't worry. It works. And we're going to run Debian 2. And we're also going to run Debian 3 because why not? We'll do live migration test later. That will be useful. And since we are in Elabor 1, we can check that the VM Debian is started in Elabor 1. So if we run libvirt, we can see that the VM is actually running. And if we are curious of what's happening on the VM, we can also connect to the console and check that it's properly booting. So we can, this is the VM booting, basically. And once it's been booted, we can connect to it. And we can check that there is an ESP0 S2. That was the bus and slot, the PCI information. And we can, we have an IP. This has been configured before. And we can connect to the machine. And since we have the same thing with, let's take a few minutes, a few seconds. And if we go to Elabor 2, the second node, and we can check once again that the Debian 2 VM is running on Elabor 2, we can verify it's the case. So basically we have two nodes, one VM on each node. And the idea is to see whether we can ping one from the other. And it works. So basically we have two VMs and they are discussing together through an OVS bridge that I extended using the XLAN. Let's talk about introspection. So now for this one, I probably use the video because it's a bit more complicated to show live. The idea is that you understand that we use basically Ansible for everything. And we also, since it's very important, we all very often get the question of how we manage cyber security. Since on this version we're laying on Debian for the patches, this is an opener. We're dedicating to Debian. But for the configuration, we might have to change things because the national agency says that we have to apply specific settings. That is a vulnerability. So let me go to the video. We're going to go a bit quick on the video. But the example I chose is to have, for instance, a secret file. Let's say we call it secret. And this file is readable by everyone on the server. And let's say we need, because we heard from somewhere that this is a vulnerability and that we need to change the permission on this file. Well, since we're using Anthras code, it's pretty easy to go to the Ansible playbook to add a task that will say, okay, this file, it needs to belong to root and to have 6.0.0 mode. So basically only readable and readable by root. And once we've done that, we apply the cluster. We apply the paybook. We reload and we see that everything has been done and we won't even have to connect to the machine. So if we have many clusters to manage, it's pretty easy. And the second thing we want to show is that it can also help with compliance. Because the idea is that Ansible is pretty good at showing you that it had to change something. Which is not the case if the settings were correct in the first place. So this is normal. This is a new setting. So it changed. If I reapply the playbook, it shouldn't have to change. But if there is some malicious threat actor that, for instance, changed the permission, I'm going to show you first that if you reapply the playbook, Ansible will show you that nothing has changed. Everything, sorry. Sorry. I say again. If the malicious actor changed, for instance, the mode of the file and put it to 6.0, or on another server changed the owner of the file, then I will say that this mode is not correct. Sorry. This mode is not correct. This owner is not correct. And if I reapply the playbook, first I can check that the playbook will correct once again. It's the job. But it can also show me that on virtue level 2, everything was fine. But on virtue level 1 and 3, they had to be changed. So we can use Ansible to detect any cybersecurity issue or problem. So it would be a recommendation, even if there is no change, to reapply the playbook regularly just to monitor that the number of changed items doesn't increase, which would mean that someone is playing with the servers. And yeah, we put this to the next level by implementing a playbook that does a lot more than that. So we use this to implement the French agency, cybersecurity agency rules, BP28, on the version. On the 2 versions, otherwise during the build time, I will detail that in the latest call with Gaon. So we made a playbook with Arden, Debian, following this guide. You can run the demo. So the playbook is running the same as the setup playbook. You can see it changes. The interesting part of this playbook, it is you can check the rules applied by Ansible, but you can also revert all the Arden in the playbook.  So you can run the Arden in the playbook. It is useful if you, to place all your clusters in the debug mode, because the Arden rules, with Arden you can't use regular debug tools. It's not a mode you use for development. It's really a production mode. Since you have to work, we still have to work on the server. We like to be able to un-harden the system if needed. This is how we un-harden. A simple asset. So one playbook to un-harden. So once again, playbooks for everything, playbook for cybersecurity, playbook for compliance. And now, so we thought about installing CPAS, we thought about configuring CPAS, hardening CPAS, managing security. Now we're going to talk about using CPAS. And Mathieu will tell you about the VMs. Yes, because the VM part is complex in CPAS. We use multiple tools, because there is the virtualization, but also the cluster part. It is done with different tools. The virtualization is done by KVM, KVM and QMU together. It can be managed with LibVert. We use LibVert to control KVM and QMU. But KVM and QMU rely on SAFE to use it. Because SAFE stores the VM disk inside it and distributes it in all the clusters. The distribution storage is done by SAFE. So QMU and KVM have to read and dialogue with SAFE. But you cannot directly manage SAFE with LibVert and QMU. And there is also the failover part, the cluster part, which is done by KVM and PESMaker. And KVM and PESMaker have limited control on LibVert, but have no control on SAFE. So we made tools above all that. We made three tools, a pattern module, which is called VMManager, which is a command line interface, VMMGR, and a non-several plugin, which we rely on VMManager. And of course, the associative playbook, which we use on this program. So the idea is you can use VMManager to manually deal with the VMs. But since we are in love with Uncivil, we expose all these services to Uncivil module that means you can use Uncivil to manage the VM as well. Yeah, definitely. Yeah, we have a demo of the command line. Just a bit of the command line interface, VMMGR. Here, this is the helper. You can do all the basic things, create VM, remove VM, list of VM. Some advancing options are also here. You can create a snapshot, a reverse snapshot. And you can also define some cluster rules. For instance, you can add collocation rules to tell the VM to always be one of the same machine on the server. This can be useful if you do not have an extended network on all the cluster, but you have just a private network to communicate between the two VMs for performance present, for instance. Yes, I can... This is an example of how to create a VM. You just need to have a name. Yes, okay. You just... Is it running? Okay. You have to set the name, the disk you want to use, and the libvert XML configuration. And the command will deploy the disk image on-safe and create the VM and configure the pacemaker to do all the cluster configuration. Yes, now the VM is created. You can see with the VM set, this is the VM is started. You can do all the... Yeah, this is just another example with another tools, which is pacemaker control tools, CRM. You can see how demo VM is created. You can also see where the machine is running the VM. On the machine, the VM is running. Here is the AMD machine. Now, for the demo, I will remove the machine, stop it before and remove after. So you can stop the VM, which it will be still on the cluster, but it will be in the stop state. You can disable the VM, the VM is still in the safe cluster, ready to be enabled, but it's not on the pacemaker cluster, or you can remove the VM and it will remove it completely from Cpass. And the disk is here. So, this is the demo. Okay, that's fine. In Cpass, we have to deal with the real-time constraint in order to have low latency network, for instance. This is a particular VM. They have to be pinned on a specific CPU because we have to use CPU isolation if you want to have a good real-time. We also need to use the real-time scheduler. Okay. And some Libertwick. All this configuration is already done by Cpass. I have just a quick demo. We showed the secret test tools. And how it differs between real-time VM and a regular VM. On the top, this is the max latency on the regular VM. And on the bottom, the latency on the real-time VM. This is the kind of latency we were expecting from the guests. And, yeah, let's talk about live migration and also fail-off immigration. Once again, I'm going to try to do it live because it's fine that way. So let's go back to our cluster. Same as before, same VMs. No cheating. So we have three VMs running on different interfaces. Let's connect to the first one, for instance. So the first one is Debian. And so we have booted it 13 minutes ago. That was for the last demonstration. And we're going to have a ping it Debian 2. So both VMs are there and they're pinging each other. Let's say you want to move a VM because you want to better balance your cluster. You want to do a test or whatever. So it's very easy to manually move a VM using a pacemaker. You use the CRM tool, and you say you want to move Debian 2, for instance, E03. And if you check what's happening in CRM at that time, you're going to see that the VM Debian is migrating. So the memory of the VM is going to be transferred from one host to another. And then once the memory is synced on both hosts, then pacemaker will switch. And so we can see that now the VM is running on the third node, and the VM hasn't stopped pinging. It doesn't even notice that it has been live-migrated. So that's pretty nice. And of course you can move it back. But before we move back, we're going to try something. We're going to say, okay, we have now two VMs running on E03. Let's say we want to do an operation on E03. You want to reboot it. You want to apply security updates. You want to do some serious maintenance, and you don't want any VM running on it. Once again, with CRM it's pretty easy to ask it to put the node in standby. So what I'm going to do is I'm going to put it in standby. And at the meantime, I'm going to show you what's happening at the cluster level. So I'm going to say, okay, put node three in standby. So node three is going to be standby. And all the resources running on node three are going to be migrated. That's it. So we're going to wait a few seconds for the VMs to be migrated. And we're migrating dbn and dbn3. And once again, we can see that dbn1 and 2 will be pinging each other with no impact. And then, okay, that's it. It's done. So now, no VMs are running on E03. We can check. We can go to E03. And we can say, okay, do you have any VM running? Or say no. And then we're going to say, okay, I'm done with my operation. I want to node back online. So if I put the node back online, then the cluster will automatically say, okay, I'm going to migrate the VMs back there because you asked for the VM to be running on E03. So E03 is up and running. So basically, it's going to migrate there. And once it's done, the final test I want to show you is what happens if we crash, basically, E03. So let's have the VM still running on E03. And we're going to crash the node. So what will happen? Of course, you won't be able to migrate the VM because the memory will be lost. But at least it can restart the VM somewhere else. So to crash E03, I'm going to go to E03. And I'm going to do something that you should never do. I'm going to sync the disk so that... This is the only real way to reboot the server. So I'm going to reboot the server very, very badly. So at that moment, everything is crashed. The server is down. The VM is down. The cluster will very promptly notice that the host is down. And it will, as soon as possible, start the VM somewhere else. And it had decided to run the VM on E03 and E03 on E03 once again. Of course, the VM has crashed. That's normal. But they will be restarted in a timely manner. At least in 30 seconds. And so if we wait 30 seconds, then we will be able to connect back to the machines. We can even monitor the boot process if we like and connect to it as soon as we boot it. And during that time, I'm going to end the presentation or the failover that's what we've just done. So this was our last demonstration. The VM is still booting. In the meantime, thank you for your invitation. With the C++ project, we have many resources. We have a Wiki, a website, a github. We have a Slack. And very recently, we have a YouTube channel where we'll be uploading all the videos we've seen today. And more. And hopefully the community will continue to upload videos and tutorials and explanation of how C++ works to this channel. So don't hesitate to go there. And we are available for some questions if you have any. Thank you. Thank you. So it was quite technical. I guess you've got a lot of questions. Again, most of the videos are available on YouTube and we make sure to upload as much content as we could do. So if you have any questions? Yes, sure. Thank you for the presentation. Exist also a life cycle for a disk upgrade? Disk upgrade? Yes. You use Debian. And the Debian life cycle is around three, four years. So you have to make a disk upgrade for security reasons. Exist also some disk upgrade mechanism? Well, I'm not sure I understand the question. But the upgrade mechanism we are relying on. The host itself. The host itself is running Debian. So basically we are relying on Debian. So if we want to upgrade, then we're going to use Debian to secure the upgrade. We are using LVM snapshots. So our recommendation. And there is a documentation on the GitHub also. It's if you want to do some... Well, I'm going to show you the best way. Well, well, yeah. So if you want to upgrade, what we recommend is to do a snapshot of the root file system. Then to check the snapshot is working, to do the upgrade you want to do or any change you want to do. And if it goes wrong for any reason, then you can also always roll back. If it does not, you can discard the snapshot. I'm not sure that I answered your question, but... Okay, thanks. And just in the meantime, just to show you that the host has been rebooted and so that the cluster has decided to move the VM back on the node since it's back online and clean. So thank you very much for the presentation. So it's great to see the guidelines for Debian. Do you have also the same recommendation for Yocto using a packet server or elsewhere? In Yocto, all the other languages are done during the image creation. We also apply a VP28 rules. I will present in more detail with Anguillon in the last session. But in Yocto, you can also have the package management. That's why I was asking what... No, no, no. In Yocto, you use software update and PC active slot. System AP. Thank you. Last quick question, maybe? Thank you. Thank you.


So we all are in for a treat today.  And I'll let you take it away. Thank you. Thank you, thank you very much. So I just make sure everything works. Looks like it. OK, we're ready to go. So hi, everyone. Today I'm going to talk about our cool research, evil PLC attack. And I'm going to show you how we were able to weaponize PLC's, programmable logic controllers. So let's start. I'm Shavant Brzezinov. I work as a vulnerability researcher at Clarity. We're doing a lot of ICS and SCADA security research. I'm a CTF player. I've played in a couple of CTFs, including Pound2Own, Miami 22 and 20. I also own a DEF CON Black Badge for winning the ICS CTF. And I'm mostly enjoying breaking PLC's. And I'm not sure if you can see in the small picture. This is our cool lab in Tel Aviv. Special thanks to my colleagues who worked on this research with me, Mashav, Uri, Nam, and Amir. Thank you very much. OK, so the agenda for today is to give you a little bit background and motivation for this research, why we started it. We're going to talk about PLC upload and download procedures. This is how we're transferring logic to the PLC so it could execute it and control the physical process. And finally, I'm going to introduce the evil PLC attack. This is a concept we have developed to weaponize PLC's so we could propagate and attack engineers and technicians. So let's start with the story time. So as I told you already, we have a super cool lab with a lot of ICS equipment in Tel Aviv. And around two years ago, we were like, yeah, we have a cool lab. Why won't we expose a couple of PLC's online so people could scan us and we will monitor the interaction because why not? So indeed, we chose a couple of PLC's from our lab. For example, what used to be GE now, it's Emerson RX3i. These are all controllers, industrial controllers. And we just plugged them to our external router and we exposed them online. We obviously monitored this entire experiment. We had sniffers and we had auditing and logs. And we waited a couple of weeks to see what happens. I bet you can guess that we've been hacked. But our honeypots, the real honeypots, were hacked OT style. Now what does it mean to hack OT style?  This is the software that the vendors are distributing along size with the hardware to program and monitor the PLC's. So basically it's kind of a diagnostic tool and IDE to program the PLC. So all these sophisticated attackers did was to use this software plugged in the IP they got probably on showdown or census. And what they did is they changed our logic using the engineering work station. Now how did they do that? They used two main functionalities. So the first functionality they used is called upload procedure. In upload procedure, you're asking the PLC, the engineering work station software asks the PLC, hey, what's currently running on the PLC? So it asked to transfer the logic from the PLC to the application, which is usually Windows-based. So we could see in our engineering work station what is currently running on the PLC. It's kind of a diagnostic tool. Now they did upload, they got the logic we plugged in into the PLC, and they modified a couple of parameters so they kind of messed up with our program and then they did upload a download. Download procedure is the opposite procedure of upload. It transfers the logic, the program to the PLC. So now the PLC basically has new software it needs to run. And obviously, boom. Now, at least this is what the attacker thought was happening because in reality, our PLC just turned the red system fault led and that's it because it did nothing. Now, we all know that the best motivation to do some kind of a research is revenge. So we were really furious on these script-kitted attackers that modified our logic and probably thought they're exploiting some kind of a nuclear plant or something, but we really wanted to get our revenge. So we came up with a plan. And the plan was to, again, use the PLC as a honeypot, weaponize it in some kind of ways that at the time we did not know how exactly, and use it as a bait. Now, we wanted to wait for an attacker to use showdown again because they're so sophisticated to scan our external IP address. And then, as we knew, their methodology is to do upload. So when they're doing upload procedure, we would somehow use our weaponized PLC to attack them and execute code on the attackers. So this was our plan. Now, in order for me to explain the concept, we need to go through some terminologies. So let's start with what is a PLC and what is a download and upload procedures. Now, a PLC is a programmable logic controllers. It's a small device computer used to control and automate the physical process. Basically, it means it controls field devices. It can be sensors, temperature sensors, for example, or actuators, such as lift or motors. Now, in order to the PLC to do something, we need to program it. And the way to program it is to use a software called Engineering Workstation. So we're developing our software in the Engineering Workstation software, and then we're transferring this to the PLC, and now the PLC has the program to control the field devices. Now, let's see a cool example. This is, for example, the GE Mark 6 platform. What you can see here is the IDE, the Engineering Workstation, and you can see here some kind of a specific PLC programming language. This is a graphical one with boxes. And we're using this to transfer the logic to the controller. In this case, this is the Mark 6 platform. And once the PLC or the controller has the software, the logic that we transferred to it, it can control a physical process. In this case, this platform is mainly used to control physical processes in the green energy fields, for example, to control wind turbines farms. Now, from a high level perspective, we first need to develop or to program the logic. And we usually use PLC programming languages, for example, ladder diagram or block diagram logic. These are usually graphical programming languages, so it's kind of a boxes or functions that we're writing our logic with. And basically what we're doing is kind of a flow to do some work. So we're asking the PLC or the controller to check value from one of the sensors and if the value is greater than x, then maybe start to defend to cool the room. Now, this graphical representation of our logic is in GE, is also represented as x and l behind the scenes. So you can see here the different pins for the functions and the function itself and some other parameters. Now, this entire representation of our logic needs to go through some kind of a compilation process. So basically we need to transfer what we want, what we have in our mind through the graphical representation to some kind of a bytecode representation so we could transfer it to the PLC. Now, in the GE example, and again, this is one example of many ICS vendors, in the GE example, they're using something they called P code. This is an intermediate bytecode that eventually will be transferred to the controller. And you can see here, for example, this specific byte represents the add function and obviously we can break it down and parse it, but this is just a general example. The next phase would be to transfer this logic to the PLC or the controller. For example, in the GE, they're using a proprietary protocol to transfer this logic. So they have SDI protocol that is being used to transfer the logic that we're writing, programming in the engineering workstation, compiling it to the P code intermediate bytecode and then using this protocol to transfer everything to the controller. And lastly, the PLC needs to execute it and what it does, it receives the bytecode, decodes each byte and then executes using native machine code. And this is how we control, this is how the controller is controlling the physical process. Now, the two most important aspects of it is upload and download, as I've already mentioned. The download is used to transfer logic to the controller. This is from the engineering workstation to the controller. While upload is usually used to do the opposite, to get the logic from the controller to the engineering workstation, so we could debug it and see in real time what's going on. So for example, we can do upload, get all the logic currently running on the controller, modify it and then use download to set back the new logic. So as I said, most of the time we're using bytecode that being transferred from the engineering workstation to the PLC. However, we discovered that in download and upload procedures, not only bytecode is being transferred to the controller, we have the bytecode itself, which is the actual logic, but we also have the source code that transferred to the PLC. We have symbol table with all the information about the variables. We have input and output configurations, device configurations, network configurations, variable definitions and more. And this is all being done so we could transfer the program to the PLC, but also so we could debug it. So we're kind of transferring a lot of data objects and metadata that is related to debug information and configuration and metadata. Now, we also have upload, which retrieves back all these data objects from the PLC to the engineering workstation. And we discovered that the PLC only cares about the bytecode and the configuration. Even though it receives the source code and the metadata, it does not parse it and it just sits being stored on the PLC. While in upload procedure, the engineering workstation has nothing to do with the bytecode because it does not have a decompiler in it, but it only cares about the source code and the debug data. So we were very curious, how can we attack advantage of this? How can we store some information or data objects on the PLC that will not affect the PLC, but when retrieved by the engineering workstation software, something will happen. So I guess you're all in the same mind with me, so we're going to the evil PLC attack. Now, in the evil PLC attack, our goal was to weaponize the PLC so we could attack the engineering workstation. And to do that, we decided to store malicious data objects on the controller and to weaponize it. So the PLC will be weaponized because it has some malicious data objects that does not affect the PLC because it doesn't parse the source code, for example, or other debug information. While the engineering workstation, when someone will do upload, for example, our skip kid is from earlier, will do upload, they will retrieve all the information stored on the PLC and hopefully we will be able to find a parsing vulnerability or some kind of a flow in the parsing of these data objects that will get us code execution on the engineering workstation. Now, we decided in our research to research seven industry-leading ICS vendors and to see if we can implement our concept on each platform. Now, when we're researching ICS vendors, the first thing we need to do is to obtain the hardware. And it's very difficult to obtain hardware on ICS. All you need to do is to go on eBay. So basically you can buy almost any PLC, turns out, yeah. We're not aware of it, but it turns out like you can buy any hardware you want on eBay. You just need money for this. And this is how we got our hands on all the equipment we needed. So we first had setups of seven different product lines, seven platforms connected in our lab. And we also configured the engineering workstation to communicate with these devices. The next step in our research was to understand the upload and download mechanisms. So what we did is we reversed engineered the engineering workstation, but also the PLC firmware. We also analyzed the communication between the software and the PLC. And we did all of this to understand what is stored on the PLC and also retrieved by engineering workstation and how can we take advantage of it and how can we know what is being parsed on the engineering workstation and eventually we wanted to find vulnerabilities in the parsing flow that will lead us to an exploit. So what we wanted to do is to store malicious data objects on the PLC, find a vulnerability in the parsing flow of the engineering workstation and then hook everything together. Now, this is exactly what we did and we were able to find seven zero days, a bit more than seven zero days, but seven chains of vulnerabilities in all the affected platforms. And to do that, we had to research all the different platforms including the engineering workstation. We had to reverse all the different parsing flows. We had to understand how they implemented the upload and download mechanisms in each platform and we also needed to implement our own client of these proprietary protocols. So we had to look at a lot of assembly code but eventually it worked out well for us because we were able to find all of these bugs and hook it up into chains. So I want to give you an example of one of the platforms that we researched to do some magic. So the example I'm gonna give today is about GE Mark 6. So this is the platform that is mainly used in the green energy fields, for example, to control wind turbines. And this platform runs the architecture of the controller is PowerPC Big Indian. It runs real time operating system RTOS of QNX which was very helpful for us because it has a micro kernel of UNIX. And the engineer workstation is called Toolbox SD. It's a dotnet application written in C-sharp. So again, it was quite easy to reverse a journey using reflection. And the protocol between the PLC and the engineer workstation is called SDI runs over TCP port 5311. I'm not sure if you can see but you have all the components on the screen. You have the engineer workstation, the Toolbox. Next to it, you have the controller and down below you have an example of one of the packets from Warshark. Now, the first thing that we wanted to do is we wanted to understand what is being stored on the controller. Again, we were looking for a data object that the controller stores but does not use or parse because we wanted the PLC to be left unaffected. We found that the binary code or the P code is stored as dot P code which was very convenient for us. And the source code is stored inside an archive file called device.zip and inside we had all the XMLs we've seen previously. Now, the PLC or the controller does not care at all about the source code. It just sits there so whenever the engineer workstation will request it, the controller will be able to transfer it to the engineer workstation. So now we knew what is being stored on the controller but we also needed to understand what are the parsing flows in the engineer workstation. So, we were looking for the upload procedure and this is the main function in the toolbox STIF program for the upload procedure. And in the upload procedure, we discovered that it reads a lot of files and one of the file it reads is the device.zip archive. Now, we found out that the engineer workstation fully trusts the controller and so when the file is retrieved, it is being extracted without any limitation or sanitization and so we were able to find our first vulnerability, zipslip. Now, zipslip gives us the ability to overwrite any file we want in the system and we needed to think how can we weaponize the controller by writing any file on the system. So, what we decided to do is to search for DLLs that are being loaded after the upload procedure is completed and our goal was to drop a DLL that will get loaded by the engineer workstation after the upload procedure completes. And we did find such a target, it turns out that blockware editor.dll is a good target because this DLL is loaded immediately after the upload procedure and so what we did is we took the original DLL, we injected our own CIL code into one of the functions that we knew is getting called by the API and we weaponized an archive, the device.zip archive and we wanted to transfer this new weaponized zip into the controller and to do that we had to implement the SDI protocol and the SDI protocol is a proprietary protocol so we had to do a lot of reverse engineering in order to break the protocol stack and understand exactly all the authentication processes and all the procedures of download and upload and implement it to our self and the main functions we needed to implement are x18, x19 and also x10. This is to let the controller know that we want to start down malicious download procedure in our case and also override the archive, weaponize the archive on the controller and we actually implemented everything so I have a cool demo to show you. Let's hope it works. Yeah, okay, the video works. So what you see here right now is us weaponizing the archive on the controller. So this is the attacker side and now from the engineer side they're doing, they have a new platform, they want to do upload to see what's going on on the controller so they will go on the toolbox as the platform, plug in the IP of the controller and do upload. So right now all the different files are being, yeah. And now we have a cool run somewhere on the system. Thank you very much. Yeah, so one platform is not enough and I already told you we've researched seven platforms and we implemented seven different chains with many bugs which are all disclosed and fixed and mitigated, yeah. And I want to show you some cool video we've prepared with all the different videos we shared with the vendors, so let's see. Yeah, so right now I've seen all the videos at the same point so right now what we're doing is we're doing the malicious downloads so we're weaponizing all the platforms and once all the platforms are weaponized we will wait for our fake run somewhere to jump on different engineering workstations. These are the original videos we shared with the vendors so that's why you see, so it started with a calc on some of them and then it transferred to MS Paint and finally to the fake run somewhere we thought would be cool to show. Eventually, yeah, eventually we did end up with a full working POCs with our fake run somewhere on all the platforms and it was a great research and it helped us to understand some new concepts in attacking ICS equipment and POCs and thank you very much.