Anyone else who's coming anyway? Hey, good morning, can everybody hear me? Maybe, possibly? Cool, I'm glad you all survived Stack City last night. I can't see really what's going on out there, so I'm gonna assume that the whole room is full of people who passed out here thinking it was their hotel and just haven't left yet. So hey, my name's Kevin, I work for Cisco. And today we're gonna talk about migrating Novenette to Neutron. This is something that we did. And are working towards doing in production as well. We've done it in the labs. And so far it's worked pretty well. Not much tenant downtime, if any at all. But we'll get to that in just a minute. So why would you wanna do this? If you're running Novenette, chances are at this point, there's still something that you want about it. It's stable, but it lacks a lot of the advanced features that Neutron has. And so a lot of people are trying to move towards that. As a community, we're trying to deprecate it, but it just won't die, it won't go away. I think it was actually deprecated once, and then they undeprecated it because there were still enough people using it, like way back in Grizzly or Folsom or something back there. It's more difficult to find help when you need it, because again, nobody's running it except you. So you go to ask somebody else, and they're like, I don't know, man, I haven't used that in forever. Actually, I sort of came into this project as an aside, and that was, I spent probably two or three weeks just getting back up to speed on it because it had been so long since I'd used it. So that was a lot of fun. Most of the cool things are in Neutron anyway, back to lacking the advanced features. So you really wanna be there. I love the idea of green-fielding things and just maybe letting Novenette die on the vine, but unfortunately, that's not always an option. And who doesn't love brown-field migrations? So that's another reason why you might wanna do this, because it's a lot of fun. So what I'm going for here is, I think there's not, nobody's scenario is gonna be exactly the same. Everybody's got different things set up in different ways. And so this is sort of like a talk about what we did, and hopefully it spurs some ideas. If you're in, I assume if you're here, you're in a boat where this is something that you might wanna be doing. So kinda get started, this is what worked for me. So I had to determine my scenario. What does my environment look like? Where am I gonna go next? And then I documented all the use cases. You have floating IPs, do you not have floating IPs? Do you have like a flat tenant network? Or do you have per tenant networks or something like that? So I defined my beginning state, so I looked at everything that I had and I said, well, this is how this looks. Then I was able to replicate that in the labs and get everything set up exactly how it looks in the eventual production state. And then I defined what I wanted it to look like. So now I knew where I was and where I was going and then the shortest way between two points is a line. So we went from A to B.  So for me, what it looked like was I started with Googling where every good project starts. And then when that didn't turn up a lot because refer back to nobody's running over Net anymore except for you. I asked the ops mailing list and I managed to find some obscure GitHub, somebody who had done something similar. And so we went ahead and so I went ahead and looked at that and it looked like it could do what I wanted. So I forked it and then it became this code here which is this ice house tree. That code was originally set to work in Juno, possibly Kilo. So it had some issues like they were missing dependencies and whatever else like some Oslo classes had changed and stuff like that. So some imports weren't quite right. So I had to change it but it also was set up for a very specific scenario. Again, back to that's what's gonna happen. If you guys go and try to do this, your scenario is probably gonna be different than mine. So feel free to take my code, hack it up, fork it, whatever you need to do. Finally, once I'd gotten everything set up and working, I automated it. I have this repo of OpenStackTools that I've been working on since like 2011. So some stuff is really old but feel free to pop that out, look in the Ansible directory. There's a lot of goodies in there. Most notably the plays that I'll show you later in the video. So our specific scenario, we were using Linux bridge for layer two. We had non-overlapping tenant networks so every tenant had like a slash 24 or 23 or whatever they had which is standard for NovaNet. You can't have overlapping subnets in NovaNet which is one of the cool features of Neutron you probably want. We used, in this current situation, we used a hardware gateway for layer three which means that there's like a CSR or whatever else upstream and I don't really have to worry, in this migration I didn't have to worry about the layer three aspect of it. So I wasn't trying to move floating IPs around. I wasn't trying to do anything like that. We're working on a procedure because we do have some situations where there's software layer three and that will definitely incur some additional downtime just because if nothing else, like you have the art problem where you're switching an IP from this device to that device or whatever the case may be. So if the MAC address changes at all, you know, that's gonna be an issue. So something to keep in mind but for this particular scenario, hardware layer three which made my life a lot easier. So a high level look at the procedure. The first thing that you have to do is block client API access. This is very, very important. We still need the APIs for the stuff we're gonna do. I tried to defer to the APIs as much as possible because I figure Nova and Neutron know how to set things up far better than I do. I don't wanna be manipulating databases directly if I don't have to and praying that I got all of the joins right and all of the whatever else is correct. So, but you don't want tenants changing things while you're trying to migrate. So you need to block client API access. So that is a caveat I should point out is that there's not a control plane outage but tenants can no longer make API calls during the time of the migration. Their VMs will still work. They can still interconnectivity between the VMs, upstream connectivity through the hardware gateway all works, they just can't make API calls, can't launch new VMs, delete them, whatever. Then we gather up all the info we need from NovaNet. This is because there's the network cache stuff and if you're in the middle of something and like a bit of data expires then you will find yourself in a bad place. So we take all that data and that's stuff like the instance IDs, the MAC addresses, the IP addresses, all that kind of stuff. We gather all that up. We install and we configure Neutron. Excuse me, we start Neutron services.  Then there's a bunch of scripts which we'll get into which create network subnet supports, security groups, all that information that we pulled out of NovaNet it basically replicates it in Neutron which is running in parallel. At this point NovaNet is still in charge of everything. In our situation we wanted to make sure the DHCP servers stayed on the same IP addresses. I would recommend that. You don't have to do it. What'll happen is normally when you spin up a DHCP server in Neutron it'll just pick the first address available in the allocation pool and then it'll just work. The problem with that is if it's not the same IP a VM will go to renew, the address won't be there and then eventually it'll just do a full release and renew cycle but that often times tears down the network stack and brings the stack back up on the VM which could cause some problems for you so I recommend trying to keep it on the same IP address. That's one of the things that we do. We'll talk about that in a minute. We attach the ports and basically and I'll get into this again too is that that just updates the port binding in the Neutron database. We don't actually want to attach the ports on the compute nodes but again I'll talk about that in a minute. Rename the bridges and the interface on the computes to use Neutron vernacular so you've got BR whatever, VNet whatever, it needs to get changed to like BRUUID and TAPUUID and that kind of stuff. So again it's like the idea that we're going for here is we're gonna yank the tablecloth out and all of the things are still gonna be there. The table's still gonna be there so when Neutron takes over everything's as it expects. And then lastly we clean up the Novenette resources, get rid of all those old bridges and whatever else that we don't need anymore. So digging into that that's like the quick high level overview so digging into that. We start on the control nodes and this is kind of a control nodes logical. We touch the compute nodes a little bit here but everything comes out of configuration files so there's well two places. We store some stuff in the database, we create a transient table to do that and then but it's like just in any file for configuration network configuration stuff for the migrations so we create those and there's scripts in the directory to do all of that so it pulls the information that it needs out of the database, out of your NovaComp, out of your Neutron comp and it writes all that information out into a configuration file and then we generate network data and that again is what shoves the stuff into the database. Again that's a script that's as part of the Novenette to Neutron code repo that I showed you earlier. Then you install all the Neutron packages, you sync the database which just puts the scheme in place, you set up IP tables. In our case that's we've got very specific firewall rules so and a lot of this will make sense when I'm gonna show a video of me actually doing the migration later and so you'll see all this stuff running in the plays. We update the Neutron and the NovaConfig files on the control nodes and on the compute nodes and again that goes back to that port binding stuff we have to update the compute nodes to use the fake compute driver otherwise when we do the port attaches in Neutron it will actually attach the ports which you don't want to happen that causes trouble for you. Not that I know from experience. So in our case I leveraged our existing automation and config management to do a lot of that stuff so like installing Neutron and putting the config files in place and stuff like that. We have a custom thing called spine that we use but if you have puppet, chef, ansible, whatever it is that you use. I also wrote some ansible plays to do this stuff but I didn't include it because one spine does it for me and two that wouldn't really be useful for you guys anyway because this is our setup. So however you do that that's sort of a black box. There's this configure Neutron and get everything stood up properly mythical unicorn that you have to accomplish so and I did it using our existing stuff. Finally you run the migrate control script, the migrate security group script and the update DCP server script and again that like literally just makes Neutron API calls and says Neutron net create and it does all the things Neutron subnet created attaches it properly, Neutron port create, Neutron port attach, all that stuff happens. So now at this point your database should look how it would look had you been running Neutron the whole time but no one out is still in charge. It's worth pointing out that the security groups and the DHCP scripts, they actually have to manipulate the database directly. Just found that to be the easiest way to do it and it's a really, they're really simple calls. It's just create the things but it was far less efficient to do that with what with the DHCP it was I couldn't do it at all from the API with the security groups it was way less efficient so it just shoves that stuff into the database and everything seems pretty happy. So when you get done with this page like when you've done all the things on here your control plane is up and running you can run Neutron commands, you can run that list, you can run subnet list, all the things will show up and everything will look happy. Your VMs are still running because we haven't touched the compute nodes other than changing that config variable. Your VMs will still be running attached to NovaNet bridges and all that sort of stuff and your VMs will still be running at this point and your network will still be up and working. Now we move to the compute nodes. Up until this point everything you've done you can undo without causing really any damage. We've only modified the Neutron database, we haven't done anything to the compute nodes again other than changing that fake driver variable. So everything up to this point is undoable. Even I've had to undo it a lot of different times I just had a for loop that would go through and list the nets and subnets and all those things and just delete them. Once you start here this is when you're in this is when you're on the dark side of the moon and you can't really come back without causing a lot of trouble. So when we go to do the compute nodes as I said here and I mentioned earlier that the migrate control also issues port attach API commands and that's why we have to switch to the fake driver. And again that just lets the Neutron ML2 agent set everything up in the database and make everything connect properly. So from its point of view it actually is the one who attached the ports. That way we know it's just working well. I tried this a lot of different ways and ultimately I came up with this idea that every compute node needs to have its own individually configured file. Config file, I'm sure there's another way you can do it where you add a tag in there and the compute node figures out which things it needs. This is the way I decided to do it, it was just easier and I'm super lazy, so it works great. But what that means is that we basically have to run it on the control node because we need Neutron config file which hasn't been created yet. We need the Nova config file. We need to pull stuff out of the database. In our situation the compute nodes are using conductor so we don't have direct access to the database from the compute nodes. So on a control node I run this generation script and then I pull all the config files to each individual compute node so that way everything gets on there super happy. I did it all in Ansible so again you'll see it when I run the video it all happens just really quick and automatically. So then we run the migrate compute script and now this is where the bridges and the VLANs and all those things actually switch over. And I mean it literally just issues IP commands, it issues V config commands, it issues VR control commands, renames everything and just that's where you're pulling the rug out.  And so we can just go through and remove that so I have a script that does it, it's hit and miss so sometimes I have to go through and I wanted to make sure I didn't accidentally delete something that needs to be there still so I made it safe and it gets sort of the majority of all that old craft and then I go through and clean everything up manually if there's anything left. So that gets us to the point so like at this point everything is actually done. Your control plane is under Neutron control, if you're issuing commands it's going to Neutron and Neutron is talking to Nova on the compute nodes Nova has got, it's set up to use Neutron as well so it's gonna be using the Neutron ML2 agent so at this point everything you do is under Neutron control, you're basically done so the last procedure is you make yourself a drink because you deserve it. If you've gone through this you have probably already started drinking but I highly recommend you continue. So all right, demo time. So what I'm gonna do here is I have about a 13 minute video it's unedited with the exception of there's one point where our config management is running and it takes like 15 minutes and so I just sped it up so it takes like 45 seconds instead of 15 minutes but otherwise the video is unedited and it's just me doing the migration so stop me if you can't see this, hopefully you can and I'll sort of walk through what's happening here. So you can see here that we run you see all of the tenants are there I'm gonna ping one of the VMs here you know just give it a couple of pings and you can see it's up there it's running so at this point this is still NovaNet nothing else has happened we haven't really started doing anything, ping another VM just because I'm thorough and so then I'm gonna jump into this Ubuntu machine so Syros doesn't have screen installed and I just didn't want to deal with trying to get it installed so I spun up Ubuntu VM excuse me I'll hop into a screen and I'm gonna ping 8888 from a screen you can see that it's working there so this VM is now going up through the hardware gateway and out to the internet so that's all working so we'll start it again I'll detach the screen and I'm gonna leave that running the whole time we'll come back to that later that's a fun little surprise for later so we'll look at our VMs again so I can get the IP addresses I also can't type you never feel more vulnerable than when your screen is on display for everybody and you're typing things wrong so that screen down there is one of our control nodes and so at one point during the control plane migration you'll see that those pings will stop and that's because that's when the VLANs get removed from the control plane so there is a slight period of time where the VMs won't be able to reach the control plane the only time I really think that could be an issue is if you have really short DHCP leases because it might take more than a couple of minutes depending on how long it takes for your config stuff to run so you can see what's happening over here in the Ansible plays so the first thing I do is I clean up my Git environment to make sure that I'm pulling the latest stuff I check out the code it runs create comp you can see over there that Neutron is not running I generate the network data and so you can see the third or fourth task down there it's running the generate network data stuff you can see Nova Network is still running Neutron was not running I'll clear that screen and I type dollar signs instead of ans that's awesome so this is where our config management stuff is running and so you'll see that those pings down below they're gonna start speeding up this is where I sped up the video it goes from like a hundred to maybe 800 or 900 pings so this is where you would have Puppet or Chef or Ansible or whatever thing you have it's installing Neutron it's putting the config files in place it's giving it all the correct value so it can talk to the database and your message bus and all that goodness ah there we go, hey it's done if only it actually really ran that fast so now the database is actually this is what actually instantiates the database so we create it and we stick the schema in place we stop all of the services this is because of the way that we use that we do high availability basically I'm trying to set things up how my config management system wants to see it so I stop everything from running I re-enable IP tables I had to block it earlier so that stuff wouldn't get flushed and restarted so we start the Neutron APIs we load all of these IP tables, chains you can see down there restart Neutron Server rather so now you can see Neutron Server is running but there's no endpoints created and that's what's happening right now is that we're populating the Keystone data so as soon as that finish now you'll see Neutron Netlist returns but it's not returning anything because we haven't actually stuffed any values into it into the database we haven't created networks or anything like that so we do lots of starting and stopping and restarting and again that's all just normalizing the system to be how our HA system would want it to be set up so you can see there the VMs are still active and running everything is happy so now we update the compute drivers now we're doing stuff on the compute nodes we update the compute driver to the fake driver that was what I was talking about so we can issue the port attaches we fixed permissions on directory so now that playbook is completely done and our control plane is prepped all the config files are in place everything's written out to disk the proper APIs are stopped and started so one of the things you'll see here and I think I call it out is some of the VMs are changing to no state eventually that's going to happen to all of them that's not a bad thing basically what is happening is there's no longer a valid compute driver the fake driver doesn't return any information for the VMs that are running and so it and so the NOVA what gets the state of the VM in the API or rather in the database as returned by the API just turns to no state that'll turn back to running when we fix it later so trust me you're fine here VMs are still running do not panic don't forget to bring a towel so now we're actually running the control plane migration so again we make sure that the config file is there and that's just sort of a way to make sure that the code is checked out and then it's been created so we you can see there I did the there's the run migrate control there's the run migrate security groups the run updates the DHCP servers and then we restart the DHCP agent and then all of that finishes and then at this point now you've created all the networks you've put the DHCP servers on the proper IP addresses and then we revert the MHVs which are what we call compute nodes and that changes it back to the real driver so now we're going down over here you can see that when the VLANs got removed those hosts became unreachable but now we're looking at you can see that the DHCP agent came up the namespaces were created we'll jump into one of the namespaces and you'll see that I can ping one of the VMs from this namespace showing there's connectivity now between the control plane and at this point your VMs have connectivity back to your DHCP servers so fear not you can renew your IP address which is really important so now at this point this is where we finish that first slide the detailed procedure for completing the control plane migration so everything is back to normal in a second here I'll go back over and you'll see that the no state is turning back to running eventually they'll all change back to running as the actual real compute driver starts returning proper states there you go they're all back to running so now we're going to prep the compute node migration so this is where we take where we put all of the neutron values in our nova.com for our computes and actually switch the compute nodes over to be talking to neutron so you can see there that's where we generated the files on that control thing and then I copied them all we fetch them and then I'm going to copy them all to the correct compute node so everything's going to be in place so when we actually run the migration it'll have all the information that it needs so copy them into place then you can see back over there again pings are still running everything is still up and your network is pretty happy so when we do a net list over here you'll see now the neutron nets are there the ports are all there so there's an when we do the subnet list you're going to see so what we did here is I created we created a bunch of different allocation pools because there's no reserved IP option in neutron whereas there is a nova net and so some people have like a piece of hardware or something in their tenant space that they don't want that IP address to get assigned to a VM so it's a little bit ugly in the subnet list but basically what that is is it created allocation pools that just surround any reserved IP addresses so neutron can't assign anything so now you can see over here we're on a compute node and everything is set up for nova net like it's got the br with the vlands and you can see like the vnet adapters and all those things so then we run the compute migration so this goes through it makes sure all the config files are in place and it actually runs the migrate compute now we jump back over and you can see everything is switched so no longer is it under nova net vernacular now everything is under neutron you can see the old nova net bridges are still there but all of the vlands have been changed to neutron stuff all the tap devices are changed all the bridges are attached to the proper neutron bridges and all that goodness so then we clean up the bridges this is that step I said is kind of hit and miss in this particular case it gets all of them but one so now when we be our control you can see we got rid of all of them except that one and that's mostly just to be safe just don't want to don't want to go through and accidentally blast something we did such a great job of keeping all these VMs up I don't want to go through and accidentally take one down so at this point you're basically done everything is under neutron control everything is happy nova net is not even running anymore its bridges have more or less all been deleted and destroyed we can see that everything still looks great so all that IP information is being returned from the neutron API now it's not being returned from the nova net API and then as the kudita will jump back into this ubuntu VM and we'll see how our ping to Google is doing spoiler alert it's doing okay so there it is you can see it's still running it's still pinging when I stop it you can see there were 1,602 packets transmitted and what is that 1,600 received so I dropped like two packets in the entire thing which is tan amount to 0% packet loss so then I take a little bit of time here and figure out how to create a new VM because I don't often launch VMs from the control from the CLI so I had to like look up the commands but we'll boot a VM and we'll see that it comes up and that it works properly and again at this point everything is under neutron control so we're completely migrated and the VMs had as you could see a modicum of downtime if you could even call it that so at one point in here when you go to so like now that you've done this you could as we move forward with doing software layer 3 we're going to this would be the point where we would migrate floating IPs and stuff like that so it's kind of a building block approach which is why we've only done the hardware gateway stuff so far so we'll hop into one of the specific tenants that you can see there it's got two VMs running and then I'll figure out what a nova boot command looks like and had I been less lazy I would have edited this better for you to not take a bunch of time doing that you know what can you do right I had 40 minutes to fill and about 25 minutes of content so that's a joke I'm just kidding yeah so as we move forward to do software layer 3 this is where you would do that now I'm starting to work on it I haven't really come up with exactly what we're going to do yet but I mean it's going to be procedurally it's going to be a very similar thing where we take the information out of NovaNet stuff it into Neutron and start agents and whatever else needs to happen so that the IPs move over and then again you can prep everything ahead of time the downtime I anticipate happening will be when you actually do the cut over and you know like the switches are updating ARP or whatever so you can see over here I'm slowly figuring out how to launch a VM creating my security groups and all that kind of fun stuff we'll just name it something that is obvious that it's a post migration VM booting and hey that seemed to work but does it actually plum again the spoiler alert is that yes it does ah I'm the worst my kids hate me at Christmas so you can see it came up there it's active it's running it's got an IP address we can ping it and it returns pings you can SSH to it if you actually type the correct IP address the first one will I'm going to the wrong VM so I get a key denied but then I realize my mistake and fix it jump into the VM and it can ping the world and everything is happy so now you have a situation where we're running Novenet now you're running Neutron you can create VMs everything is happening the way you would expect it to happen and your boss hopefully is going to give you a raise that would be nice wouldn't it so that's it that's how it happens that's how it works are there any questions and if you have questions please go to the mics because they're recording this and I can't see if any oh I think I see a silhouette I have two questions the first one is just because I was five minutes late to the session sure the migrate control script you talked about is that something that's out and ready for people to use or was this met as a proof of concept of a future thing yeah so I'll make the slides available I've got a bunch of code repos in there with those Ansible plays also with all of that Novenet to Neutron migration stuff but the caveat that I gave at the beginning was that I mean you can probably use it it should work but you might have to modify it because I originally found it somewhere and it was for like Juno and Kilo and I had to heavily modify it but please like take it actually I can go back to the if you want to take a picture of the of the thing there you go so like those are kind of the repos where I pulled it from what I did and then my Ansible stuff so yeah feel free to go for that but I wouldn't recommend just running it you're probably going to want to look at it and make sure you understand what it's doing first but I mean you can hit me up on Twitter which might be out of scope is that every time that I revisit the issue of migrating away from Novenet I'm told by Neutron people and also by the network guy at my work that Neutron doesn't support our current use case which is having a single pool for all internal IPs I think that it does we can probably talk about that more if you're interested like I would have to understand more about your specifics in the area what did your Novenetwork topology look like we had individual tenant nets and obviously they're not overlapping but I know that Novenet supports like the flat network model like the quote unquote flat DHCP and I'm pretty sure you can do that same thing in Neutron I haven't so that's why I'm a little hesitant to say for sure but I'm pretty sure you can but there wasn't like a big announcement that this thing that didn't work before now suddenly works I know where he's thank you Hi in during your development cycle how often what percentage do you say that the DHCP servers changed IP addresses from what it was supposed to be almost every time and that's just because again it takes the first especially after I implemented that allocation pool stuff because it goes into the database unordered and so and then the DHCP the DHCP agent just pulls the first available IP out and so especially after I did that it would come out to like 154 like every time and stuff so I mean it's fairly repeatable like you can see what it's going to do and how it's going to do it but at the same time when the data gets shoved into the database that is completely random so you know so that's why I just went ahead and did the way that I did it and basically what happens is I update the database restart the agent it tears down everything and brings it back up with the correct IP address and so I just found that to be a lot safer and safer yeah it is thanks you said this was an ice house is the neutron you're running still ice house as well it is and then when you oh is it oh I'm sorry my mistake yes it's Juno neutron yeah thank you chat and then when you switched what is there bugs that cropped up or bugs that got fixed or anything like that operationally that would be interesting between the two you mean between ice house and Juno no between no the network and neutron yeah but I mean more than anything it's really just we're trying to normalize our system so that we can so that everything can do because we have some deployments that are running neutron green fielded and some there now are obviously going to be brownfielded neutron so that was really the biggest impetus for it was to get everything flatline and we also have tenants who just want to be running neutron for you know whatever other features it offers so that was that was the big thing as far as bugs go I mean that's a whole other discussion you know why would you run no vanette over neutron or vice versa and I know that she would probably love to talk to you about that as well right right exactly it's I mean it's the windows versus Mac thing again right it's like which one's better it doesn't neither is better or worse it's just what's what what are you trying to accomplish and what's the best tool for the job we started with neutron I was just curious though yeah I'm sorry so he said that they're bait it's basically apples and oranges was what chat was saying that you know but most importantly is if you're running if you're running no vanette and you think you need to go to neutron don't just take someone's word for it you should probably stand it up greenfield it somewhere make sure it does what you want it to do and make sure that it does it like you know well or whatever and then make a decision if you actually want to be running neutron over no vanette but also keep in mind that again the community is sort of moving away from no vanette which a lot of people find to be an impetus but we also will keep supporting it as long as we have to but your support may be slightly diminished thanks yeah thank you sir so you're running ice house nova network with vlan manager and with an external gateway l3 type setup correct it was a hardware external running multi hosts with nova network no no so you weren't doing the per nova compute correct and snadding and dhcp correct all that happened on the computer on the control nodes and then we had a pacemaker running h8m manage all of the failover and what not are you aware of floating ip l3 stuff aside but at least on the every nova compute host and multi host runs its own dhcp server and fakes out all using the same dot one address in the gateway with eb tables rules and all that stuff if that would migrate over that's a really good question I don't know and so when you would migrate over conceptually the database in your new neutron database would reuse all the dot one dns mouse server ip yeah so that way you would have one kind of dhcp server host per network pertinent network yes so it's not that it wouldn't migrate over it just would migrate over with different caveats in architecture right yes yeah that makes sense the second question I had which was kind of related to the previous one a bit um given what I understand of neutron in the ice house juno time frame there's a pretty different network security feature set between nova network and neutron like thing uh ip address spoofing mac address spoofing did you put any mitigations in place to prevent that because I believe the stock neutron stuff allows allows just anybody to spoof at that point yeah so we did previous to this um I'd done a lot of work to help with that because I mean I know one thing in particular that you're talking about is the ARP spoofing and stuff like that and so we basically kind of had to pretend we had to do fake layer 2 protection at layer 3 um and some stuff like that and so we've been able to mitigate it I wouldn't say it's certainly fixed but yes you're absolutely right that that that's that's but you're mitigating it up at the l3 level not at the yeah so basically what happens is you can poison bridge level where like if you're if you've got two VMs on the same host one could spoof the others address or well there there's so there's layer two protections because it's all it's all VLAN segregated so you can't like if there's two within the same project VLAN yeah so so a tenant can do but because we have pertinent VLANs and there's nothing shared um we felt like you can still so what can happen is you can still spoof so like VM 1 can still spoof VM 2 but then we've stopped it at layer 3 so it won't come back so like you can't actually steal any information out of it we felt okay with it because um it's only a tenant and if there's a compromise in the tenant then I mean that's a different problem to solve anyway but then on that same topic any major issues with security groups moving over between yeah but we just shove them in the database and and start the layer two agent which creates all the proper IP tables rules and everything works alright we're about out of time so this will be the last question oh yeah I was just going to add that um we're kind of still stuck on Nova Network in Kiello because we use it in multi-host mode and DVR look like network witchcraft you're not wrong we have like a couple hundred tenants in some of our clusters and we weren't confident that it would scale to um you know 200 neutron routers on every single compute node um across hundreds of compute nodes it's like you know M times M kind of disaster waiting to happen so um I just wanted to add that to the discussion that really complicated things a lot for us and we're probably looking at like a green field kind of rebuild to get to neutron right which kind of goes back to what chat was saying like at the end of the day you got to look at your scenario and that was why I started out by saying you know you gotta look at what your start is and what your end is and figure out how you can do it and if you even should do it um this isn't necessarily for everyone but this is what we did and I think it's a pretty a pretty sane path so I appreciate the comment thank you guys I really appreciate it enjoy your summit now if the most important session is out of the way you can get drinking


Well, hi there. I'm Sandy Allknock, artist and paper crafter here on YouTube with a stamped watercolor Christmas card and a marker conversion chart. I've been asked forever for conversion charts for all different kinds of things and I finally have done one for 20 colors in water-based markers for Tombow, Marvie, Zig, both the clean color and the twin tip markers works for this and distress. They're not perfect matches because there's just no way to make perfect matches and there might be other colors you could substitute here and there but I wanted something available for a particular reason. There are classes coming up at art-classes.com and we're going to be doing some stamped watercolor classes with art impressions watercolor stamps and I wanted to have a common set of markers or common set of colors that everybody can use so you can grab that conversion chart and pull out the colors that you're going to need or you can wait to get this particular set of Tombow's. It's the one I'm going to be using and Ellen Hudson is going to be making a set of just those colors eventually and there will be a discount so hang tight on getting your supplies for that class but I'll talk a little more about it shortly. For the card for today, I'm going to be using the Winter Kids stamp set and it's got lots of elements to make a scene. I love the fact that the snowman and the kid are hugging and they're in one stamp so you don't have to actually merge them which is nice. You can use these other stamps to create a scene though and I'm coloring this one with a black, a brown, just kind of a couple different colors, a little blue at the bottom for the snow and I did some tan in the area where the fire is going to be in the lantern and the reason is because that won't contaminate my yellow so I can color the yellow straight in there and make a little flame and not be concerned if I have a black or brown on the that line down the middle that it's going to just kind of bleed out into my flame and then all I have to do is add some water to it. I'm using a number eight silver brush and if you did want to get something for the class that is one of those things that you're going to be able to get and that's the size that I'm going to be using throughout the class. Most of the art impressions watercolor stamps are not ones that would warrant a number 12 brush so when I tell you you can get an eight or 12 that won't be one you'll really need probably for this particular class. So now we're going to work on the big image and you could do a card with just with that lantern wouldn't that be beautiful? Just something really nice and simple. You can add all the stamps. I like to color all the things so I'm going to color all the stamps from the set into my little scene and I'm just going to use various colors to add on to this. You could do all of this in one color and then paint in all the other colors but I kind of like to get a start for them and use the color that's in the outline when I actually do the painting when I start adding color to it. Going to be showing you some real good tips for strengthening color. You guys know me I love to do contrasty things so I'm going to be trying to add a lot more punch to this than you would normally get just from using the watercolor outlines. So I'm going to wipe off this side of the snowman for a particular reason. You'll see why in just a few minutes because I want to stamp the little kid that's leaning down and grabbing a snowball. I want to stamp that one right there and I don't want to have the snowman in the way so actually that whole left side of the snowman disappeared entirely. That was the first thing that I colored and part of it might have dried so more than just the area that I wiped off ended up disappearing but that's okay. Doesn't really mean that any kind of problem we're going to be able to fix that. This was a little bit of a problem that I did on purpose. I wanted to show you how to fix that if it happens to you because when you're coloring multiple colors on a little stamp like this you're very likely to have one color on one side one color on the other. Fortunately with this it's red and green. Red and green are complementary colors and they're actually going to work fine together because we're going to make that little snow suit on that child. We're going to make that kind of dark and the red is going to actually help eventually. So here's the little kid that I'm stamping right in that empty spot and when you're stamped something over top of an image that way it starts to make them interact in the picture instead of just making a whole row of evenly spaced images or everything looks like it's on the same plane. This makes it look like one is in front of the other and this way I didn't have to do any crazy masking. I just wiped off that half of the snowman and I could stamp the child down there. This little kid I had a happy accident happen. The hand I forgot to put any color on. Whoopsies. There's just a little hand held out but I'm going to change it. I'm going to just do something different with it. I'm going to add a little bit of green because what I'm doing on each of my kids they're either wearing a green outfit with red accoutrements like hat and scarf and boots or vice versa. So she's going to have green gloves on and then I did a little circle, a little half circle of blue that is the snowball she's holding. So she's just kind of sitting there ready to get involved. Now you'll also notice that that red went really pink and most reds do go really pink. That's just the way water-based markers are. They're not intended to be really dark in color. So I took two reds. One is a deeper red than the other. One is more of a tomato red. The other is a little more of a cherry red and I'm mixing them to create a darker color. I took one swipe of black and just put one swipe of black in the middle of all of that and look at how much richer the color is. You can increase that. You can add two swipes of black to make a darker color. That's one of the reasons why you don't really have to have a whole ton of markers just like you don't have to have every watercolor known demand because you can mix them and we'll be doing a little bit more of that as we go here. So the lantern here is going to cast some light down below it. So I'm going to put a little bit of yellow. I'm risking yellow snow. I realized that. I didn't realize it completely until I started putting it down there and started thinking yellow snow might not be a good thing but there you go. I'm just going to blend that really quickly. I didn't let the yellow dry before I put in all of this other background because then I'd get a hard edge between them. If you want something to be really well blended then you've got to do it right away. Just do it while it's wet. In the stamped watercolor classes we're going to be having soon. I can't wait. They are filmed and the reason actually that I'm holding off on launching them if it hasn't launched by the time this video goes live which I don't think it will is because we're waiting for all the supplies to get in and we're waiting on manufacturers to ship things etc and I want to have everything available for you and a coupon code at the same time. So hang tight and that will all be coming very soon. So notice that I'm painting all of that blue on the snowman because the child is shading the snowman. I want all that light to just be hitting the edges of everything that's facing the light. The light is kind of in a center spot so the child on the right has highlights on the left side and the kids on the left have the highlights on the right side and that sort of thing. So when you have a light in the middle like that it kind of changes things a bit. So I mixed up some extra of the darker red color and started putting a little bit more in the child on the left and then I'll just start watercoloring some of these other portions and on the block I just kind of keep putting extra colors that I want to paint with and that I want to use for the watercolor and just picking them up. I can control how dark they are by how much pigment versus water there is and that's something you kind of have to practice but what I'm doing here is actually using the marker directly on the paper. It can be scary to do that because not all watercolor markers will water out the same and that is one of the things that in the pre-class I'm going to have a whole video talking about it but I tested out all the different brands of markers. The Tombos do the best job at not sticking and being unable to blend so that's why I'm recommending using them because all the other marker brands are just tougher to use. If you've tried using these art impressions watercolor stamps with other marker brands it's possible that may have been your issue but also I find that individual colors sometimes are harder to move than others so sometimes it's just the marker itself and it could be that it's got dry and it's just not moving very much could just be the pigment itself. There's a lot of different factors that it could be but look how rich that green color is when I went directly into painting it on the paper itself. I'm doing it quickly I mean this is sped up but I'm doing it quickly. I take the quick swipe of the dark green on the paper and then immediately go in with pigment from the acrylic block to get some blending going so darkening up a little bit of the red I mixed up some of the darker red again and just going over one more layer on that you can get all kinds of crazy with your depth on this and mix more of that dark color again put more of the black in there to darken that red but I was kind of feeling this was good enough for horseshoes and hand grenades as my dad used to say and now I wanted to put a sky in the background and I'm going to have that horizontal line the horizon across there so I wasn't worried about getting a hard line there but I'm going to be really careful about creating any kind of hard edge on my sky and I'm actually painting this upside down I turned the video right set up so you could see it a little bit easier rather than watching me paint upside down but I've turned upside down so that I could kind of get access to everything more easily but I'm making sure that I keep all of the leading edges wet and this is the same with watercolor except it's more urgent when you're watercoloring with water-based markers. Water-based markers don't like to move nearly as much as watercolor. Watercolors have a lot more time to deal with the things like this but whatever the outside edge is that and that's what I call the leading edge that outside edge needs to remain wet even if you just wet it with with water alone that will at least give you a soft edge to blend with and I keep adding more color onto my block and I have to do this quickly so that I can get the whole background in there. I'm also risking I know here the possibility of bleeding that hat out into the rest of the picture. The folks over at Art Impressions I've noticed when they do backgrounds they'll leave a white edge around things. I don't like to leave a white edge that's just the way I am so you can leave a white edge around things. Go look at some of Bonnie's videos and you'll see some of the beautiful work she does. So my final touch is going to be to add some strong shadows and the shadows are all pointing out from the light out in a variety of directions and if you've taken some my other classes you'll know I've talked a lot about shading and stuff. I'm not going to get into a lot of that in these two classes that we're going to be having. There is going to be a jump start and a mini intermediate class. I recommend definitely doing the jump start before you do the mini but if you're very experienced you'll be able to jump over and do the intermediate if you wish. But both of them are going to be so much fun. I can't wait. Now I'm adding some just kind of extra finishing details. I wanted a little stronger shadow on my snowman. You thought it was dark before when I had the blue on there. Look at me adding some gray to it and then once it was all dry I added some snow to it which I always do with my white pen. Don't we just love the white pen come Christmas time. I actually have already run through so many white pens so I'm going to have to order some more real quickly so I can finish making Christmas cards this year. Stay tuned to my channel. Make sure you're subscribed if you want to get the first word when those classes are live and available. And I will see you again very very soon. You can find links to everything in the doobly-doo or over on the blog and more information as well about that conversion chart. And I will see you guys very soon. Take care. Bye.


Cool. So hello, everyone, and welcome to Cube Day Japan. And we'll start the session now. Secure and debuggable, debugging, claim, slim, scratch, distillate, Kubernetes containers, a lot of terminologies over here. We'll try to, you know, simplify all these and see how you can debug it. So expected background, as, you know, Kyle mentioned some of the container fundamentals is needed. You should know what Kubernetes is, you know, the basic knowledge, namespaces, fundamentals, architecture, nodes, and all these things. And then we'll cover the rest of the stuff. So this is about me. I'm working as director of technical evangelism at Sevo. And I'm a CNC ambassador, very active on Twitter and also have a YouTube channel. So you can kind of follow me everywhere. I might reply you in between the session as well. And I'm the Docker slim guy. So keep going. Okay. So the next slide. So this is pretty much one of the best ways to describe the problem using minimal container images. You want to use them, but you can't a lot of times because you need to debug them. And up until pretty much now, Kubernetes 1.25, it was one of those things on the wish list. And you couldn't do it. Well, you could do it the hard way, but in terms of mass adoption, it wasn't possible because to use smaller containers, you still need that debugability. And this is what we're going to talk about. And apparently, you know, there's a lot of interest in it. So let's keep going. So let's get started with the general debugging techniques for for Kubernetes, which everybody, like we have been doing it just a quick preview of that. So whenever there's any failures, we go through the events, the logs that describe host and node level debugging. So we as a search into that, use privileged containers, use some embedded debugging tooling. Also, the Cubsidial exec, which is pretty common. If you want to get inside the container, you use Cubsidial exec, go inside and do a bunch of stuff to see what went wrong. And then some sidecar stuff, which is there. So yeah, this is some of the stuff like if you do Cubsidial describe, you can see the, you know, the events like if a pod is spending in sufficient resources, all these you'll get. And if you have a crash loop back off, you can do Cubsidial logs. And sometimes you have to do you can also attach that hyphen from previous so that you can get the logs on the previous container to see what failed. It's useful. And then the exec one, the Cubsidial exec interactive pod and the bash or SH whatever is there inside that particular container that you can attach to and run the commands with the debugging utilities. So let's let's actually see that in the action. So I do have some of the files. We'll try to apply the pending part one. So this is just I have taken from Robusta. They have some good examples. So you can see the pod is already in pending. I can describe that and we can see that, you know, it's not there because I didn't match the parts node affinity. So when you describe, sometimes it helps, it will give you the useful information that you can fix it and then, you know, use that similar to that. Another one example that I have is crash loop. So it's content. So it's already crashed loop back off. You can see that there's a pod which crash looped and I can see the logs for that. So environment variable is undefined. So these are standard and very simple to use the first level of the basics that you have to go through whenever you are debugging the containers that describe the logs, the exact events and all these things. So this gives you a bit of gist of how, you know, you do debug it now over the time what has happened is we have moved to like we have been creating the containers, but now we have moved we want to move to slimmer images to, you know, the images which contains less so that, you know, we have less secure vulnerability vulnerabilities in that and the image size is less. So with that, there are some concepts scratch, destroy less and slim. What are these? So scratch image is basically an empty image. It has nothing in it. It is used to run the binaries which don't have anything linked to that. I mean, yeah, there are some gotchas that we'll be covering later on.  So not everything that you can run with that. So it is a compiled binary which is there that have no dependencies and if it has the dependencies that probably you have to know all the linked dependencies and you have to, you know, manually get that because it doesn't even have the like you can see the CA certs, ETC password, nothing is there. So you have to make all the things up in the multi-stage builds and then you can run it. Yeah, that's where it gets complicated. That's where it gets complicated. Coming to destroy less. So destroy less is not completely empty. It has some of the base directories which helps you in creating the containers and running them. It has like ETC password, CA certs. It has the app plus the runtime dependencies. Both of these have no shell and no package manager. So that's where it becomes tricky, right? These are good to have, but these are only good to have if you have everything sorted. Like if you know your container won't fail, but if your container fails, then probably since it has no shell package manager, it becomes difficult to debug these set of containers and that's what we'll be discussing and showing you the demos for. So the gotchas, extra dependencies are super complicated. Whenever you have extra dependencies in scratch image, you have to know them beforehand if you even have to run it. No debugging tooling, no shell and not always a static binary. So this is the three apps that we have. You can see there is a big image. The size of that is one gigs. Pretty big one gig. And it's a simple hello world app and it's already one gigabyte in size. Then you have the distreless app, which is 166 and then you have the node app slim. Okay. So you have a lot of options in terms of how you want to get to, how you can get to a minimal container image for your application. Some of them require a lot more work than others. Distreless is probably one of the most known ways to get there. There are several flavors of distreless starting with the most basic static that gives you mostly a directory layout and a couple of things on top of that, including the zone info. That's pretty much the biggest chunk of that image. And then we have other images like the next level is the base image, which includes the static version plus a few extra libraries, basic OS libraries. And then you have application specific distreless versions. They're much bigger. They include the base plus the additional system libraries and the application runtime. Now, and I'll talk about the gotchas later on. So let's take a look at the distreless app and all of them, they use the same Kubernetes manifest for the application, but the Docker file for them is a little different. So, but this Docker file, one of the interesting things about it is that what you end up with, you might have a multi-stage build and then in the final stage you copy stuff to your deployable image. Now, in most cases, you end up with those kind of copies where you copy a whole directory with things because otherwise you're not really sure what you're doing. So that's okay. It's better than using a FAT image, but you still have a lot of attack surface left in the image when you do that. Let's keep going. So, this is an example for those compiled languages because sometimes, you know, with Go, the first thing that comes to mind is static binaries, a single binary. You can easily create a scratch-based application image and that's pretty much it. In reality, for real applications, real world applications, you actually have extra dependency certificates, user information, and in some cases, it's not even compiled statically. So this, let's see, okay. Yeah, let's keep going. So in this case, this is a sample app for Docker Slim, one of the third-party apps, and it's not compiled statically. So we end up pulling stuff like lib, pthread, shared objects, etc. So there's a lot of stuff that gets pulled in and that's when it gets tricky. When you're trying to do it the hard way, build those minimal container images by hand. So let's keep going. With a Slim image, the nice thing about it is that you have a regular Docker file. This is probably the most basic node application Docker file. It's not multi-stage, it's simple, derives from your node base image. You copy the package manifest, you install the dependencies, you copy the app, and then you just run the app. That's all. Now, that's all you would have for a regular app and then with the Slim images, it takes that and then it produces a much smaller image, kind of like out of scratch. It does it for you. So let's keep going. So now we have discussed what Slim image is, what Distillis is, what Scratch image is, and we have discussed some of the toolings. But I mentioned one particular point that it's hard to debug the Distillis and Slim and Scratch images due to the lack of shell and debugging tools within that. So we cannot use the standard stuff which is kubectlxxcp, embedded debugging tools because it's not there. So then it becomes difficult. And kubectlxcp, just it requires a tar to be there in the container else it will fail and it copies files from the directory. Exec needs a shell to be there and debugging utilities like et cetera, the netstat, whatever you need. These are not there. These are the standard ones. And if you try to do that over any of the images of Distillis or Scratch, you'll end up in getting the error that it is not able to process that because the shell is not there. So it won't be able to run. You won't be able to do the standard easy debugging that you are used to. So that's where we have the concept of ephemeral containers. So ephemeral containers are the containers to help debug the ports where there is no way to debug directly using exec. So what happens is it is when you write a pod it will you like you will be using kubectl debug to create those ephemeral containers and it attaches the in the live pod state itself. You will have a pod in the spec section, pod spec section, there will be a container that will be added and it will also auto update the container statuses with ephemeral container status. So that's the very good thing and it attaches to the container to the same pod, share the namespaces so you have that same process. You will be able to see the process of a Distillis or that particular container itself and you can have the file view as well that we'll show you. Some of the things to be taken care it doesn't have any readiness, liveness, probes the resources are not allowed on that and you can create it using the ephemeral containers API. Kyle will show the demo for that as well. Question for you, why not use regular containers? Why can't you just add a regular container to your pod? So again, adding a regular container won't help because you have to add that and the mounting becomes the issue with the process ID, the sharing of the namespace, all these things won't happen. So that is where the ephemeral containers will help and basically not touching the existing pod, not starting the existing pod. So that stays there. Your application stays there and if anything goes wrong, you still will be able to debug that. So this is the simple example of like Qubectl debug and let's say there is an engine export running before that. So you are debugging that and you are using the image busybox and you are getting the share. So what it does it, it is adding a new container with busybox image to that engine export. So if you see the like if you get the Qubectl get pod siphono yaml, you'll be able to see two sections which are important to work here, which is ephemeral containers. So whenever you do the Qubectl debug, there will be this section added automatically to that particular pod spec section and also the status, ephemeral container status will keep on getting updated. So this is just pod level debugging, same pod copy for advanced debugging. There is one command which we'll be discussing later and some of it uses ephemeral containers and other stuff doesn't because you don't always need to have ephemeral containers. For example when you get a pod copy there, you can use a regular container because it's the time when you get to decide. Again, this is a simple Qubectl debug demo, but it is creating a container debug sidecar box with the image busybox and attaching to the target container app. It is also like you can see the process over here. So this is the process. You can see the process one, which is the node application, but the current process for our ephemeral container is 13. And one thing you can see the difference between the file. So it doesn't look same. So your file system won't be exactly the same like when you do a Qubectl exec, you are into the pod. You will be able to see the exact file system over there, but when you are in the ephemeral containers, you won't be able to see the exact file system directly. Yeah, and that's why you need to go to the proc file system and you need to navigate to the PID you're trying to debug and the root section. And so this is odd. If you haven't done this before, this is odd and not great from the debugging experience and we'll have a demo to address that. So you can also use ephemeral containers without the Qubectl debug commands. So you can use the internal APIs and that is very interesting because there's a demo that we have to set the security context. Like if you have to set the security context or if we need any additional mounting, which you cannot do by default using the Qubectl debug, so you have to use the APIs for that, the call examples. Some of the gotchas, you can't remove the ephemeral containers, not all container properties are available, which we discussed, process name, space sharing and the security context and mounting volumes are not there out of the box when you use Qubectl debug. So ephemeral containers like Kyle mentioned, it was a fantasy to use before the slam scratch and destroy list and they have been a lot of work effort and getting this set of ephemeral containers support natively into Kubernetes and that happened very recently. So it went in GA 1.25 and you can see like all the local clusters, Rancher desktop or Docker desktop should work and even the cloud providers, you know, C, O, GK, EKS, they all have 1.23 onwards. So you should be able to use the same demos that we are doing with Qubectl debug with those clusters as well. Okay, so let's say you want to use Qubectl debug to debug your minimal container image application. So what do you do? You have a few options, you can create your own debugging image which is kind of this option you can use a couple of existing debugging images. The most popular one is Natshoot. It's a well known system and network level debugging image, lots of great tools there. Then there's also a set of debugging images by LightRun, they're called CoolKits. They support several runtimes and will use the Node runtime. And in the Do-it-yourself bucket, one of the nice things that you can use in Nixor, you can build a debugging image on the fly by specifying the packages you want and we'll see it as well. So we'll have three demos we'll look at how to debug a Node application, kind of an application level demo, and then a system debugging demo with Strace, and then a demo that shows how to make the debugging experience look similar to what you used to. Let's switch. So first we'll in the demo application, the Makefile has a nice menu of different options that you want to use. So we'll start with creating the application. And it's this application it has a deployment with OnePod and the demo app image and a service. So the application is running and we're using the Slim image. So first we'll try to debug the Node application and let's try to see that's all it does. It's a Hello World app and this is the app itself. It has a couple of endpoints and we'll try to debug one of the endpoints. So what can we do? So I'm going to use a debug, a Node cool kit image to debug my application. So now I'm connected to the debugging site car and I see the application and again and this is us and this is our file system and this is the file system of the target application and this is the app. So I'll want to use a debugger that I have in my cool kit image and with Node there's a way to force the application to run in debug mode and you do that by sending a user one signal to it. So now if I switch to the other okay. Now I see that the Node runtime switch to the debug mode and it's listening. Now I'm connected to the target application. I'm going to pause it. I'm going to set a break point. It's going to take a while. There's a delay. Okay. I'm going to set it on line 13. That's inside of the main handler. I'm going to continue. I'll want to curl again. Now we hit the break point. So that's an example of how you can use a debug image to do application level debugging.  I'm going to use strace to do a lower level debugging session. So now strace is connected to the Node application I have. Now we've got a whole bunch of stuff there including a response right there. We've got a response written from the application and then I'll try to do the same thing with a couple of other images. I'll try next. Now here, if you look at this funny looking image name, you'll see that it has a lot of different names in it and it's the names of the packages I installed. For example, LSOF, Natcat, TCP dump and strace. It takes a little bit to pull it because it gets created on the fly. We have the Node application too and now we get the same result. And the next one is going to show a bit of shell level trickery to make the developer experience a little more straightforward. So I'm going to use a busy box debugging image. Again, we have the same file system problem. So what you want, you want to see the file system over the target application image but you want the tools from your debugging image. So for that you need to bring those debugging tools into your target namespace. Well, figuratively speaking. I'm going to copy and paste because I'm really bad with typing. So first I'm going to link the debugging tools from my debugging container. Then I'll add them to the path and then I'll change the root file system. So now if we do a less what we see is the file system of the target image container with the debugging tools. So now if I can run any of those tools there and I can go to our source. So it looks the same. And I can do everything I would do as if I was exacting into the fat version of the application image. So you get the same kind of experience there. Yeah, so these are also available on GitHub as Kyle mentioned. So you can try on your own as well. We got just one minute left. So the key takeaways is minimal container images on case are ready for mainstream because of fmware containers. Now you have the tooling which exists. So you have seen some of the cool demos using the tools like nixie, net shoot, cool kits that you can use to debug your slay, destroy, or scratch containers as well. So you can actually now use them for in production. So fmware containers make it possible to debug, scratch, slim, destroy images. Thank you. And this is this is the demo repository. Cubeday Japan demo and you can try fmware containers on Civo and create a minimal container images using Docker slim and then try to do that and shout out to Akira for the nerd CTL. And if you have any, yeah, it's an awesome tool. And if you have any queries, we don't have time, but we'll be hanging around here. So feel free to ask any questions as well. Thank you so much for joining it.


How are you? What's up guys? This is CurzPike. My friends call me Big C. Back in action today. I want to show you how to create free AI images. And not only that, I'm going to show you how you can create them for free in under a quarter of a second. Yes, I'm not lying. You can create images in less time than it takes to speak what you want. Don't believe me. Look at this. Right here it says inference time, 252 milliseconds. That's how long it took. That's a fraction of a second. So I'm going to go ahead and delete this prompt. And again, this is for free. And right now it's unlimited. So just go nuts, try it out. I've got two links for you. There's this one here. And then there's a link for hugging face, which is also quite good, but you can go ahead and add more steps in, but it doesn't quite have the computational horsepower. So it takes a few more seconds. But this one here is in real time, it is wild. So here we go. Like I said, in under a second, under a quarter of a second, so a cinematic, watch what happens. It starts to create it for me as I type it. This is absolutely nuts. The cinematic of a French bulldog. I don't know what do you want him to do? Reading a book on the beach. See how, oops, I got to go ahead and learn to type. On the beach. Okay, come on, catch a eating pizza. Red sky, blue glasses, whatever. I mean, I'm just having some fun here. Oh, it's catching up there. Okay, the glasses, I got the glasses wrong. Let's go with maybe red glasses. All right, red glasses. Okay, and look at this, there you go. It's creating this and it's telling you that it's doing this and how long it takes. Less than, that's almost a fifth of a second. Let's say you get something you like, you can right click on it right here and just save the image as. So I'm just gonna right click, save image as, something, whatever that is. Put it on my desktop. If you don't like it, but you wanna keep the same prompt, just click on this randomized button here beside seed, watch this. It's just gonna give me another crack at it. And another crack. And again, guys, this is completely free. I'm not logged in, nothing. I'm just going ahead and clicking on it. Okay, so let's say this one's pretty good, but I don't like the fact that it's an English Bulldog. It doesn't work for me, or a French Bulldog. Watch this, English. Now it's an English Bulldog, literally in a quarter of a second it's done this. This is awesome, guys, SDXL Lightning. I got links to both of these, both to the websites here, it's free. Give it a crack, let me know how it works for you. Thanks for watching.


Thanks to the organizers for allowing me to share this experience as my last experience as HIV TV advisor. I'm here to share with you our experience implementing systematic HIV testing in a nutritional program in Niger. So, as you might know, there are different projects in Niger currently tackling different situations from conflict in DFA, but mostly malnutrition and malaria, who are the main killers in this area. Sorry. So, about this project, which is Madawa Project, some particularities it is rural area with a Muslim population and the reason why I mentioned this is because this population has been almost non-exposed to any HIV information so for us was quite tricky to decide to start these activities. The main killers, as I mentioned before, where malaria and malnutrition, this is why we decided in 2005 to open a nutritional project including inpatient therapeutic feeding center, ambulatory therapeutic feeding center and hospitalized management for kids under five years old. Just for you to give you an idea of the volume of patients in the ITFC, more than 2,200 kids are admitted per year, which is an average of 400 kids per month and this is a lot. So, just for you to keep in mind for the following slides. And regarding hospitalization, there are more than 1,500 kids admitted per month. Regarding HIV, as you may know, Niger is one of the lowest HIV prevalent countries in the Western Central African region, however, with a very low ART coverage as well. So the other population is estimated to be 0.5, but the ART coverage in the country is below 9%. However, we find out in our blood bank that the rate of HIV prevalence among blood donors was 8% and the rate of 12% of hepatitis B prevalence. This plus some publications where there are not many reported articles about it, but there were two of them that really pushed us to move forward on this regard. One from Niger that find out an HIV prevalence among malnourished kids of 9% in the capital. And systematic review of Saharan African countries where they find out HIV prevalence between 8% to 30% among malnourished kids. This is a reason why in 2014 from the AIDS Working Group and the Pediatric Working Group, we recommend to start systematic HIV testing in nutritional centers. However, we know that pediatric access to HIV is a challenge even in those high settings where we have the beautiful HIV projects. So we thought that in these particular settings where we are only treating malnutrition and malaria, it would be even more difficult. Particularly in Niger, there was lack of access to early infant diagnosis, trained staff in existence, also due to the low HIV prevalence, people has never been exposed to clinical management of HIV. There is poor or almost inexistent supply of ARBs, pediatric formulation. And it was a difficult programmatic decision to tackle only children. It was a difficult programmatic decision to include HIV. There was quite some reluctance from the team due to the stigma, to the unknown situation of this community, the perception, and only the question of what do we do with the rest of the family? What do we do with mothers and fathers? Fortunately, there is an MOH program in the same hospital. Although quality is questionable, at least there is a place where to refer the family. And there was a strong fear of jeopardizing the nutritional activities. I think the stigma and the unknown situation of this community was the main barriers to start this activity and the team fear that by starting HIV testing, mothers will not bring their sick kids to the hospital. So the objective of this study is to describe the implementation model and also to assess the HIV prevalence among this particular population. So it has been a polypoly slow process to do it. The first thing we did was to send what we call a flying HIV TV doctor who was a person that will be only focused on HIV TV, mostly training MSF and MOH staff, training community health workers, training health promoters, and starting talking in the community with community leaders and with the religious leader about their perception of MSF starting HIV activities. Slowly the team felt much more confident and in 2014 they started targeting HIV testing. So they were only doing HIV testing in those kids who were suspicious of having HIV positive. So they were not doing systematic HIV testing. The reason why is because they wanted to gain confidence. They wanted to be sure that mothers and kids will not be stigmatizing the community. By December 2014, among the children tested, which were more than 100, only 3% of the total admissions, 7% were found out to be HIV positive. So by June 2015, the team felt confident enough and they launched the systematic HIV testing. And when I say launching systematic, I'm talking again about testing more than 400 kids per month in an intensive, in patient therapeutic feeding center where kids are very, very sick. So by December 2016, more than 7,400 children had been tested. And I add here this information about 2017 starting reproductive health activities because what we find out is that most of the mothers who are coming with the kids to the nutritional project, they've never been tested for HIV. So during the period of implementation of systematic HIV testing, 98% of mothers or kids accepted to be tested, the kids be tested, and due to the difficult access to the NAPCR, all these kids were tested with rapid diagnostic tests. Among the more than 7,400 kids tested, 0.7 were found out to be HIV positive. Among those, only 25 were older than 18 months, so the diagnosis was confirmed. For the rest, samples were sent to NMA for the NAPCR and 13 were found out to be HIV positive. So in total, during all this period and with too much effort, 38 kids were found out to be positive, which is a prevalence of 0.5. 84% start ARTs during admission, still until now, 88% are under care, three kids die before ART initiation, and there has been only one kid lost a follow-up. Fortunately, unfortunately, I'm sorry for my English, there's no date on biological situation because there is no access to viral load in Niger. So about the challenges, as I mentioned before, I think the poor awareness about HIV in this population was one of the main barriers or fears, the lack of trained staff, the lack of access to early infant diagnosis in this population. Nutritional malaria being the main priorities was a barrier in the sense that the team did not feel comfortable starting HIV activities. Monitoring and evaluation tools were not available, so the data for HIV was paper-based data, basically data reported in the Ministry of Health books. About limitations, as I mentioned before, this is a retrospective review, so there is limited data, we would like to have more data on these kids. HIV kids, there is no segregation per age, either they were older or about 18 months old, and there is no date on biological outcomes. As a conclusion, we can say that implementing HIV testing in nutritional settings is feasible, with some little extra support, which is highly recommended. However, we find out a lower HIV prevalence than what has been reported in the same region, and this is why we think that in settings with low HIV prevalence, it might be worth to start by offering HIV tests to the mothers and only to test those kids whose mothers are positive, and this has been a matter of discussion for a long time, even among ourselves in the AIDS working group as well. When it is a new activity, we strongly recommend that you reassess HIV situation in a short time to redefine your strategy, and it's urgent to advocate to have early infant diagnosis at the point of care in these settings. It's totally unacceptable to wait three, four months to have the results for these babies. So what's next? We are going to review the data from all our nutritional projects provided in HIV care right now. In particular, in Niger, we are going to change the strategy, and we are going to offer HIV tests first to the mothers, and we are going to test those kids whose mothers are positive. We would like to share this experience and more similar projects for a wider rollout. We have still quite a number of nutritional projects without access to HIV care, and we would like to simplify the package of HIV care, pediatric HIV care for these settings. One thing that I didn't mention in the slides is that all this fear and concern about stigma and jeopardizing the nutritional activities were a myth, but this took us some time, it took us some time for this flying HIV TV doctor to sit with the mothers, to sit with the community leaders, to sit with the religious leader, and be listened by themselves that they really wanted to have this service in the area. So finally, what we thought would be a blockage has been quite encouraging for the team to move forward with this activity. So I would like to thank mostly all the staff and the patients from Madawa Hospital, and the AIDS Working Group and the Pediatrics Working Group for really pushing very hard this and the agenda and keep fighting HIV. Thank you very much. Excellent work in one of the most challenging and poorest parts of the world. Questions for clarification or technical questions? Right. So we have one, two, three. Yes. So we'll start with the person at the back there, and then we'll move to the person in the middle, and then we'll go to that end of the room. If you can speak clearly into the microphone, please, and introduce yourself. Estrella Lasry, MSF. Thank you for an excellent presentation and for excellent work. Do you have an idea of what the, if the reaction to testing for HIV is going to be different when you start testing the mothers, and what are you going to do if the mother refuses to be tested? Is there a plan to still test the child? Okay. Can I go question by question? Okay, sure. Because I don't have that memory. That's allowed. Go ahead. Let's wait. So the reason why we will offer HIV testing to the mothers first is to avoid testing 400 kids to find out very low number of children. So that's the main reason regarding the second question. If mothers refuse to be tested, there is nothing we can do apart from counseling and explaining again the situation. Right. Thank you. Should we move on to our next question? Hi. Can you hear me? Yes. Sorry. I'm Hannah, an ID doctor from North Manchester Hospital. I was just wondering about the follow up for these patients. So I worked in a similar setting in Sierra Leone, and at that time we weren't offering HIV treatment because the program with the MOH was so poor. I just wondered, has MSF made a long term commitment to following up these patients or was an agreement made with the MOH? And if so, what challenges did you face in sort of making that agreement? That's a great question. It's a great question because it was an initial discussion we have with within ourselves internal discussions. I was pushing forward for a longer follow up of these patients, and there were some reluctances, again, the fear of finishing with a huge cohort of HIV patients. Finally, the realities that we are doing the follow up of these kids, together with their mothers. We are lucky because in the same compound there is the MOH HIV clinic. With a questionable quality of care, but still we managed to do what we call family approach, so mother and child are being seen by MSF in a pediatric consultation with an MOH staff for on the job training. Thank you for that. There was one more question at this end of the room. Hi, I'm Madhu from MSF. What explanations or ideas do you have to explain the differences in HIV prevalences that you found? So your data and malnutrition in your malnourished children corresponded with the national data on HIV prevalence, but contradicted your own data and your own blood donors and other publications. So any ideas about that? Yeah. The first assumption that we have, this is why we want to review again all this data and we want to see what is going on in the following years. Regarding the HIV prevalence in the blood donors, you have to keep in mind that MSF protocol says that is the first we do only one test in the in the back. If the bug is positive, this is disposed. We've changed in either that by sending the blood donors who are positive in one test to be to the voluntary counseling and testing. This is the real prevalence that we have in the blood bank with one test, which means that if we go further, this prevalence might be lower than that. The second reason is that in the in the same study done by Saltis in EMA in the capital, they they they themselves they explain that there is difference in prevalence between urban and rural areas. And this is a really rural area with a particular population. So we believe that in fact, this prevalence might be lower. We will have more data once the antenatal care clinic and maternity activities start because HIV testing is going to be offered to the mothers. But we believe that this is the prevalence. And this is why we want to review all the data from all projects. Thank you. So we're doing so well with time that I'm going to ask one question. You alluded to the MOH clinic being next door to you. And you're changing your program to testing the mothers first. What is the sort of coverage within antenatal care if any exists? So before the babies are born in terms of HIV testing, it's below 9%. Right. Okay. So it seems to me that there's a big reason why we want to get involved into SRA activities. Right. Okay.