Well, hello and welcome and it's time for another Dev Nation Live. I'm super excited today to have Etzin Yanaga with us. Say hello, Etzin So Etzin Yanaga is with us today. He's specifically going to be drilling down on zero downtime database migrations So for the world we're living in now with cloud native computing and people working in containers and microservices We tend to forget about that little old monolithic database that we all have and I got to tell you The presentations I deliver I tell people the database will the date database administrator will hunt you down and kill you Right, that's a common problem but Etzin's been traveling the globe for the last several months and studying this bigger space around data and Data within the context of this new world of DevOps new world of microservices and cloud native computing And he has a lot of great tips and tricks and techniques to talk to us about and Etzin I know you also wrote a book on this topic also, so I imagine you'll talk about that as you get to your session So at this point, Etzin take it away Okay. Thank you very much Burr Well, oh, it's a great pleasure for me to be here talking to you at that national live I hope you can enjoy the content I'm presenting today and let me share my screen I Hope that you all can see now my presentation And as Burr said today, we're going to talk about Zero don't tie migrations and particularly how can you update your database schema with zero don't tie migrations? Which means that you'll be able to like roll out new versions of your applications without disrupting your users into production My name is Ed Sonja Naga. I'm a director of developer experience at Red Hat Just in case you want to follow me. My Twitter handle is at Yanaga And I talk a lot about DevOps microservices Particularly like data in a microservices world Java and software craftsmanship and other other subjects to I I'm also a Java champion and a Microsoft MVP and the book that Burr was talking about is this one Migrating to microservice databases in February this year. I was able to release this book from Riley and just in case you want to electronic version of the book Red Hat developers has bought the royalties of the book. So it's available for free. If you just go to this URL You'll be able to to download electronic version Or if you don't want to type every all of this you can just go to my Twitter profile at Yanaga The peanut treat has a link to the book. So if you think it's easier, you can go there and check the URL too so All of the things that I'm going to discuss today They're available in in chapter three of the book. So I'll be talking about zero downtime migrations But today I'm going to show you something that I've It's the first time I'm presenting to to an audience How can how can you do that in prets? So today we're going to demo how these zero downtime migrations work in prets so just keep tuned so be able to check that and If you want to start this discussion about zero downtime So I have to tell you that the single requirement for Sing the for for zero downtime Deployments in your production environment is for you to have a blue green environment and when you have a blue green environment to perform blue green deployments It means that you're going to have like two equal production environments and you you will be Very likely having two different versions of our application running at the same time in production. So we're talking about The code it might sound like easy Because this kind of blue green deployment scenario is the things that we've been doing for the past five or six years Today we might have like much more advanced deployment scenarios If you're using platforms like kubernetes and open shift you'll be doing you'll be doing like canary deployments or rolling upgrades But it's sticking to the simplest blue green deployments This is the the the single requirement for you to be able to achieve zero downtime But if you think about that having two different versions of your application running into production Regarding to code you might think we'll be thinking it's not that hard But if you think about state state is definitely much harder. That's why I like to say that state We've stated with code is easy, but the state is definitely much harder And when we're talking about states you can have like two different kinds of states You can have a femur states Which is kind of state that you usually store in http sessions and you also have a persistent state Which is kind of state that you usually store in a relational database system and We are going to do specifically with relational databases on the stock so because After traveling and talking a lot with different teams and people worldwide Of how are the chat how were was the challenges that they had trying to deal with with data In a microservice world number question that always receives is this How do I give my relational database and since we're talking about zero downtime migrations? I'll have to tell you that you're not allowed to do like migrations manually anymore And to give you a technical term Migrations is the term that we use in database words To say that is the set of cco statements and possibly code statements that you need to apply To transition your database schema from one version to the other. Okay, this is the the concept of database migration So if we're used to like emailing your cco updates script To your dba through the night or if you're used to attaching that to a geo issue You're not allowed to be doing that anymore You have to automate everything And just to show you two tools these both flyway And liquid base. These are the most popular database migration automation tools in the java space Not just because they're in java But I just learned that they're they're very popular in other for other kinds of developers too except from the guys from the ruby on rails world because ruby on rails has their own beauty migration to What if you talk to php developers or Python developers? Or even dotnet developers you have many people using flyway and liquid base on the on their deployment pipelines to update their database my ischemas automatically as they They release new versions of software into production So how can we use these tools to achieve what we call zero downtime migrations? So if you want to achieve zero downtime migrations, I have to tell you that If you have two different versions Of your application running at the same time into production, they must be back and forward compatible Which means that your database schema must be must be compatible with the next version and the previous version of your your application And it also means that we're not allowed to have migrations with like a hundred seco statements at a given moment You need to break that into big stats, which means the smallest possible batch size So for many of if not all of the cases you're not allowed to issue like two or three or a hundred or a thousand secret statements into a migration Each one of your migrations will have a single secret statements It also means that If you're using like like an update statement to your database and these update statements take like five minutes to run You're not allowed to have a five minutes lock time in your database because it means downtime into production now. We'll have to deal with We've we've this long locks by sharding our database updates So if you're updating like a billion rows at a time, maybe you're not allowed to do to do that Into a single secret statement Maybe you need to shard your statements to to update your database like a thousand rows at a time Or a million rows at a time There's no specific amount that you can do for that You need to rehearse your migrations in a test environment. For example for just for you to access How much time does it take to update your your database then you'll be able to measure? Well, maybe one second long time Uh downtime is something I can bear into production. Then you just stick to the size. Okay So sharding is a very fancy word in the in the data world just to tell to you that well You need to update your updates. You need to to split your update statements So if you're used to do something like this Outer table customers rename column wrong to other thing Then you're not allowed to do that anymore. Maybe you have to split your secret statements into something like this Maybe first you have to add a column. Then then you have to To update your your database using shards. Then maybe after some time you you'll be able to delete the column so After talking as I said to many many different teams world wine I was able to collect some of the scenarios that the these teams are using successfully into production To update their database schemas with zero downtime So the four main scenarios that I was able to collect were these ones Add a column rename a column change the type and format of a column and delete in the column And to be very honest with you and most of the teams that I've talked to that were able to do that in production Were from startups or very small teams very engaged developers Because they were able to deploy like multiple versions of their applications in a week or even in a day But in the past year I was very happy to see that after publishing my book. I was able very happy to see that some enterprise teams We're also trying to engage with some of these practices and I've received some some some very Very good news from some of these teams that were they were doing these zero downtime migrations into production So I hope that after seeing this session you will be able to apply some of these techniques into your production environment to achieve this zero downtime migrations And I'll try to explain Some of the scenarios very quickly because we have a demo for one of them Today so the first scenario that we have we'll be able to see Is add a column if you want to add a column into production with zero downtime You can see we have like four different steps. We have step one two three and four Each one of these steps is going to be a new release of your software into production So it's going to be a different version of your your application into production So in the first version, you're going to alter table at column the second version Your code computes a read value and writes to the new column third version you update the data using shards And fourth version It might be the last one code reads and writes from the new column And I have to tell you of course between each one of those releases into production You have to open your database cli and issue some cico Select statements to check if your data is being written in the wrong In the right column in the appropriate format and something else So this this is the simplest possible scenario and again all of this information is described in all my books I'll be able to get like much more details if you read the chapter of my book and Next scenario, which is the most complicated one renaming a column with zero downtime Yeah, you can see like we have at least three defense six different steps for you to be able to to achieve zero downtime migrations First version alter table at column second version your code reads from the old column and writes to both Third version you copy the data using small shards Fourth version your code reads from the new column and writes to both Fifth version code reads and writes from the new column and six step Sixth version you delete the old column much later not right now because You know deleted a column is a destructive statement You lose the data into production and some of the secret ingredient from all of these zero downtime migrations Is that is that I never issue a destructive statement into production your Your database schema is always back and forward compatible because the previous version of the of the data is always there That's why you have much more confidence when releasing the software version into production in fact These migrations are not the only way for you to be achieving zero downtime migration to production They are also the safest way for you to be Updating your database schema into production because you never lose any data And this is the scenario that I want to show you today in prets Because I feel many people after reading these steps and thinking about that I don't know if it works into production because you know I'm suspicious about that And I wanted to show you a very nice demo. How can you achieve that because I'm not the only person saying that it works because as I said before I've collected the spreads from many different teams worldwide and they are using these steps They're using these scenarios Successful into production. So let's see how can we achieve that into production too? so now I'm going to show here my project. I created a very simple wild fly swarm project Using some domain driven design techniques. So which will probably the subject Of an another session that we have later. So here I just created a customer class And that's a jp. That's an ejb jpa jsf application, which is basically a crud For a customer entity and this customer entity It has some fields. It has an id. It has a name and has a surname Okay, you might be thinking well, why the hell did he put like this name and surname class name surname? Well, in the end I'm storing this information as varshar columns in my database But I'm using custom types in the domain driven design world. We call this value objects. So We you also have to code something to be able to to to store and retrieve These things from your database if you're using value objects, but again, this is the subject for another day Just bear having in mind that this is our Playing Java classes and they are being just think of them as the strings because I'm storing them to To my relational database as varshar columns So what do I need to do? What do I want to do? I want to rename this field surname Maybe somebody asked me to rename that to last name So I want to show you that this application is running successfully today right now And I have my application here I want to run that with my wildfly swarm plugin So while I'm running my maven build. So it's a wildfly swarm swarm plugin.  It will be a finger version of my jar and it's already running my application Okay, and since I told you that you need to automate everything your application. Yes wildfly swarm already has a plugin beauty for for flyway so flyway Is a database migration automation 2 and if I just create a special sql files in this Um a folder db.migration and I use this special naming for my sql statements. It's automatically will create an and An execute these statements, uh as soon as I'm my application boots up So that's what happened here. I had an empty my sql database in my machine So they just created say boat and now I have inserted two different customers myself and work so if I open here my Application which is already running you can see that I have had this this data already populated And if I go back here to my terminal I can see yes, the data is there if I want to show you the tables Basically flyway created the customer table for me and also created this table schema version Which is uh the table that flyway uses to keep track of which migrations were successfully applied already to my database schema So if I get here, select my schema version You can see that I have like my initial uh Migration successfully applied on this sql from this sql file. Okay, so I have the data First version is running. So if I want to check its application if I want to check here It's an aga. I'll put the number one here. You can see what's successfully updated if I want to create another um Customer here. I'll put the names of the other people from our team. So I added a reference working successfully. This is the first version of My application. So if I want to go back to the heat to talk to To the steps first there. I need to add a column that I want to to rename So we're in a column. I want to rename for a surname to less than first the step that I have to do is to add a column So what do I do? I'm going to get here. I'm going to keep this application running. So keep you have keep in mind that's my My qn version is running and I'm going to update the source code There so I'm going to alter table and column and you know, I wanted to be very cautious about that So all everything that I'm doing right now Already committed this this application to git and zabela mongit hub. I'll share the link later Or maybe we're already shared the link on the chat so Here I'm going to I tagged some of the versions. So I had version one. So I'm going to check out version two And let's see what changed here in my application code. You can see If I come back here to my customer class, I didn't have a change anything yet But I just added a second migration to my application, which is at less than column So first I'm going to modify the surname column because currently has a not no constraint I'm going to to remove the not no constraints and I'm going to add this second Column called last name to my database Schema. So that's the only change that I've performed here so far And let's run this application into production. Let's see Now I want to do the same I swam run So now we still have like fly. I'm missing an F and that Swarm http port. I want to change the port because I want both versions running into production at the same time So I have version one running here and now let's run version two Which will run on port 8081 So let's just wait for It to finish the deployment If you want to know usually warm flies form like starts much faster But since I have a lot of things running in my machine and I also have this hangout running on the background It takes a lot of cpu So if I get this new version of my application You can see they have two different versions of my application running and they are holding the same data If I want to change that I change the key here to version two And it's working successfully so on version two Yes, I just added a column no big deal. If I add a new thing I add another person in my team I'm going to add don You see it's working properly. Both versions are working properly into production. Okay, so I just did a blue green deployment I rolled everything to my new version So Maybe now that I've seen that everything is running. Okay, I can stop The first version of my application because I only need now the second version of my application to be running successful into production and second step Now I've added a column ready. Now I need to my code reads from the old column and writes to both. How do I do that? So I want to get here in my code So I'm going to open another terminal. I'm going to Out the third version So if I go back here to my id if I go to customer class What did I do? I just created The surname few because I'm not messing with the surname yet. I'm just adding a new field called last name I'm using this jpa tags called prep persistent prep day to be to To make sure that my data my last name column is going to receive the surname column whenever I save or whenever I merge my my objects And here in the center too I'm getting like this dot surname and this that last name because I don't want to be correct only on the database layer I want to be correct here on the code layer too. That's why I've just changed the center to set the last name And here I'm changing the the prep resists and prep dates Notations to be updated in my last name column. So this is the second step Let's run my application to productions to check if it works so odd fly swarm run Swarm htp port and that's 8081. This one is going to run on 8082 So I already have a column in my database now. I'm going to release a new version of my application Into production. I have version On port 8081 and I have another version now on port 8082 Okay, now it's running. So you can see here. I'll check this This version is still running successfully. Now, let's go to 8082 You can see this application is running at least at least it's reading from the database But let's see how it works when I try to update that I'll try to update myself And you can see data successfully updated if I check that here. It's running too Let's see what what's happening inside my database So if I select everything from customer You can see that whenever I update or whenever I create a new New one in my application. It's going to write both surname and last name. So update it at some So now I have the the data value in both surname and last name columns. Let's try to create a new A new customer here I'll add kamesh So you can see I'll add a kamesh on my new version It works successfully if I get back here to my old versions working properly, too If I go to my database, you can see that all the updated records and all the new records. They are already right into both Columns, but the old columns the old rows They still have a no value in the last name because nobody's updating that so that should be our next step Everything is running properly So well, I just rolled out a new version of my application. Now I can stop the previous version from running I'll stop this version from application and I'm at version three. Let's allow check out version four Um, now it seems that I'm in version four. Let's see what changes here I'm going to update my database using shards since I only have like five rows It's not big enough for me to be charting But just have in mind if I have the billion rows, I will be able to have to split everything That I'll be doing right now. So in the this version of my my application I'm going to update the customer set last name equals to surname. I'm going to roll this migration into production. Okay, so Let's run wildfly swarm And now I'm going to run Orcs 8083 Let's wait for that And you know, they always takes long as we're looking that at and doing something for a youtube live transmission Okay, now it's ready. Let's go to our application. So it is not working more and now let's go to localhost 883 My application is running successfully. Let's do previous one previous one is curing Successfully two and let's see the data on my database. If I got my my sequel terminal You can see that all of the columns are populated Right now if I decide to change here Before you can see that my new version is running Is updating successfully everything if I want to add a new Let's add a new guy here at pole, which is watching us If I had pole it adds successfully on both versions and if I go to my database both of my versions Are reading and writing the same thing from both columns. So that's the version that I wanted to show to you So now let's get back here I did a success for deployments. I can stop the previous version and let's go On version five, let's go to the next version If I did v5, I'll go back to my presentation here and just copy the data using small charts Now I can change my code to read from the new column But keep writing to both because I need my previous version to be working successfully So that's what I did right now if I go here to my customer class you can see That's now Instead of being updating the last name with the surname value. I'm doing the opposite I'm updating the surname which is the old column with the column from last name Which is the current property that I'll be using in my code and also here In my setter I'm setting the uh, just change it here because I'm setting both variables at the same time You can see later. I'll be renaming this get on the setters to surname too. So it will be a successful migration So so now updating surname I'm just using last name into production because you can see here get surname is returning the value from last name column And surname is not being used anymore, but it's getting updated on the database So let's deploy this into production and check if it's going to run this one It's going to run on port 884. It's ready. So now let's open So 884 New version is running previous version is running too. So let's try to Create a new customer. It's going to be jessica And sorry jessica. I don't know how to have your name. So it's going to be jessica crow And if I get let me get back here to the previous version is running I can update here my data too in my previous version Go here to my new version. It's working successfully so Let's select everything from customer and you can see My application is working and writing the correct values to both columns At the same time and just in case you will do you're thinking well am I not cheating with wild flies form or something like that So let's stop the previous version for running. I just have the current version running and I'm gonna get the current version running here and I'll get Version running version five. So I'll copy my artifact, which is zero time immigration swarm dot jar to another folder And I'll call that v5 dot jar and I'll run this Current version With java dash jar. You can see wild fly swarming bootstrap. Let's successfully again There's a question came. It's been really I'm sorry I threw a question at you while we wait for that start and For sequel update command and a separate version file because of clarity Or is there some real use case behind it into a single sequel file? Now which one the update statement Yeah in version four you had the sequel update command and a separate file Yes, is there any is there a reason for that? Yes, it is because first I was I want to use the fly way for updating my database And if I use this is as a fly update statement I'm committing this migration into my source code and I have to release a new version into production So this way this this new version will be compatible with the previous one And it doesn't happen all the time if you just decide to use this update thing using your database cli Okay, so I want to keep track of all the changes and it's a very good prep If you do that everything all of your database migrations fool you through your source code repository That's why I'm issuing this thing inside my my Flyway folder here into my project Okay, so it's a requirement. You need to release a new version of your application production for you to be able to Next version to be compatible with the previous one. You can't just update that using your cli Okay, so have a new version. So let's get to the last release which is My code reads and writes from the new column, which means everything is working perfectly fine. So let's release the last version Let's get right ahead zero down time As version v6 If I go here to my project you'll be able to see that Version six. I don't have the surname field anymore I renamed my garrison status to last name. I don't have any other reference for that too, I don't have any other new migration happening here too Uh resources Okay, so that's Dave steps really so now now let's just run that into production Remember 85. Oh this one. I didn't uh give the parameters or it's running on 8080 So previous version running on 8080 and latest version running on 8085 Let's wait for you to deploy We're almost done with the demo And it's ready. So if I go here Give us version That's my application. I'll get to my previous version on htp for 84 Let's go to the latest version Just 85 You can see my latest version is working properly Here up already is ready. So if I go here to 84 Previous version running successfully too, but you can see if you go to my previous version Can I create another person from the team? I'll put andre If I have select assist uh from customer I get it's writing for both But if I get to my new customer and I get to update and create a new customer here I'm going to add Sarah It's both are running Successfully into production. You can see everything. It's fine. But if you go to latest version It's not writing to the previous version to the previous column any more certainly because it's not needed but both versions previous one a new version are working properly into production and I was trying to use Everything is working successfully into production now Now that my code's breathing rats from the new column only I can much later delete the old column But I never delete now because you know, maybe I need to roll back my application to the previous version So i'm just going to mark my version for deletion because maybe the dba on the maintenance window He will he or she will be able to delete the column because after performing a backup. Okay, so That's the most complicated scenario Renaming a column and I hope I have shown you to you that I've done that successfully running different versions of my application into production and I have other scenario here change type and format of a column and if you can see a pattern Change type and format of a column and rename a column. These are exactly the same step So everything that I did right now you can do that if you want to change the type of format for And less scenario delete a column. No, you never delete a column into production Because that's a destructive statement you lose the data So you just stop using the read value but keep writing to the column because you know, your previous version might be needing that In a third release you maybe if everything went well You might be able to stop writing to the column and again if everything went well again Maybe you can delete the column later. You can just mark the column for deletion and your dba will be able to do that Okay And another question that frequently arises that what about my referential integrity constraints and not no constraints too There's no easy answer for that. You have to drop everything and recreate These constraints when the migrations are done So just as I removed the not no constraint before if I wanted to have another step to my migrations I would just add like a modify column and add the not no constraint back again to my application And if you want to check what I did Step by step in all of my source codes I tagged each one of these versions of my applications on git and I've uploaded this The source code on this git repository a github.com yanagas zeroed on time migrations So we'll be able to check everything that I did right now And That's what I wanted to show you today. I hope it has been like Very like productive 39 minutes of your life And I'm ready to answer any questions that you might have for me Well edson, we're totally out of time as far as the session goes But I do want to leave you with one quote a fantastic quote that we got from reverto It's in fact the main challenge when moving to microservices Which answers the question of what to do with this monolithic database? How can this be done etc read the entire book it is excellent So reverto obviously loves your book and again I also highly recommend it and people can grab it from the link that we provided in the chat And of course we can all tweet it out as we can But I thank everyone so much for your time today We try to keep these relatively short and sweet. That's an awesome job on the demo I really love it and then this recording will be available to the masses at large on youtube Almost as soon as we hang up the call today. Thank you Thank you. See you in our session


Welcome to the private property home shopper show. My name is Chad Rivera and here we are walking through the beautiful gardens of Isabella State on offer by Demeter Developments and their fantastic sales team. Today we're gonna be showing you a two-bedroom, two-bathroom offering within the Supportment Complex, but that is not the only thing they have. They've got a one-bedroom bachelor pad going for 900,000 rent, a two-bedroom, one-bathroom, a two-bedroom, two-bathroom. Like I said, that's what we're gonna be showing you and that's on offer for 1.3 million rent. And then to cap it all off, they've got a three-bedroom, two-bathroom apartment as well. So guys, if you're looking for luxury living within the Four Ways area, if you're looking to live life at your finest, this is exactly where you need to be. But every single one of these apartments also come with basement parking. So we're gonna be showing you the elevator access. We're gonna be showing you the clubhouse. We're gonna be showing you the pool areas and we're gonna show you just how beautiful living in an apartment complex can be. So you guys need to come with me because it's time, baby. Every single block within this complex comes with very own lifts. There's gonna be no carrying groceries up and down stairs. Anyone who's ever done that, I feel your pain of being there and you'll be very happy to know that's a thing of the past within the Isabella State. So here we are within the two-bedroom, two-bathroom apartment. Immediately as you come to the door, you find yourself in the kitchen. And this kitchen, my favorite aspect of this kitchen as someone who's come from living in an apartment is how much time and effort they've put into giving you plenty of cupboard space without taking up space in the kitchen. You never feel cramped here. So if you wanna whip up a meal, you're gonna love it. Let's talk about some of my favorite features of this kitchen that it has a double sink. This is so rare to find. So many of these apartments just have a single sink. Yeah, you have a double sink with a drying area just off to its left. The gray quartz countertop and now this countertop extends into your splash back above your Bosch touchscreen burners. Yeah, you've also got your Bosch extractor as well as the Bosch oven. Again, these cupboards are beautiful. You've got your white lacquer on them, super upmarket, super upclass. And that's what we wanna do in this. We wanna show you that luxury living doesn't have to be exclusive only to overpriced properties. This here is very affordable, a beautiful choice for an investor, a fantastic choice for a first-time home buyer and this is luxury living. This is luxury living within an area that you've got easy access to every single thing you'd need from your grocery shopping, four smallies around the corner, all the schools around here, easy, super easy access. And here we are. So let's go take a look at the living room area. Here we are. You'll see again just how much space has been utilized in this apartment and I love just how big it feels. There's a two bedroom, two bathroom within a block of glass and it feels spacious. It feels luxurious and I love it. You've got your sliding door here that gives you an open air feel because this sliding door almost turns your living room into an oversized balcony. I love how they've incorporated that. You've got some inset windows over there giving you some light and some air and you can sit down here and immediately imagine enjoying a football game with some friends, having some drinks, living your best life and that's what private properties are about and that is living your best life. I've spoken so much about space so let me show you what I mean. How many apartment complexes have you ever been in that easily fit a four-seater dining room table without making anything feel cramped? Look how much space there still is around this four-seater dining room table and I love these features because this is what turns a house into a home. But it's a two bedroom, two bathroom. Let's show you the first bedroom. Let's go. And here we are within the first of the two bedrooms and those white lacquered cabinets now transform into white lacquered cupboards. The design language continues throughout the house so these are on rollers, easy open, silent clothes, lots of space here whether you want to use this as a bedroom, a spare bedroom or even an office space. You've got a wall-to-wall window leading in a lot of light so this never feels cramped, it never feels musty, nice and open, nice and airy but there is more because just outside of this bedroom you've got your first bathroom. Let's take a look at it. So coming into the first bathroom you'll see space again is evident throughout. You've got a free standing bath and all the tap-fitting throughout the apartment are huntscrow. Then you've got your LED lighting within the bathroom, all of the bedrooms as well as the living room. You've got your single vanity with plenty of undercount to storage and the toilet has got one of my favorite features in all toilets. I love this because it makes me feel so fancy. You cannot slam your toilet lid. Nobody's going to be waking you up in the middle of the night with a slamming toilet lid because those toilet lids are soft clothes. Coming out of the guest bathroom you find yourself within the master suite. The creme de la creme and this master suite comes with an on-suite bathroom that I'm sure you can see to my left and looks very cool and it is. So stick around for that.  This again on rollers, easy to open and just as easy to close. And now let's talk about the size of this master suite. I think so many people are worried when they do buy an apartment that size is an issue and this bedroom fits a queen bed with no problem. You can see just how much space I have around me. And what I like about this as well is your on-suite bathroom. Because of the glass facade blocking it off is it feels even more open, even more spacious and even more airy. But come with me and let's take a look at this. I love how it's separated from the bedroom with this glass.  You've again got your vanity out with plenty of under-counter storage. Your toilet with a soft closed lid as well as an extractor fan just to make sure things in here don't get too steamy. Just before we wrap this beautiful two bedroom, two bathroom up, let's talk about a couple of other very cool features you'll find in the apartment. One is the incredible views you find from all the windows. Number two is that the light switches glow in the dark. So there'll be no fumbling around for light switches if you decide to go find yourself a midnight snack. And the third one is an issue of convenience in that your DSTV and your fiber is piped immediately to the unit you purchase. All you need to do is set up your accounts and you're ready to go. So ladies and gentlemen, I'm standing outside the Clubhouse of Isabella State. This estate is not only about incredible living and beautiful grounds, but there's also comfort. This clubhouse right here houses one of the two pools found in the complex, but there's more. There is also a play area for all your little ones. If they want to scamper about, there's an area in this complex dedicated just to them, but this is where I need to say goodbye to all of you. So if you could find yourself living within the Isabella State, what I'd like you to do is contact Robbie at the Meet the Developments, give him a call, but don't just do that. If you can think of friends or family who would love to live in a location such as this, tag them in the comments below, let them know Isabella is here for them. So for myself, Chad Rivera and everybody else at the Private Property Home Shopper Show, thank you so much and goodbye.


In this video, we will define abstracts and discuss their purpose, identify strategies for writing effective abstracts. Let's start by answering the question, What is an abstract? According to the Writing Center at the University of North Carolina at Chapel Hill, an abstract is a self-contained, short and powerful statement that describes a larger work. Components vary according to discipline. An abstract of a social science or scientific work may contain the scope, purpose, results, and contents of the work. An abstract of a humanities work may contain the thesis, background, and conclusion of the larger work. Abstracts serve several purposes. Abstracts help authors summarize their work. They help reviewers assess the components of the presentation, article, or other larger work the abstract describes. And they help other researchers discover the research and databases and search engines. Students often ask questions about what exactly should be included in an abstract. Thankfully, calls for proposals, or CFPs, will often state the requirements. Make sure to look for those requirements when writing an abstract. Next, we'll cover five tips to writing more effective abstracts. Abstract tip number one, structure. As just mentioned, the structure of an abstract can vary depending on the field you're studying. However, there are common elements that most abstracts contain, even if the wording is a little different. Let's take a look at the components often included. Background. What is my research about? Aim, purpose. Why am I studying this? Method, approach. What techniques or methods were used? Results. What did I find? Conclusion. What are the implications impact of the research? Abstract tip number two, length. Most abstracts are between 100 and 300 words long, give or take. Normally, abstract length will be defined in the call for proposals or in the author instructions online. Make sure to stay within these requirements. Abstract tip number three, synonyms. Include synonyms for words and concepts that appear in the title. For example, if the title of an article uses the term dairy cows, then the abstract should include synonyms such as cow, dairy cattle, bovine. Abstract tip number four, consistency. Mention only the points actually covered in the research. Organize your abstract with the most important information first and try to avoid referencing other works. Abstract tip number five, clarity. As with titles, minimize the use of abbreviations and use common word order combinations. Some of those examples include writing out the word cancer rather than the abbreviation CA, using full scientific names like Escherichia coli rather than E. coli, and using the scientific phrase like juvenile delinquency instead of delinquency among juveniles. After reviewing these five tips and the basics of abstracts, you're probably ready to start drafting your own. As you begin looking for calls for proposals and thinking about the research you might like to conduct, don't let the proposal process hold you back. Keep in mind that when you're writing abstracts, you may not have your results yet since researchers often write proposals before they finish conducting their research. Depending on the type of proposal you're writing, you may only include what you hope or expect to see rather than actual results.


Welcome all to the talk about discuss booting of Raspberry Pi's and more in generic the Raspberry Pi 400. You should see them here. Who's of you, for example, or has Raspberry Pi style at home and works with Raspberry Pi's? A lot of good. I like them. A good company. By the way, I have two of them here. I got two more there and a little bit of luck. We're also going to use them all in the presentation. As you know, these are actually very rare and the big guy actually at the outside of the room that's actually protecting me in my Raspberry Pi. So, might you get any of these? Okay, good. Let's get started a little bit. I will tell you also a little bit about myself. What I normally, actually this ramp on course is actually a little bit normally out of my comfort zone. I'm not really doing a little bit of all the things and that kind of stuff. Actually, this was a really nice project that I picked up actually with my sons a time ago. Okay, let's see. First, the introduction. Yes. What we're going to do is I'm going to tell you where I'm. Then I'm going to tell you a little bit about the Raspberry Pi 400. I do see that a lot of you people actually already know it, so we'll keep that short. Now, actually, okay, what were actually the reasons why we started this project? Because actually the idea was quite nice. These Raspberry Pi's stay are really powerful. These things are actually eight gigabytes for CPUs, ARM based, that kind of stuff. All the world is not Intel only nowadays. Luckily, have other processes like these arms, like the wrist files and that kind of stuff. One of the nice things actually that got introduced in 2020 in November were actually these Raspberry Pi's that were encased actually enclosed in a keyboard. And they're actually really nice actually to use as a, you know, computer for classrooms where you actually don't need to have the big laptops or if you don't want to have desktops, etc. So I'm going to tell you a little about it. Then I'm actually going to show you and tell you actually how we actually did solve the problem, that kind of stuff, the multiple ways actually to do that. There's still some quacks and some open issues. I'm going to try to demo it. The nice part is that you see I did bring a lot of stuff actually here. A little bit of a mini data center, a switch, an odroid, all things ARM by the way. So let's see if we can work again. Like with many demos, this is my only option. I do not have a backup plan, so we're going to make it as good as it gets. Good. And then actually for any questions that are left. Good. Let's first answer the existential question, who am I? Well, I'm Pascal Van Dam. I've been a trainer also for Linux Foundation there since a long time. Started being a trainer actually in 1999 with Uli Peckert. Started actually delivering courses for HPOX. A little bit of the old iron with the property unixes and that kind of stuff. I'm also the architect in this case for ATOS. I'm doing actually in the Netherlands their Linux and their Kubernetes work and that kind of stuff. Next to actually being in Linux for a long, long time. Actually being there since 1991. So even when we had these floppy disk, for example, I was already onboarding it. I did some kernel hacking and that kind of stuff, created a process resource manager and encrypted file system, that kind of stuff. Nowadays, I'm also doing trainings on Kubernetes and also creating, for example, the operators. So the deep technical stuff, that's reason why now this is actually a little bit of a challenge for me, but I really like it. Kernel Internals, Node.js and lately actually developed two courses for Go and Rust. And I'm proud father of four sons and the two elders actually now have joined me in the company. They're actually now at the university. And one of the reasons actually that we started this course is because my elders actually decided during a sabbatical to join me in the company and we had, okay, let's do some really nice and we'd like to do something where we have some common technologies like iSCSI, like NFS, like TF2P boot. And how can we actually integrate that? And for example, in a project to boot Raspberry Pi remotely. Okay, let's see. So Raspberry Pi four and four hundred. And they're actually the four was introduced in 2019, a successor of the successful range of the Raspberry Pi, the two to three and that kind of stuff. What's really nice actually was that the Raspberry Pi four is a lot powerful. Actually Raspberry Pi as we have it now is actually powerful, for example, than laptops that were actually introduced four years ago. The arm a based nice is actually in 2020, we even got a gigabyte model. And that really makes it nice because the next project actually had okay, I really love these Kubernetes actually these Raspberry Pi's in a Kubernetes environment. What would be greater just to have a Kubernetes that I can just switch on, add new notes actually as Kubernetes of as Raspberry Pi modules, etc. So that's also what we did. Now I've got one gigabyte ethernet port in SD card slot, two times USB three and two times USB three. We'll see a little bit later on is that that SD card is actually a little bit of the Achilles heel of these Raspberry Pi's. Now what we have there is actually we need to put in SD card. And even for example, if you're running Kubernetes on this kind of the condition of writing, for example, on the LCD database will completely vary on this card. So that's not really something that's really nice. So we need to need something else to see a lot of people actually nodding. So you know that one. The other one came a little later in November 2020. And that was actually in a Raspberry Pi four, but then enclosed in a keyboard was actually based on an upgrade of the C. In this case, the C zero stepping and allowed also for higher frequency. So instead of the base frequency, we could actually step it over 200. Even if you were lucky, even more, the Torback was a little bit, they only had four gigabytes of RAM. Actually, if you open up such a Raspberry Pi 400, for example, what you see is a major heatsink with a computer on it. And one USB port less, because the keyboard has actually also needs to be routed. And also this one had an SD card slot and actually ideal for running Linux desktop. The only thing what you actually do is you plug in a monitor, you make sure that you put your SD card in it, a mouse, and actually you're completely in business. Okay. Oh, well, also, if you have any questions, let's keep this as interactive as possible. Just raise your hand and I will give you and you can actually ask me and I will try to answer. Good. Okay. So the shortcomings. The first one I'm always running to is these things do not have a real time plug actually built in. And that means that when we switch them off, they actually have a way to store the time in the file system. But for example, if the shutdown doesn't work, go very well. Actually, it means that the RTC, for example, the time in your Raspberry Pi will be somewhere in the past and that kind of stuff. Would be actually really nice if we would have a battery backup RTC there because that would make life a lot easier. For example, with running these things as Kubernetes nodes, for example, that is really mandatory. In the other way, we already told that the SD cards, they actually proved to be very fragile. And the problem is that the SD cards are normally being used for things like cameras and that kind of stuff. So actually, sequentially, right, there's not much reading on that one. If we use them actually in computers, and we have a file system, like XFS or xd4, for example, we have, for example, a database actually hammock on down, you see a lot of wear and tear and actually, yeah, the cheap ones, they actually die within a few weeks. And the longer ones, they die when you do not want it. Good. So the Raspberry Pi foundation on itself also, of course, I found out that the SD cards were not ideal, actually. So they also actually improve something. The nice thing about the Pi 4 is that it has its own eProm, its own boot eProm, actually, that it's actually bootloading that one, they can actually upgrade that one fairly easily. And gradually, the last firmware version that we have now is one actually from the second of September 2022. But gradually, in the past, booting from USB storage was added, actually, two forms of USB had a mass storage and the normal one, booting from network, using TF2P was added, that's actually what we're going to focus on in this talk. Lately, booting from network using a TF2P was added. And the nice thing about this form actually is that you can now, for example, in print as a scribe store your SD card with the OS image, without actually having a, yeah, have to put a for example, in the second computer and write there your SD card. That's actually nice. We're also now working actually on that to see if we can actually boot it directly so that we don't need to have it written first to SD card. We're actually using it, but we're still in in the investigating that one. And the other one, but that's only actually for the CM4. And the CM4 is actually the third version that we have of the Raspberry Pi. They actually allow us to use NVMe devices and to boot from that on the Pi 4 and the Pi 400. Unfortunately, that's not possible. Good. Challenges. So actually, why do we do this? So now, first things that we are delivering courses and doing to COVID, pre just COVID, I would actually allow to travel a lot and go in a lot of time in the island into India, South Africa and that kind of stuff. So actually there, you work with people and also there, the systems actually that we use for classes are mostly on the cloud and that kind of stuff. But classes like the kernel internals, that's nice to do that actually on a public cloud. But yeah, the machines get rebooted a lot. We need to do box stuff. A lot of the clouds actually have their own kernels optimized and that kind of stuff. So actually, this works much better with physical machines. So actually having a Raspberry Pi 4 or phone that actually really nice works with that. We have another class, for example, kernel management, exactly the same that works the best if we have a small machine that we actually can do in the window, actually would stand not any ballooning in the public cloud, etc. And this was a nice one. Yeah, next to being in the IT engineer, I'm also doing stuff actually with children's coaching for children. And I'm also, for example, teaching computer science or ancient Greek for classes of highly gifted children. And actually, to have them work, for example, with logo, actually, these Raspberry Pi 400 are really actually nice to set up. But actually, we'd like to make it very simple. So that the teachers also can do that. So I would like to have a fast setup. Actually, plug it in. Or once it actually plug it in or plugs in, it should actually know, okay, I'm, for example, this Raspberry Pi 400. And we're going to, for example, boot it up with a Ubuntu distribution. Another one, for example, we're going to plug it in. And as well, actually, start up with a Pi OS distribution, for example, and then load it, for example, with a kernel rust implementation, etc. Central management, and important, not relying on the SD cards, or USB storage. USB storage works better. But still, the little things actually also classroom, for example, they get actually, people lose them and that kind of stuff. So we really would like to do it without any storage. So the second project was actually nice. And actually, I offered this also to the keep calm and that kind of stuff. What really would be nice, and actually also to teach this, for example, in Kubernetes classes, you know, in the public cloud, we can actually do something called horizontal note scaling. That means, for example, if I would like to scale up my cluster, add all the extra worker notes, I can actually do that. And what would now be nice, if I could do this actually with Raspberry Pi 4, the nice thing about these guys, actually, not this one, but this actually recipe by inside, we can actually put a PoE plus hat on that.  Not only that, that actually means that we can start them up, power them up over the network, we can power them down. That means actually that I could create, for example, a controller inside of Kubernetes, just a process, which I'm going to say, I would like now to scale up my cluster and it automatically actually starts up, opens up one of these PowerEE ports, for example, this one powers it up, waits actually till the Raspberry Pi automatic consensus boot request and boots with the image actually to directly being able to connect and add another note to it. So actually to deploy Raspberry Pi Kubernetes cluster on demand, scale up and scale down scale up by just not doing more than actually enabling the power on a PoE plus port and scale down actually by disabling the PoE. So of course, first, we need to do that in a nice way. So we're going to shut down the note and actually scale it off and et cetera, and then actually power them off the port. Good. So how does a Raspberry Pi actually boot? Good. One powered on. What actually happens there with the Raspberry Pi is that it starts up. You see these red flesh light flashing. If it flashes too much, then you know that something's wrong with your SD card or with your boot process. The boot EEPROM, sorry, I forgot an E here handles the boot order that actually since 2020, we are actually able to configure that. You can use in this case the so-called RPI EEPROM config. I will show you that a little bit later. And then you can actually change the boot normally. It's actually default on starting from the SD card. And what you actually can do is put a number there. For example, this is F241. And that sounds a little strange, but actually we're going to read it from the sketch right to left from my side. If I'm in the front of your view. One means boot from the SD card. Four means boot from USB card. And one means of the two means boot from the TFTP from the network. Why do I do that? Well, quite simple. This will always help me, for example, to intervene in the boot. For example, I need to flash the ROM outside of the network boot and that kind of stuff. Or I need to do some other stuff. If I actually have first let them connect to the network, as soon as I actually find my TFTP server, it will directly boot from that one. And the other actually means try it an infinitive time.  It will do one, four, two, and again, one, two, four, etc. Always. So that way we can make it a little bit field proof. Good. When the network boot is actually the one that's been selected, then a TFTP request will be done. And actually that's really nice. I will show you that in a few moments. Let's see. I don't have one. I have one half.  The TFTP request will be done. And I will actually look for a directory that's actually named after CPU serial number. You can find that one. I'll show you when I actually have my laptop ready. When you do, for example, a CodProc CPU info, you will see there the serial number. And that serial number, that's actually where the TFTP will do, actually, and check where it needs to file. So with normal PCs, for example, they advertise, for example, their MAC address. And it actually looked there for the boot files. In this case, it will actually look for the serial. You can change that if you would like to, but I really love this one because this is actually means that I can recognize each and every individual pie for that one. So it downloads the config.txt that actually consists of some of the device tree declarations, for example. It sets, for example, what's the speed of the CPU is. It sets, for example, the which kind of HDMI mode it needs to start up, etc. etc. Then it also downloads command library text that will actually tell which kernel needs to be booted. That will actually also tell, for example, what command line options are needed. These are very important because the whole goal is literally that we are going to try to boot the Raspberry Pi completely over the network without any local storage. So things that we could use, for example, full NFS. The problem only with NFS is that it doesn't scale to is a bad NFS is not something, for example, that would use for my etc database of Kubernetes to run on. So actually we chose in this case for ice because he discs and also going to show you that. So actually in the command line context, you will find actually the command line for the kernel to boot up with the target ID, the target initiator from the ice cozy and actually where to find in this case the ice cozy provider, for example, plus a lot of other stuff, either to have a quiet boot or to have a more noisy boot and that kind of stuff. But also, for example, a panic equals 15. They'll actually tell, for example, if something goes wrong and within 15 seconds, we can actually not get a boot root volume for example, then I'll actually restart and we'll try again. Okay, after that, the kernel will be loaded, the initial RAM disk will be loaded and then actually the kernel will be started, will boot from the initial RAM disk and then actually later on this will actually pick up the ice cozy disk and boot from that further on in the second stage boot. Okay, so the kernel is downloaded in it are these downloads, the kernel is started. What are we needing for the solution? Okay, I need to boot because the whole boot principle of the Raspberry Pi is actually to boot off the USB. So what I need is a TFTP server. Of course, I can use one from Peter Andrew, for example. Another one that's really nice actually to use is the one that's already available in the NASMASC. The good part is actually that will give me one ADHD server. So I get my IP addresses. It will give me TFTP and it will actually give me a caching DNS, my DNS server. So also all the pies, for example, that get booted automatically get a name in my DNS. So that's actually one solution, which is really small and actually fits them all. So that's the reason why we chose actually for DNS mask. And the nice thing is actually this one. And again, this is not a Raspberry Pi. The reason why I chose this one is actually this in Odroid. So definitely still are. This one has the ability to have an NVMe disk actually inside. So he has a two terabyte NVMe disk that will actually have all the logicalized QC disks for the Raspberry Pi on it. If that would be possible Raspberry Pi, for example, I could also use a Raspberry Pi for that. Or for example, I could use Raspberry Pi and add, for example, a USB stick or for example, a SSD disk to it. Good. Now we need to use a nice QC server. That's also in this one. And I'm going to demo that a little bit. I'm actually going to show you actually how this works in a few moments. The QC server actually serves the disks out, the root disks of all the Raspberry Pies that are attached to that one. The nice thing actually we're making use in this case of logical volume manager two. And of course there are more ways because important of course is that we try to create a kind of a template for our Raspberry Pies to provision. For desktops, what we actually pick up is a Ubuntu image for example. We need to clean it a little bit. We need to actually add some extra drivers to that one and then actually we need to copy everything over to the logical disk to the iSCSI one. We created actually a ansible playbook for you for that. So I'm actually going to show you that also. And another nice thing of actually using LVM2 is that I have the choice for the iSCSI actually to have my devices, my logical devices being false. Or for example having them as logical volumes. Now we chose actually to use logical volumes in this case. And one, quite simple, it means that I do not have the falseness interface that I have less over at. And the second part is really nice. With LVM2 for example I can actually create snapshots of existing logical volumes. And that means for example I can have a template and I can actually snapshot this template and actually have another Raspberry Pi for example work from that one. And that will cost me one less time to copy it. And the second one actually that will give me a faster startup and actually less space of users of storage space. Good. Then there's another one that's actually nice because what we have now is actually a system, a Raspberry Pi that can boot from TF2P. It will get this kernel because actually that will be an a TF2P actually file system. I'm going to show you a little bit of a drawing and a little bit. And then actually we do have the iSCUSY device. That will be given. That's great. But there's one problem because these Raspberry Pies they have this and depending on what kind of distribution you're actually running. If you have an example of Ubuntu, then your firmware files, your kernel and all your device refiles, they actually boot firmware. That means that boot firmware is actually copied to your TF2P server. That's nice. But if I'm going to actually, OK, if I'm actually going to upgrade, for example, the kernel, it means that these files, the RAM disk, the kernel and maybe even the device refiles, for example, they get updated. But TF2P is a one-way road in this case. So I need to make a way actually to copy back to. So actually what we do, this boot NFS of this boot TF2P, boot firmware, we actually also mount that as NFS. And that means, for example, if I'm going to upgrade the kernel, it will automatically put these things on the NFS file, shared file system, and that's actually also directly available on the TF2P. Then another one, and that's less needed for these ones, for example. But we use that a lot for the Pyforce, for example, when we actually deploy them in a Kubernetes environment. I am making use of CloudInnit, which is actually a tool that has also been used by the big public clouds and that kind of stuff. And with that, we actually can post config or Linux distribution. Now what we can change here, for example, is your hostname because I'm coming from a template and that means actually that from the start, actually, all my Raspberry Pi's will have the same name. That doesn't work very well. So what I'm going to do is actually for every Raspberry Pi that I'm going to deploy, there will be a special directory with the two famous CloudInnit files, user data, metadata, metadata containing the hostname, for example, user data, containing all the items that actually configure the system, for example, changes in the users that need to be made, adding users, adding keys that can stop. Eventually, for example, adding logos, banners, issues, maybe some software, all that kind of stuff is actually possible. And also here, this CloudInnit, what we actually do is in the command line, we actually refer to CloudInnit, in this case, to get it from this web server. Okay. The next thing that we need to do, this is nice. Now, I'm working with a template, but how do I get this template? So actually, what we do is actually installing a Raspberry Pi on the normal way. For example, with an SD card, we need to do it once, or for example, with a USB stick, we boot it up, we install the software that we need. So for example, if I have all my classes in Go, a Rust, for example, I need stuff with a VI, for example, completely configured for Rust. I need to have the latest Rust compiler, for example, and all that stuff we actually put in. If I need a special desktop, for example, for Kits, there's also what I'm going to put in onto it. That will be my base. But we also would like actually to have a distribution that is stable. So I would not like to have, for example, a single file system. I would like to have everything we need and clean in Logical Volumes. So actually what we do then is we have an Ansible Playbook and that will actually run on a remote system. And we'll go to the Raspberry Pi and then actually mount the iSCSI device and then actually rSync almost everything over with some changes. And also for example, they're making sure that the Logical Volume Manager is being used there. So we have far on a different file system. There's a lot of different file system, file log, file log security, all that kind of stuff. Okay. We chose two distributions to actually work the more. Also with, for example, the Ubuntu, the previous ones, like 21.04 and 22.10. This one is actually now the most stable one that we have. Pi OS 64 is also very nice. The nice thing with Pi OS 4, and unfortunately I cannot demo it because the HDMI cable doesn't like the output of my other Raspberry Pi 400, but that one, for example, is running KD right nicely. And also with some of the acceleration actually that the GPU inside of the API 400, for example, does. The reason, so for choosing for Pi OS 64, for example, is using the acceleration, for example, of the graphics what you have, for example, in Mozilla and Chrome, sorry, not in Mozilla, what you have, for example, in Chrome and that kind of stuff. For Kubernetes, we use the 22.04 most. Okay. So, let me see. I need to add a little bit. My laptop refuses to mirror the screen. It could be that my laptop is a 4K and this screen is actually a 2K. Let's see if I put it to the screen. Then I have to make it a little bit smaller. Yeah, I'll pull it just for this. Make it here a little smaller and need to do it a little bit with some apologies. Okay, what we actually have here, this is, for example, in this case, the Odroid. That's actually doing the function of, yeah, excuse me, the server, the CloudInnet server and the TFTP boot server and of course also DNS mask with DNS and that kind of stuff. With our lab, we actually have this on a, running on a NAS, for example, we're using a QNAP for that one, but you can actually use anything that I actually did is port all the functionality in such a small system. Okay, first thing that we actually have is the TFTP boot and that will actually, in this case, provide the kernel and all the boot files and also all the device tree files, for example, to the Python or the Py4 over the network. The other way around, we also have one line because quite simply, when we upgrade, actually, the kernel, all the upgraded images, the upgraded init ID actually needs to be written back in the same directory here in the TFTP boot. Then another stream, actually, that we have is the CloudInnet. So once the kernel actually has been booted, in one of the first stages, actually, what it will do is it will execute CloudInnet. That will actually look for the user data file and then actually apply all the customizations, actually, to the image. And of course, the ISCSI, which actually is providing the real root disk, actually, for the system. Okay. Okay, Quicks. Yes, we still have them. What happens, and we'll see that actually, when I demo it, we'll give two demos, one actually for the Py400, to see how it actually boots into Ubuntu and that kind of stuff. We'll check then also on the Odroid, actually, how that works. And the second one will be, we'll try to actually boot up two of these guys. They actually will completely set themselves up, so they're completely empty. The Py400 actually here has already been set up, so we don't see too much of them, actually, that it just works. But Quicks, that we have, for example, is in the CloudInnet. Sometimes we run into a race condition. That means that most of the times, actually, and we might see that here with the Pyforce, for example, we see that it does have actually access to the Nginx set, to the web serve, actually, for the CloudInnet files, but still isn't able to do that and actually will fall back then to the fallback resource and actually means that we will still be stuck with the old hostname that's actually on the template. The way to fix that manually is actually just logging on to the system and then, for example, the way CloudInnet is cleaned, for example, and then that will actually fix that one also. And another one that we have, for example, is, for example, Ubuntu is there notorious for. It has the option, actually, to do all the updates, actually, of security features. And sometimes that doesn't go really well, but actually leaves out certain drivers and what, of course, is disastrous for or set up is if the LVM2 driver is gone or the exclusive driver is gone, which actually means that the system cannot boot again. We can easily fix this also by just copying, for example, the files of the previous backup files on the nasty files, in this case, on the boot supplier. Good. Let's see if we can get it demoed. Okay, I need to do this a little bit, switching one and forward and back. Again, apologies for that, but let's do that. This is my Raspberry Pi. I'm going to actually now unplug it from the power. So here it is.  Now we are, I plug this one in. Then the Raspberry Pi will get power. I believe we'll get an image in a few moments. Yes, here it is. I'm going to, this is the new boot screen action. This is a fairly new, actually, firmware. This one has already the HTTP boot in it. We're not going to use that. What I'm going to do is I'm going to use the escape key here to, and again, it's not looking really very readable. But what it does here, you see here the boot order. In this case, the USB, then you see here twice again for the USB, says that it twice the SD card. And now it actually goes into the boot process of the network. You saw the TF2P boot. You saw that it actually got its own IP address. It did find the IP address of the TF2P server. And this is, of course, the Chroma screen that we have. And it's actually that now it's going to boot the kernel. This shouldn't take too long, but every time that you look at it, it's taking its time. Okay, this means that the kernel has been booted. And in a few seconds, we should actually see that the video adapter is actually getting screen output again, okay, there it goes. Again, here you see it booting and goes a little bit fast, but you have to believe me. This is actually the cloud image. You can see that here with the pluses and that kind of stuff. On the top of it, you could have seen, if you read very carefully, the attaching of the ice because he devices, actually the volume groups are actually being activated. That's actually now coming here. Now it's going to start up the desktop environment. Now you see the cursor of the desktop environment. And here we have a nice 22.4 and again, you can check it. There's no USB cards. There's no SD card in it. This is just booting actually just from this cable, from this switch and from that apparatus. Okay, what I'm going to do now is I'm going to switch back again to my laptop and I'm going to try to get on board of the Odroid. So actually you can see the locks of the DNS mask and I'll show you that. And we can see, for example, what happens on site of the Raspberry Pi. Okay, let's do that. And again, it's a little bit unfortunate that the screens don't work. Here we have it. I'm typing a little bit in the blind. So forgive me my typing mistakes. So I have to look up here sometime. Now I'm actually on the Odroid. Let me see if I can make this a little bit more readable. Okay, we have to do it this way. Again, here, this is just the normal login screen what I'm going to do now. I'm going to, for example, yeah, it works better. This is readable. This is my DNS mask print code. Now we'll see some special stuff actually in this one. This is just the address I'm listing on. This is actually the internet device that's actually here. Here, I have a private range in this case from two to 100. This is will be given for 24 hours. This will be the router, which will be that thing itself and putting up the NTP server. The good part is this one normally does do have a battery backed up RTC clock. Unfortunately, it did not survive the travel with the plane, so I need to check in that one. The DNS server, the net mask, et cetera. Then what I do here, I actually put in the different, so here, we have, for example, some of the DSP hosts actually going to fix, for example, the MAC address to a name. I'm not going to use that here. This one actually does, but that doesn't work for me. Here, I'm going to log all the communication actually the DNS mask and, for example, for the TFTP protocol, that's very handy. I actually see what's happening to see if actually the Raspberry Pi picks it up to see which files, for example, getting sent and that kind of stuff. For example, if I didn't prepare my TFTP boot, for example, for the serial number of the Raspberry Pi, I could see that that it couldn't find a file, for example. Here with, I'm actually enabling TFTP and also setting the directory. We're going to look into that a little bit further. And actually what I'm doing here is this needs to be specifically there for the Raspberry Pi. Normally, we do not do things, for example, for x86 boots or UFI boots, for example. But by using this, PXE serves as zero Raspberry Pi boot. We can actually recognize or have TFTP recognize Raspberry Pi is actually to do a TFTP request. So here, this is the TFTP boot directory. Unfortunately, this is a horrible color actually to read. What you can see here, these are actually the serial numbers of the CPU purchase. So you see here, a few of them, you see sometimes a little bit of a different name. There's actually where I copy the snapshot, for example. Where can you get these? Well, quite simple. Let's see, for example, first, what the IP address was that we booted from. So, it's of which Pi actually booted from us. Let's see that. You see here, the TFTP boot process with all the options that have been sent. What I'm looking for is actually the IP address. Let me see that. We'll leave this one came in at 712. If nothing is actually down the other line. That's my start. It's a request for a 7.3. So most probably we have 7.3. Okay, now logged on to my Raspberry Pi in the DTCN05. And what we can see here, for example, if I pick up the FSTOP, you can see here nicely, see here, these are actually the file systems that have been created here important. You see here the NFS boot of the distribute with exactly this number. And that means actually that the serial number of this Raspberry Pi, you can see that here is indeed this number. You have to strip off, by the way, the first numbers. So here, 669, 2D, E2F, that's actually my serial number. And so every Raspberry Pi, when it boots off DSTOP, it will look for files in a directory that's been called after this serial number. And if we go, for example, in this case, this is a Ubuntu one. So you can see here, all the files that are in the DSTOP, well, that doesn't look like it. You see here, all the files that are there in the DSTOP boot directory and by investment synchronized, actually, with the boot server. And you see here, for example, for an update for a new eProm. And this is also possible. We could, for example, put a new Pi eProm, for example, a package here. And when that's actually been picked up by the boot, it will first upgrade to eProm and then actually do normal boot and that kind of stuff. You'll find the run disk, all the overlays, a little bit under that, you'll find again here, we have the command line context. That's also nice, actually, to look into that. This looks impressive. We can see here, in this case, a ZSTOP enabled, that's actually for the newer Ubuntu's, they have that one. Here, it's getting actually interesting. The console is TTY1, root of S type is XFS, we did change that. This is actually telling me that my root disk will be on a VG root, LV root. This is my initiator and this is my target. So actually, there I can get my ice cruising devices from. In this case, learn one, for example. I could also have multiple ones like this, for example, for the Kubernetes one, we actually add an order disk for the file containers of our Docker, for example. And here we have Dan, the root bytes, the elevator deadline, the root weights, which is actually for root to be active. Fix the RTC, this is the fix actually to get the latest time actually on the file system when the system actually stops. Panic is five, if we didn't face in seconds, we do not have a mountable root disk. Now we actually go to reboot. And this one is the cloud init one. So actually the provider is no cloud net. And then actually from this one, which is actually the IP address of the engineering server there, we're actually going to, in this directory, get the user data and the metadata files. Okay. So that's how a 400 actually boots and how we did do this. Okay. Let's go a little bit back, see where we are now. I'm on the other road again. That's good. Okay. So the other stuff on the other road, we can find in VARWW. Let's see. Yes. That would be a nice one. Yes. Now I don't have it installed. Now in the bad of actually, the bad luck is actually, I did have a second monitor actually also to show me that, but I didn't bring that with me. So in this case, we see here the different systems. Let's pick up the five. Here we have the user data. There's a lot of stuff actually here. What we actually have here is so the preserve hostname, that means you're going to reset the hostname. And normally we would have to reset the hostname actually here from the template. Here I'm actually going to define all the users that need to be created. And at the end, we're actually doing one more trick. Let me see. This is all the users. And then we have here run command. And that actually downloads a script and actually does the last configuration that what we actually could not express ourselves in cloud in it. So in this case, this does the following. I'm going to do this one, then I'm going to cloud in it. I have here, yep, let's go in that stage. For example, what this one does is actually putting a banner and then also make that actually readable when you're actually using a console. For example, the other thing that it does, it makes sure that the ice-coosy, because that's also coming from the template. The ice-coosy initiator is actually rewritten that kind of stuff. We're going to delete, in this case, the old registry actually from the ice-coosy devices. I'm doing some check here, the CPU info, because we also have the old FSTAP entry actually from the template, the Raspberry Pi actually that was used to template it. So we need to change the serial number. Put that in SFTAP. And then actually the last thing that in this case user data does is reboot it again. It will then reboot with the new house name, it will reboot them with the correct ice-coosy initiator and that kind of stuff. And then actually then we're in business. Good, one more thing for this one. This is actually a dance for Playbook graph, which my older son actually has created when I asked him this is all actually a pick up a Raspberry Pi where we actually have installed a template Ubuntu or a template PiOS for example. And it will then actually transfer that to a ice-coosy device that we can actually use as a template. So I'm just going to do it on a high level. What we do here is we're going to check it. Then we're going to do the disk layout. So I'm actually going to describe everything. Let me show you how we do that. Everything with rules. Those of you who did attend the previous talk here in this room, the guy was already talking a lot about Ansible and that kind of stuff and a lot of what he talked about. You can also see here. So we go to disk layout as I roll. Then we go to tasks. A few things that we do here. First of course is making sure that we get the ice-coosy number one, then actually creating the volume groups. And once we done that, creating the file systems, we have a special dictionary. Actually it creates all the files and we can actually customize that kind of stuff. We are going to write everything in a temporary FSTAP file. Then we actually going to write it back in the write format. And then of course we're going to pick up the serial number and also put that in this case in the FSTAP. So then it's actually quite simple. What we're actually doing in this case is the R-sync to the new, the R-sync is what we do here. We actually write it to the ice-coosy device that we have. And then it's quite simple. Then actually we can copy that ice-coosy device. You can just do that by DD. You can also make a snapshot from of it and then actually have that one attached to your Pi. Okay, let me show one more thing. This was the Pi 400. I'm going to hurt it a little bit. So I'm going to say, I don't like this. Okay. All we're going to do now is to see if it works. I'm going to provision one of the Pi 400s. Lucky, change this. The problem is most probably that the screen is not so readable. Let's see that. The good part is this one has a power button. Yes, now it's starting. There it goes. This one has a little bit of a older boot loader, as you can see. No, actually not. It's the same as new one, but this one actually directly starts. You can see here that it tries to boot from the USB. That one won't work. So in a few moments, it will actually go over to TF2P again. Okay, link is ready, it says. You see, not a time. You see that it actually tries to find the boot images by actually looking for the serial number. This is again the Chrome screen. Obviously, after that, we'll actually boot up the Ice-Cuzzie again. And what it does here, in contrast to the Raspberry Pi phone, which was already initialized actually, when we're a little bit lucky, this one will actually initialize itself using CloudInit, so it will actually boot twice. And unfortunately, the screen is not readable. And I cannot change that, unfortunately. As long as the screen is calling, everything is okay. And the good part is that this is actually how we then also add, for example, Raspberry Pi's as nodes to our cluster. It goes really fast. Here you can see, with the pluses, this is actually the CloudInit that is running. You see multiple sections actually. So this actually means this is the initial one. Let's see how far it comes. If it reboots, then actually it's okay. And it does do its job. If it doesn't reboot, it means we have this race condition. Okay, there it goes. So this is CloudInit. Having the final reboot, what it actually did now is, it changed the Ice-Cuzzie initiator back to what's actually used for this system, named after those names. It did change the everstop. And so that actually the boot firmware, for example, is now connected to its own NFS directly on the TF2 Reboot. And then it actually reboots again, and I have to wait. But it in, yeah, let's say 80 seconds or so, we can have a new node actually there. And actually using an answerable playbook, what we do is we actually pick up the new node. We actually go back to the master nodes, request there the join key, and then actually directly join this one. And that means that Kubernetes is on itself, for example, and it's two minutes actually to pull in all the images. And then actually everything can be just useful production. Here it goes again, the chroma key. Are there any questions already? Please. You did put a lot of work in all of this. Are you publishing? Yes, yes, definitely. I will make sure that I already was busy with it last night. I did miss the Guinness. A lot of you all did it also. But yes, I'm going to come to put it on GitHub so that you can all get this kind of stuff. Yeah, you're welcome. Now a lot of people actually work with this kind of stuff, but not many people actually have done the eyes because you won.  And actually we did it with NFS for example. It's not the most elegant way, but actually it works like a charm. The other way for example, yeah, what I really would like to do is put some more effort in it. For example, I have this one function as a NTP server. So all the Raspberry Pi's actually directly when they come up, they do have to use it here. Using other technology, actually making the best of these Raspberry Pi's because they're really great work and great toys actually to work with. More questions. Good, now you can see it's online head. Now it's actually has its own logo and that kind of stuff. If you read very carefully, you can actually see it's host name. Good, oh that's a very good question. If I, within SD card for example, depending on one from Ali Baba or one from the AM, how do you call it? The Samsung ones for example, the premium ones, you get 80 to 90 megabytes per second. On this one with ice-cozy for example, and again, this is just a one gigabyte, so it is not really super, so the processes in these are actually a little bit more powerful than in the Raspberry Pi, but I get some 320 megabytes per second read, 320 seconds per second write. So for Kubernetes, that works. More questions. Yeah, you could do something like it because the question was not, is it possible to have a single disk actually? And then you can do two things, a single template or really a single disk? Yeah, what you could do in this case, we chose for LVM2. So yeah, you can actually create one disk and actually give the older ones actually a snapshot of that disk. So they still have their, because they need to have their own storage, of course. These are not containers, but they need to have their own storage. And the other way that you could do, for example, is overlay it with a temp file system, for example, a ROM disk or something, that could open. Well, my choice would be actually do something indeed, have one real disk, have the other ones actually snapshots from that one. And give them some extra space, for example, 10 gigabytes, for example, so that they can make the changes. And later on, for example, for some of them, you can even merge them back if you'd like to. Yes. Yeah, what you have with it is really nice actually to see. And you see it more, for example, when you have, and we can also do these things actually with highly available Kubernetes, for example. HCD is a key value store. And that means that it actually makes really fast updates. And actually the whole idea is that HCD doesn't go and search, for example, what was the next thing that I need to write. It actually tries to, each time, write in a different place and that kind of, things is actually horrible for an SD card. And in this case, for ice-cusy, yeah, we don't see the different. For NFS. Yes, for NFS, yeah, sorry. Yeah. Yeah. Because you have all these ops, the mutiny of the file system, that kind of stuff. If you have a really big NFS, you should get it to work. But actually, yeah, okay, let's do it as it's actually supposed to do. Normally, you don't run actually SD database on NFS also. And that's why we actually chose for, yeah, a normal logical file. More questions. A very good question, by the way. Okay, yes. Yeah. Good. More questions? Okay, then let's round it up. Many thanks for your attention. Really love it. It's late on the day and that kind of stuff and yeah. Thank you.


Everybody forced here being a box of 2023 93 and 2 baseball hobby for Daniel F. Appreciate the order Daniel Let's find an LED like cruise auto here It's a big name right now A lot of good names to hit in this though. See Willie Mays autos in this. It's a All prospect team Kevin Parada to 50 sluggers Eddie Murray out of 150 and Sluggers Cal Ripken out of 50 Recolored jumbo patch Brady house on the prospect Jersey to 25 Yeah, Lane to say hey kid To a 25 Brady house three color patch. It's crazy. How deep the nationals like bats are in their farm system Nationals legend actually on the Lone Star Bat Relic auto Ryan Zimmerman to 25 Nats legend on the Zimmerman. All right, that'll do it appreciate the order Daniel. You will get these to you