Well, in many years ago, we always remember that some hackers, they are going to directly solder some wire into PC, and they can use PC to control various things, but nowadays, all the personal computers are with very high speed interface, something like PCI or USB, and you cannot directly attach a cable into your computer now. So, because what I'm working on is something like software-defined radio, or where has the data transmission, or data processing, making something like FPGA code processor to accelerate the data processing that is much faster than CPU. And so I need something like, I need a way to process to transfer data really, really fast. So now I'm working into this topic. I try to use PCI Express Bus to directly assess the main memory and the transfer data really fast. And here is a brief introduction. So for Nubos, I just explained the PCI Express in a quick and fast way. So let me make sure everyone understand this. Well, PCI Express is a serial link, so it cannot be directly converted into IO. It's not like a parallel bus. It's not like a printer port or serial port. It's high-speed series, so you need some kind of very complicated thing, but that's not important. We can buy it or use a converter. So the first thing we know is PCI Express is something mysterious and very high-speed data link, and it can go, nowadays we have PCI-5 version. So it goes to 64 gigabytes per second. That's blazing fast. And for the slot, the mysterious slot that is since fragile, all the high-speed data flows into it, it's not a very complex structure. Actually, the slot itself is something like a clock, a reset signal, then beta pairs that we call the serial link. So that's all in the slot, also with power supply. So that's it. That is what happened on the motherboard. Why we use PCI Express not Ethernet or USB? It's because the latency of Ethernet or USB 100 or 1000 times higher than PCI Express. Because PCI Express is directly connected to CPU. That's this. That's this one. Well, logically, you can see that the PCI is directly soldering all the primary DDR memory, main memory, directly soldered all the pins out. Logically, it's like you directly tap into the main memory. So the latency is low and it doesn't require CPU in the CPU process. It doesn't require CPU to interact with data transfer. Actually, you can think the CPU, the PCI Express core in the CPU is some kind of switching. It's something like an Ethernet switch. You can think it's a switch. So when you write, when you try to, let's say, the Ethernet card is running like CPU. We don't need CPU to receive the Ethernet packet. The Ethernet card can directly write, receive the packets to memory RAM, directly write it through PCIe bus without the CPU's knowledge. So that's why the PCI Express is the best way to do data transmission or to make a core processing. Because you can see that is your card, your PCI Express device is another CPU, and then you can access the main memory or other cards. Remember, the PCI Express is a memory, it's a memory bus. It's not something like I send a packet to CPU and the CPU goes into interrupt and grabbing data out, not like this. Actually, the CPU doesn't know what you are doing on PCI Express. Well, for the data processing, you can directly read all the data out of main memory, then process it in your card and write the result directly into the memory without CPU's knowledge. So CPU just leave there and check whether you have processed all the data or you can directly make a link. You can use PCI Express to connect the two PC together, derive a transfer of RAM contents from one PC to another. Well, that's very practical and it's ten times lower latency than Ethernet because there's no IT protocol stack or something like that. It's not secure, actually, because anyone can install a PCI Express card on your computer, that means they have total access to your computer. So game cheaters now, they use PCI Express to do cheating because the hardware cheat can directly read memory contents, so without CPU cannot detect this. Yes, that's the perfect way to do game cheating. Also, you even can write that can change your mouse position, be ready to enemies have it. And CPU doesn't know, it knows nothing about this. It's undetectable. Well, I mentioned this just now. Well, for, let's say the, what is DMA? DMA is something like you can use the main memory as you wish, and without CPUs knowledge, so you can directly read or write main memory. That means that's a huge advantage to USB, because USB when you transfer data block, something like a 4k bytes, you need an interrupter or CPU need to grab the data block out to process it. In, with PCI Express, you can take a one terabyte RAM on a server or huge server, and you can use FPGA to fill out all the DDR RAM, all the one terabyte RAM. And that could take something like several tens of seconds, and your CPU just can do other things without any problem. You don't need to do something like every data block you need to interrupt or process it. You don't need to. So that's a huge advantage when you're doing a huge amount of data. And also PCI is very low latency. Actually, I'm using PC as some kind of real time controller, because we know that PC is much, much faster than microcontrollers. It's running on four or five gigahertz clock, and you can do tons of floating point mass in your computer. Then you can control a machine, something like a radio or some quantum physics device, which needs very fast reaction. You can use PC to do something like 15 meg cycles per second to do a very fast responding control. Well, also for robotics, the PCI bus is much reliable than USB, where on a server platform, it's, it doesn't failure. Yeah, it doesn't fail. So when you're controlling something like a car or a robot from a computer, the best way is you try to attach it on PCI Express. Well, the last one is for high interconnect is since we know that the PCI Express is some kind of a memory bus, so you can write some data into memory or write a write data into another card, and the data will flow through the CPU as this. We can take this. You can see that if the data flows through the CPU, it's fast, and the CPU don't need to take any time to process it. So it can be used as some kind of something like a Ethernet switch or package switch, but it's much faster than Ethernet, lower latency than Ethernet. Now, why the AMD, new AMD Ryzen or EPYC platform has so many PCI LANs is for, they are actually those server users, they are using the PCI Express as some kind of, you can think it's an equivalent to Ethernet, but its latency are much, much lower. Also, the logic is similar. Well, to connect something to PCI Express, I use this one, LT5031, where it's available on Chinese Taobao, and also you can email me if you want something like this. Well, that is the lowest, I think this is the lowest cost of high-end Chinese FPGA board. Well, this one, the 3K, the 7K series, this FPGA chip is labeled like $5,000 on DDT, but actually when you buy it in massive quantities, in a massive amount, you can get a price something like $100. That's 50 times different. And the real transfer speed we measured here is 3.6 gigabytes per second. Well, it's because all APJs, APJs are generally they cannot handle the latest PCI Express version, it can do something like PCI Gen 2 or Gen 3, it cannot do Gen 5 currently, or if you want to do something like PCI Express Gen 4 or 5 or APJ, that could be really, really expensive. Those chips are hard to obtain and the price goes up to like several thousand dollars, but why I choose the PCI Gen 2? Gen 2 is old, but the chip is low cost. This one, this board can be bought in less than $200. Anyway. So, PCI Gen 2, does that work? Yes, yes. This one cannot do Gen 4 or 5 because the internal digital circuits in the FPGA are not fast enough. Is it more motherboard compatible backwards? Yes, all PCI Express are backward compatible. You can attach something like Gen 1 to your latest, very expensive gaming rig. What does that mean? Because you have all these components. You have one component that is slower, much slower. Yeah. How does that affect the rest of the system? Because physically, PCI Express is point-to-point lean. It's just like Ethernet. So, actually, the fastest component in your system is the CPU. So, CPU won't be the bottleneck unless you use a very old CPU, and a single device, very slow, doesn't affect the total system performance. You can attach something like 20 cards from Gen 1 to Gen 5 on a Gen 5 server. And the server will run in Gen 5 speed. Yes, because it's packet-based. Physically, it's packet-based, so it's just like Ethernet. The very slow device doesn't affect the fast one. Well, how does PCI Express transfer the data? It uses TLP packets, a transaction-layer packet. You can think this one is like UDP, but it's much simpler than Ethernet. You can only do write. Well, if you want to write a main memory, something like change some data in your game, you just directly send the TLP packet to your computer, and it will change the data in main memory. That's it. It doesn't require something like acknowledgement. You just directly send the write request. Well, for reading, you need to send the read request and just several nanoseconds after you get the reply, the read reply with data. Well, also, PCI Express can do interrupts. It's just like interrupts in microcontrollers. You can interrupt the system kernel, and that could be very low latency. Nowadays, nowadays, more than CPUs, they don't use wires, pins to do interrupts. Actually, they use message. It's something like when you want to interrupt the CPU, you send a packet into PCI Express, or internally, it's a packet in the CPU. And the CPU will receive the packet, see it as an event. A packet is an event. So when the CPU got this packet, it knows that, okay, this is an interrupt that happened. We can process it later. Well, here is the way the software code used to do data transfer. It called REFA. Well, REFA is originally, it's written by UCSD, American University, but the original version has some bug and also missing some important features. Well, originally, REFA can only do direct memory access, something like read or write your main memory, but I add AXI4 logic. What is AXI4? It's something reverse. It's CPU assessing the PCI Express device. DMA is device assessing main memory, but it can be reversed. You can use CPU to access registers in your APGA card. That can enable something like a 10 bus or wireless user defined logic. So I add an AXI4, which basically is registers. So with this modification, you can, the CPU can change settings in the card or change register wireless in the card. And also I add IRQ functions. So the APGA can send the interrupts to CPU. Well, here's some demonstrations about the latency. This is how the APGA card attached on my testing computer. And the laptop on the left, this one is attached to our logic analysis. Well, because I need to do some low level debugging. Well, for PCI Express, it's hard to debug, actually. So writing the, modifying the software took me a very long time. But now it's working, now it's working perfectly. And it also requires some kind of special tools. You want to do something like a very low level of physical debugging, because the data is really fast. So even you have a high end oscilloscope, even you have an oscilloscope with tens of gigahertz bandwidth, you cannot, you cannot record enough time for finding the failure. Something like if it fails once a day, you need a huge amount of memory to record it. So I got this, I got this one, the Keysight logic analyzer dedicated for PCI Express. This is called, when new one is something like $150,000. But I managed to connect this thing from eBay garbage. And after I connected something like, I connected something like tens of components, I put them all together and now it's working. Well, it is for very difficult low level bugs, something like if it fails in the very bottom layer, cannot be analyzed by FPGA debugging tools, I need this thing, yeah. Well, also setting this thing up is, you have time consuming and you need another computer to handle it. So I needed something like three computers, one for compiling the firmware for FPGA, and another for running limits and testing the PCI Express card, and also the third one is for running the logic analyzer. Also, the logic analyzer itself is PCI Express, the green cable you see, it's attached to PCI Express on a desktop computer. This is how the logic analyzer, the logic analyzer looks like. Basically it's the package running through the PCI Express LAN with the debugging tool by Keysight, you can see all this happening. Normally in 99% of my working time, I don't need this, I don't need it, but if I run into some common cases, something like someone reported a bug, send an issue to me, I need to debug it with this thing. Well, this is the latency measurement, it's 200 hertz, once five millisecond, the FPGA send an interrupt to the Linux operating system, and the operating system will give the recorded time stamp. So with the time stamp values, I can check the control latency of the PCI interrupts. Well, the test read is something like I3, with all CPUs calls running to 100%. How I reach 100% is by using 17. I do a 17 benchmark and the saturated CPU, and then I measure the latency. Well, this is actually very impressive, it's something like the nominal value of the interval should be five millisecond, but the actual measure the value is minus one or positive one millisecond. So basically that means the CPU can answer the interrupt in less than one millisecond guaranteed. Well, even with CPUs, all the CPU calls are saturated. Well, that means if you control a robot in something like 200 hertz, that could be very stable. Well, if I lower the CPU usage to 17%, the latency will go down to positive or minus 100 microseconds. 10 times slower than the saturated CPU. The peaks, actually these peaks are caused by loading their else into application. Well, this is very interesting if you keep everything static in your application, you have an application and the receiving interrupts from PCI Express and do some data processing. Without any memory dynamic allocation or dynamic module load or unloading, something like if you don't try to load that DLL, the latency can be very, very stable. But if you try to load this application or load that DLL into your system where that can cause the latency goes high. But by it's not very high, it's something like four, three or four hundred microseconds. It's not as fast as embedded controllers since we are running desktop limits. But I think this is good enough to build something like a motor controller in one kilohertz. The reason why I use a PC to control motor is because I can run various experiment and checking the waveforms or doing various fancy stuff in PC. It's much faster, easier to change PC applications than doing microcontroller. If you want to debug cable jpeg, you can record everything into memory and check your status. So basically that's all. Any questions? How long do you think to put this together? Your logic analyzer? My logic analyzer is six months and this one, the PCI spread code to me another six months. But since at that time it's COVID, I have final time debugging at home. Is that your DLL? No, no, no, it's not my DLL. It is something like I want to do signal processing, SDR acceleration. So CPU is not good at doing this stuff. I want to process all the data in APGA.  Anything commercial without open source is a disaster. So I walk into the source solution. But there's a hidden bug. Hide the big, big into the river and it can just interrupt your data transfer. It can break the data transfer at a random case. So I took something like two months to hunt it down. But now it's very stable. I can write something like 3.6 gigabyte per second, 7 by 24 hours and for one month without any errors in the data stream. You mean the PCI interface? You mean the main one. The logic analyzer, that one is called the PCI interposer. We use it to amplify the very weak and high speed signal and pick up by the logic analyzer. If I don't need to use the logic analyzer, I can remove the bottom part. I can remove the APGA both into the computer. Well, this one just has some minor issues. It's not a standard PCI card form. It's because I draw the PCB design really really fast and I forget to check the outline of the standard PCI card. So I leave it there. Maybe several months later I will try to reform it. I will try to change it to standard PCI card so I can sell it. But this one is really good for doing development. Well, because this card has already used all transceivers, high speed transceivers, so I cannot attach something like a 10G internet to it. If you have a better APGA, you can try to do something like a real-time graphics or APGA.


soldiers of the press this week phantom enemy this is Reynolds Packard United Press correspondent in Italy I want to tell you something about what it was like on the Italian front during the days when the communiques read no action of consequence something about the place where doughboys were fighting wasn't a place really not a place where you could put a mark and say here it begins and here it ends not even the boundaries were clear for a mist and fog had smudged the skyline into dim hazy shadows war had made grotesque and weird what the four was a languid valley weather blotted out the rest soaking the land shrouding it in clouds until it seemed as if it were not a place at all it had been raining for four days hard resistant ground had broken given way rain had become as much a part of the battlefield as the gray hills in the distance rain had become the third army on the battlefield jeep slid it off the roads pools of water seeped under the tents the mist was over the battlefield to like a sullen gray panther heavy paws covering everything there was little fighting just small bands of patrols waiting into no man's land hardly able to see each other like this quick sand Johnny stick close okay Dave I'm with you just tell me one thing just one thing yeah where's the war huh where's the war where's the man with a gun that I can shoot at oh I give my eye teeth for something good and solid I could shoot give mine for something good and solid I could eat Hank yeah but you're old man in the last war yeah my old man was in it but he never got further than Virginia my father used to tell me he got over one of the first he used to tell me not much but enough you know Hank yeah I guess in a way I never much believed it but I think it must have been a lot like this those days in France must have looked like this you know blind fighting against a lousy mist sticking up to your knees and mud crawling on your belly getting blown up by some on you can't even see take it easy Bob take it easy will you you know Hank I bet I even look like him now I bet with this mud all over me I could be him fighting the same war the same way it's a damn funny feeling yeah yeah I know what you mean he used to laugh when he told me what a lousy way it was to see the France everyone raised about you know how the travel folders call it sunny France yeah yeah I guess the travel folders fooled us too well I guess they have different weather during wars skies been the same color for four days I can't even tell whether it's day or night anymore it's day lunchtime I got my own way of telling yeah well I'd rather be in a feeling bright sunlight where the Germans could see me then I could see them too listen Johnny I was at Casper that was a sunny day and that was hell yeah I know but it wasn't blind man's bluff that's what this is I want something to put my hands on maybe you'll get it maybe you won't funny thing about a war keep low that came from over there to the right it might have been a blind shot they can't see us either look over there yeah just about make them out white shadows looks like a looks like a bunch of them shut up and keep down now if we start running they'll see us they come in this way if we keep low we can surprise them when they get close enough ambush and boss yeah bugs on our side this time get your grenades ready okay they look like a ghost patrol well they aren't you all ready yeah we're all kept don't worry okay here they come you can tell them now two three five six enough to shut up huh count to four then Hank shut up I tell you shut up yeah but Hank cut it Johnny they're coming close I know but it's our own man it's an American patrol that's right we say you're right we were about to take your prisoners we had the same thing in mind nice weather we're having oh boy is it glad to see you fellas so long fellas so long what do you know it's like meeting a long lost friend during rush hour in a New York subway you can hardly see them anymore like they'd been swallowed up yeah strange place there's no man's life well come on fellas keep on going this way okay it's now three patrols yesterday didn't yeah none of them came back to me no no none of them came back I was just thinking I wonder if any one of them even saw the Jerry's that got him maybe just got kicked from nowhere doesn't seem fair you know not knowing what you're up I don't let it get your kid you know it's getting so I just like to see a rock a tree than the tall real we're standing on a piece of ground that stayed there we're getting closer to enemy lines from here on we keep our heads down and our tails lower doesn't exactly smell like lilacs down here yes lovely shut up you guys don't forget we're moving into German territory come on holy smoke get back quick German outpost what we practically walk right into it how many do you think but eight at least I didn't count but they've got enough there to blow us up now we can beat it if you want we can report the outpost to headquarters or we can stay and grab them I mean we can make a good try what do you say I say we make a good try meet your wife well it's not easy they're eight of them maybe a couple of more four of us but if we work it right I think we can do it say if we can come on if we split up now one thing we've got to remember the less shooting the better we want these guys alive a question how we're gonna get them without fireworks there'll be fireworks all right you just have to think we're a squad instead of a patrol that's all now I'll see how this sounds I had shoot two of us several northwest of the post one goes over northeast one advances from here opening fire shooting wild and making a lot of noise see that'll give them a scare and they'll open up on them and split second later the others charge them from behind they'll think they're surrounded maybe anyway it's worth a try huh sure it was a try right I guess you guys know whoever stays behind open fire there's an even chance of being knocked off I'd like to do it nothing doing listen Hank I know you're in command of this patrol but when it comes to taking a chance like that we all are to have a say in about who doesn't Dave's right Hank I think we ought to draw for it okay okay yeah let's draw we'll draw here's a piece of paper kind of dad that's all right okay long straw gets it one two three four get them ready okay there they are ready draw Dave okay take my Johnny Bob yeah thanks and I get this one well I've got it okay Johnny now fixie watches to exactly 245 at 246 Bob and Dave take the northwest position I'll go northeast Johnny stays here by 255 we all should be where we belong at 257 Johnny starts shooting got it that's okay then a second later we're rushing from two directions now if we've got a shoot we shoot see and Johnny yeah we won't give them a chance to get their rain thanks you're ready yeah we're all ready we're all set it's 246 let's go good luck Johnny here's the war you've been waiting for so long so long the little skirmish in the mud and rain went off according to plan Hank and Dave and Bob charged in two directions the German outpost was completely surprised the Germans had more ammunition and more men than the little patrol but they surrendered four men had gone out four came back but only three of them alive at 257 Johnny opened fire at 257 in the quarter the Germans returned fire his buddies carried Johnny's body back through the rain and they led their prisoners back to their job was over that night the communiques reported only minor patrol activity on the Italian front today it might have been a quarter of a century ago the men who were over there in 1917 knew the same kind of war the same misery of rain and fog sudden burst of shells the sudden bitter skirmish is on ground that's slid under foot only the place was different and the songs were different then they were heading for Berlin this time Rome not a war of front-page action and big battles no just a few men waiting into a bleak dismal no man's land fighting an unseen enemy getting hurt dying while the word went back to the people at home no action of consequence on the front today United Press correspondent Reynolds Packard has just brought you the story of a small American patrol fighting in Italy not only in Italy but on every fighting front in this global war United Press correspondents are with our armed forces sending home reports on the stories of courage and endurance that the official communiques must leave untold sending you up to the minute dispatches on the action that will ultimately spell victory so look for United Press dispatches in your favorite newspaper listen to United Press news on the air it is your guarantee of the world's best coverage of the world's biggest news


One of one! Brock Purdy! NFL Shield! Go into the Niners! Come on, Sean! There he is. Let's go, man! There he is.  My God, Sean!


All right, we're going to go ahead and get started. I'd like to thank everyone who's joining us today for the webinar, Immutable Infrastructure in the Age of Kubernetes. I'm Taylor Wagner, the operations analyst here at CNCF, and I will be moderating today's webinar. We would like to welcome our presenter today, Tim Gorilla, the CEO of Tello Systems. A few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee, but there is a Q&A box at the bottom of your screen. So please feel free to drop your questions in the Q&A, and we'll get back to you as soon as we can towards the end of the webinar. A reminder that we want to use the Q&A box and not the chat window. So please direct all questions to Q&A. This is an official webinar of the CNCF, and as such is subject to the CNCF code of conduct. So please do not add anything to the chat or to the questions that would be in violation of the code of conduct. So please basically just be respectful of all of your fellow participants and our presenter today. And one last thing, the recording and slides will be posted on the CNCF webinar page at cncf.io slash webinars later today. With that, I'll hand it over to Tim with me to pick off today's presentation. Thanks, Taylor. I appreciate it. Thanks for joining everybody. My name is Tim, and I'm going to be talking a little bit about Immutable Infrastructure in Kubernetes, a little bit of the history of what Immutable Infrastructure is and how it's kind of evolved from the early days. So first of all, a little bit about myself. My name is Tim. I've been working in infrastructure software for, you know, most of my career. I spent some time at a company early on called RPATH, and we were doing some interesting things around kind of automatically generating system images based on applications and their dependencies. And I think that some of that work kind of paved the way for my interest in customized Linux operating systems, containers and virtualization, obviously. I spent some time at a company called Eucalyptus. We were building an on-premise implementation of Amazon's web services. I'm sure some of you remember Eucalyptus and probably kind of open-stacked the spiritual successor, so to speak, of Eucalyptus. From there, I went to Ansible. I was one of the co-founders there at Ansible, and we ended up selling the company to Red Hat. And from there, I co-founded a company called Talos, and I'll talk a little bit more about Talos later on down in the slides.  So that's me. So enough about me. Let's talk about immutable infrastructure. First of all, what is it? Let's take a step back and start with some of the traditional methods of systems management from the 1990s, maybe the early 2000s, and what systems management would look like back then, at least in my experience, that there were some exceptions, I'm sure. But the general rule was you had your identically configured systems running in a data center, sitting in a rack, you had your web servers, your app servers, your load balancers, etc. And these probably would have been configured by hand. Logging in, a system would log in, run commands, update packages, maintain these machines manually, kind of on a per machine basis. Operators would apply security fixes, change configuration, and do what they could to keep these systems consistent across the fleet. Over time, though, this manual approach usually led to divergence between systems. So you'd end up with slightly different packages on different systems. You'd end up with slight configuration differences between systems. And the operators of these machines, they're well-meaning, working to solve problems on a case-by-case basis, fixing each system as necessary to keep it running. And if you're old enough like me, I'm sure that you probably remember operating some of these machines this way. And so the end result of this kind of process was a series of so-called snowflakes. These are artisanally handcrafted systems that were built over time to solve a particular problem. And they were difficult to replicate if something happened, if the disk crashed, or if an operator moved to a different company, it was sometimes rather difficult to get back to a known state and get back to a situation where you felt like you could repeatedly recreate whatever system you had in your data center. So what did that do? That gave you a fairly brittle infrastructure. If something changed, if a system went down, if a disk crashed, if an operator left, you didn't really have a path forward to get to a known good state. So back to immutable infrastructure, what does that exactly mean? My definition is it's a system that does not change once it's been deployed. If you need to patch it, if you need to upgrade it, if you need to change its on disk configuration, you'd shut it down and you'd launch a new one. So you wouldn't make changes to that system midstream. You wouldn't apply security patches on a per package or a per file or a per application basis.  What's the benefit of this model? So I've identified a few here. There's probably others. What it does is it makes a lot of things related to the deployment and management of the system easier. If you're dealing with immutable infrastructure, you can basically eliminate configuration drift. If nobody is modifying these systems as they're running in place, you have a much greater assurance that the system, as it was deployed, is the one that's currently running. Along with that, you gain improved consistency between your development stages. So if you have a strict workflow and a strict process, each step along the way, whether it's your developer on the desktop or laptop, into QA, into production, ideally you'll be dealing with the exact same system image, with the exact same bits, and within reason, the same configuration. What you also get from an immutable system deployment methodology is much easier access to more sophisticated mechanisms like rollbacks, A-B testing of environments. It makes staging upgrades easier. If instead of modifying a system in place, you're just throwing something away. You do need a lot of infrastructure for this to happen. This wasn't really possible back in the 90s and 2000s before virtualization came onto the scene and gave you a lot more agility and flexibility around how you deploy systems, how you throw away systems, and how you launch new ones. But of course, nothing is free. Immutable systems do have their downsides. And in the early days of immutable infrastructure and immutable systems, there were some problems with the approach, some issues and some things that were difficult. Primarily maintaining that golden image was a bit of a chore. You usually had to be dealing with an entire operating system and you had maybe some rudimentary tools to make that happen. The tools that we were building at ARPATH were intending to solve some of these problems. But what you ended up with is a system where the OS and the application were fairly coupled. They were tightly coupled. They were inextricably linked. If you're deploying these to a virtualization platform as opposed to a container platform, your application images had to basically contain the entire operating system and everything else involved. I wanted to talk a little bit more about the history of immutable environments and immutable systems. The first hint of immutable systems in my mind came from Red Hat about 13 years ago in 2006. Red Hat announced a project called Stateless Linux. Stateless Linux was intended to solve the so-called snowflake problem by using an immutable read-only file system for the root partition of each class of machines. By each class, I mean web servers, database servers, app servers, etc. Red Hat developed a whole methodology and put some engineering work into making this happen. When you needed to make a change to the system's configuration or upgrade a package, that change would be applied to the golden image and it would be rolled out via different mechanisms to the appropriate machines. This ended up eliminating the snowflake problem and it ensured that the configuration and versions of the underlying software did not diverge between systems, between deployments, between dev test and production, etc. So for various reasons, and I'm not sure of all of the reasons, Stateless Linux's specific implementation of immutable infrastructure didn't really catch on. Instead, I think that the industry kind of moved more towards tools like PuppetChef and Ansible. Things that allowed you to automate the process of logging in and applying security updates and making configuration changes in a declarative way or even in an imperative way. So you ended up with the same result. You had a common state deployed across your infrastructure, your configuration was consistent across your fleet of systems, using a different technique, using playbooks and manifests and so on to describe the state of your system and then there was a tool that went out there and made it so. And the benefits of either platform here are pretty clear. You can eliminate that configuration drift, you end up with more similarities between test and dev, and so on. So I like to look at the PuppetChef and Ansible style tools as kind of a diversion that the industry took on the way to getting to real immutable infrastructure. When Amazon rolled out their EC2 elastic compute platform and as virtualization kind of gained favor and gained performance and began to be deployed in people's infrastructure, those advances kind of made the idea of a stateless system more approachable. And I think that Netflix really kind of helped popularize this approach as they really went all in on the Amazon cloud platform. They popularized the concept of the blue-green deployment where this methodology involves deploying an entirely new group of servers, which in a pre-cloud environment would be very difficult and very expensive. You basically have to double your data center. But since you're paying by the hour, paying by the machine in the cloud, you could do that and you could stand up this entirely new group of green servers and instead of logging into the live, the blue servers and updating them, you'd stand up your green system, you'd test your green system, you'd make sure it's good to go, and then you'd flip your load balancer over to talk to the new servers. This means that you could do a lot of upfront testing and validation against the green stack without impacting users. And then once that testing and validation passed, the traffic would be shifted to the green stack and the blue stack would be decommissioned. So using these tools, using these concepts, a series of golden images that represents your applications and your components, Netflix and others got quite a bit closer to the goal of having these systems be fully immutable and see all the benefits provided by that. There still were some downsides, as I mentioned in a previous slide. Building the golden images was a difficult process. Depending on the particular cloud you were working on, the process for getting those new images, those AMIs up to the cloud platform involved quite a bit of work. It was slow. You're pushing hundreds of megabytes or maybe gigabytes up to the cloud over your 2007 internet connection, et cetera. The other thing that you had in this situation was that the application and the operating system were still tightly linked. You couldn't really separate those two parts if you needed to move one piece forward or keep one piece back. You had to worry about the dependency closures and so on. So there were still some challenges. A big organization with a lot of resources like Netflix could certainly take advantage of these concepts, but it might have been more challenging for a smaller organization with less resources to really see those kind of benefits. Moving ahead kind of in the timeline, after Amazon EC2 came out and some of these new concepts were being deployed, tools like Packer from Hashicorp came on the scene and Docker as well. Both of these systems had ways to generate these golden images based on a descriptor. In the Docker case, of course, it would be a Docker file. Packer had a mechanism to assemble these systems and output them in a certain format. So we started to see some tooling emerge that made that image build process a little bit less troublesome. We had improvements in the way that these improvements in the cloud APIs to receive these new images and it became easier to iterate, to update. It became easier to track changes and update existing platforms. All of this is kind of at the beginning of containers taking over the industry a little bit. And you still had some challenges in terms of, you know, how do you run these containers? Who is responsible for making sure that the right containers are running at the right time? Where are they running in your infrastructure? What happens if they go away? What happens if a node crashes? What happens if you lose access to one of your systems?  And that's where Kubernetes began as this container orchestration platform to solve some of these problems to again kind of shift the burden away from the manual step-by-step process to handle some of this stuff and move it more to a declarative, more to a, you know, desired state configuration. So Kubernetes really made this the container workflow possible. It began to decouple the operating system from the applications. Containers allowed you to really pare down what goes into your systems and decouple the application components from the operating system components. It took the burden off of the administrator to decide where and how these application containers could run. Things like storage abstractions and network abstractions made it easier to begin to separate the ephemeral storage that you might need for, you know, local state for a particular application and persistent storage, which is, you know, of course, the storage that holds your application data and your user data and database storage and so on. So going back to the Ansible Puppet and Chef style tools, as containers and Kubernetes and Dockerfiles and so on, as these things gained prominence in the industry, I think that those, the imperatives and declarative configuration management tools like Ansible Puppet, Chef and so on, I think they become less important because you're doing less work on the live running systems and you're doing, you're kind of pushing the, you're pushing the changes earlier and earlier in the development cycle. So more of the, more of that change and more of that image churn, those cycles, updates and so on are happening earlier in the development cycle. And of course, that also began to continue this decoupling of app and OS. So I think kind of, you know, talking about today, we've got, we've got Kubernetes, we've got container image build tools, we've got a lot of interesting componentry out there at the application layer and I'm sure a lot of you, if you're using Kubernetes, if you're deploying your applications on Kubernetes, you've seen and taken advantage of some of the concepts of immutable systems for your applications and hopefully you've seen some advantages in terms of agility and, you know, time to deploy and uptime and so forth. What I think hasn't happened very much are changes at the host level, right? So the machines that run Kubernetes, the machines that run your container orchestration tools, what if you could apply those same immutable immutability concepts to your Kubernetes host environment? And, and I think you would begin to see some of those same advantages. You would see more stability, you'd see less sort of, you know, handcrafted hand-configured systems and you'd be able to upgrade faster, you'd be able to roll back if you needed to. So a lot of this stuff in terms of running your Kubernetes infrastructure entirely in an immutable way, that's been one of the design goals for Talos, the company that I'm involved with. It's a new operating system designed specifically to host Kubernetes clusters and to be a container host. I'll talk a little bit about Talos and then I'll talk about kind of how these concepts of immutability apply to your Kubernetes environment. So one of the design goals of Talos has certainly been to be immutable and for these systems to be immutable and ephemeral, meaning you don't change them once they're launched and you can throw them away at any time. So Talos is an open source operating system based on Linux. We've been in development for just about three years and last year we launched a company to fund its development and to build out services and support organization around this new method of hosting Kubernetes. Talos is very, you know, highly inspired by CoreOS and a couple of other platforms that have come before and I'll just talk a little bit about the architecture of Talos. Again, it's an open source OS based on Linux. You can see that the Linux kernel is at the base of each of these components and the goal has really been to build an operating system that makes sense in a modern distributed environment. So we've made a number of design decisions in that direction and some of them I feel like some of them are a little bit radical, right? So we have actually removed console access. There's no SSH. There's no shell in the system. Instead everything is managed via an API. So as you can see in the diagram we've got the OS CTL command line tool which communicates with the API. You can also communicate with it directly with the API and we've tried to, you know, build a solid and modern API to do these OS management tasks that you need to do. A little bit more about Talos' architecture. Just like Kubernetes, we've got it split up into control planes and workers. So Linux, Talos, and the kubelet run on all of those. The system has been designed from the beginning to be immutable so we actually launch into a squash FS root file system. It runs out of RAM and it never touches disk. Kubernetes does need some ephemeral storage so we do have a provision for that. But what this means is that even a dedicated attacker, from a security perspective, even a dedicated attacker who manages to access the system, they can't change that root file system. They can't log in. They can't remount it with write access, et cetera. Along with this, the fact that there's no shell, no SSH server, no other way to obtain console access to these Talos hosts means that the system is very difficult to modify while running. I would imagine that if you have access to the data center itself, if you could roll the crash card up to these systems, I'm sure you could find a way. But we think that we've added a few layers of safety and security there from an OS perspective. So the immutability of the system gives you some security benefits and as we talked about earlier, you also get the benefits that you know that the configuration is what you tested with. You know that there's going to be no unexpected changes from either well-intended operators who might make a mistake or might make a change and not tell the rest of the team. You're also more secure against nefarious intruders who might exploit a security problem in various ways. So I could talk more about Talos and I'm happy to answer questions there. I'll leave some of that for the Q&A afterwards. But I did want to kind of walk through the upgrade process, kind of an illustration of one of the important workflows related to an immutable system. How do you update it? How do you upgrade it? How do you change it when it does need to be changed? So this is just one example. This is a workflow that we have implemented as a Kubernetes controller as an operator underneath the hood. So this is automated. I think you could probably take a similar approach to handle your applications. There's a lot of good examples out there on the internet. But let me just walk through this little workflow here and talk about what it takes for Talos to upgrade a node. Again, it's similar to how you'd upgrade your applications. But this is for the host environment that's running Kubernetes. This process is the same no matter where you're running Talos and Kubernetes, whether you're on cloud, virtualization, bare metal. It's all handled automatically by our upgrade controller. And it starts with someone or something making an API request to perform the upgrade so that API call comes in. We cordon and we drain the node of requests. If you're familiar with load balancers and how that flow works, this should be familiar. We stop any new requests coming in. We do what we can to get rid of any existing requests, wait for them to finish. Once that's done, once we're cordoned and drained, we'll stop the cooblet and remove all the pods on that system. We'll verify the upgrade path. We'll do some work within etcd to make sure that all the members are set up and started appropriately. And then for this particular node that we're upgrading, we'll unmount the ephemeral disk, we'll reset the partition table, and run the installer. In the Talos example, the install is pretty quick. It's a fairly simple setup. I don't have a lot of technical detail in here. But if you'd like to know more, feel free to join in Slack afterwards, or you could throw a question into the Q&A and I'll do my best to answer. Once the install is complete, we'll reboot the node and we'll verify its health, we'll uncoordinate, and we will bring that machine back up and it's hopefully ready for workloads. So that's kind of a general upgrade path that applies to Talos and applies to Kubernetes. I think it could also apply to your applications and it might actually be quite a bit simpler for your apps. If you look at the history of immutable infrastructure, the styles and technologies, and fashion changes throughout the years. I like to think of each of these iterations and evolutions in the way that people manage systems. I like to look at it as incremental progress. We're all trying to make computing a little bit safer and a little bit faster and a little bit stronger, more resilient to problems, more resilient to attackers and improve the security. And so I feel like every little thing that you do to get a little bit closer to a better system is worthwhile. So I'm hoping that by using the concepts of immutable infrastructure and immutable systems for your container hosts, for your Kubernetes hosts, I'm hoping that we can just make computing a little bit safer, a little bit faster, and a little bit more secure. So I'd be happy to run over to Q&A now. If you'd like to learn more about either Talos or what we're doing as it relates to Kubernetes, maybe start with the last link on this slide, our documentation page will walk you through it. You can see the source code on GitHub, and we would love to have you join our Slack if you'd like to have a more in-depth discussion. We've got the creators of the system and our engineers there, and we'd be happy to talk. The project has been around for about two and a half years, as I said. We would love to have more contributors, more users poking at the system and finding ways to improve it. So hope to see you there. I will stop here and I will switch over to Q&A if I can get this panel to show up, and it looks like it's not actually going to show up. So Taylor, I wonder if you wouldn't mind reading me off a question or two, and if I can answer it, I will. If not, I will take a pass. Of course. Okay, so the first question is from Samuel. Is there already a plan to have TALOS OS working out of the box would be awesome with other CNCF projects like Q-Router and Rook? Sure. Good question. Yeah. So I don't know the specific technical details of how to connect those other CNCF projects to TALOS, but what I can tell you is that our architectural approach is such that we don't want to reinvent the wheel if we can avoid it. So if there's a good storage platform or if there's good networking platforms out there, we will work and integrate with those. We do have plans to put together a simple plugin mechanism where you'll be able to kind of have these things out of the box if you so choose.  Okay. Next question. How is TALOS OS Linux different with Core OS? Sure. Good question. I think you'll see a lot of similarities. We were heavily inspired by Core OS. We thought that Core OS was moving in the right direction. I think what you'll see in terms of differences is we've gone further sort of in a more radical direction than Core OS did. So we're not based on any Linux distribution out there. We're quite a bit more minimal than Core OS. We've removed SSH. We've removed console access. I believe that Core OS was a little bit more of a general purpose operating system. It was suitable for application hosting as well. Today, we're very strictly focused on the host, on the host environment. And so everything that we do in the system is going to be oriented around making your Kubernetes host deployment easier and safer and faster. Okay. The next question is from David. Is TALOS HA, can you run more than one control plane? Let's see. I believe the answer is yes, but if you want a definitive answer from our engineers, let's see. I've got a live update right from Andrew here. Yes. You can, I think you can run as many, oh, hey, I see the questions and answers now. Excellent. Okay. Yes. You can run those in an HA environment. And I believe we've kind of, we've done our very best not to impose any additional architectural restrictions on your TALOS-based environment. So each Kubernetes master is a TALOS master and you can have multiple ones. All right. Next question. Can you mix TALOS hosts and non-TALOS hosts in the Kubernetes cluster? Oh boy. Let's see. I'm probably going to have to wait. Okay. I'm getting yeses from the engineering team. Yes. I, my understanding is that as long as the, as long as those two, as long as your control plane and the non-TALOS hosts are compatible version-wise, et cetera, and you've got the right bootstrap tokens and so on, yes you can. Whether you'd want to in a, you know, long-term, I'm not sure, but yes. Okay. Questions. Just keep coming in. Next one from Dimitri. First impressions of AWS Bottle Rocket OS. Yeah. Good question. Yeah. So we, we, we had, we heard a rumor that Amazon was building something similar to TALOS and sure enough they are. There, there are definitely some technical similarities. I think that, I think that Bottle Rocket was, was somewhat inspired, the design of Bottle Rocket was somewhat inspired by the design of TALOS.  We're, we're glad that, that Amazon is building something similar to TALOS because it kind of, you know, it justifies our existence and it justifies the work that we're doing. We like a lot of the technical aspects of the system. I think that, you know, I think that technologically, you know, we're, we're a little bit further along in terms of our development. And I know that Amazon is kind of, you know, they're naturally, they're focused on the Amazon componentry. We are, we are very strictly cloud agnostic and we have support for as many, as many cloud platforms and as many other, other sorts of deployment infrastructure as, as we can. But yeah, we're looking forward to, to seeing what the, what the Amazon folks are doing and if they're, if they're good ideas there, hopefully we can share. Okay, next question. What is the release cycle of TALOS? How far is it behind Kubernetes upstream? Yeah, great. So we've been, we've made a commitment to, you know, kind of track upstream Kubernetes as closely as possible. So, you know, as opposed to some of the other more complicated platforms like OpenShift, we are able to, we're able to be more responsive to the Kubernetes releases. Previously, we had, we had our release cycle was kind of, you know, kind of mirrored, it mirrored the Kubernetes release cycle. So every three months we'd release something, it would have the latest version of Kubernetes there. We're in process of kind of changing that deployment model and moving to kind of a faster cycle. But I think we're, we'll still maintain that commitment to shipping within reason, the latest version of Kubernetes that we can just, you know, it's a fast-moving project and, you know, our approach might change in two years when things are more, things are more mature and further along. But for the time being, we're gonna, we're making a commitment to be as quick as we can to release after a Kubernetes release. Okay, next question is when you coordinate a node, the Kubernetes service might still route the traffic to that node and thus you the client can experience some percentage of errors. How do you avoid that? Let's see, that would be a great question to ask our Slack channel. I see that Andrew is piping a bit of a response. So I'll see if we can get something quick. But if you'd like a, if you'd like a longer response, please drop into the, into the Slack and ask us. I, the, the architecture there is a little bit, a little bit above my, above my pay grade, so to speak. So I'm, I'm sure there's a good answer, but I'll, I'll wait for, I'll wait for the team to chime in. I think it's probably not going to be specific to Talos though. So any solution that you can find in kind of the, the broader Kubernetes ecosystem, you, you could apply to a, a Talos-based system. If we move on for a second, come back to that one. Are you? Oh, I think, yeah, I think that's the best I can do. Okay, let's do that. Thank you so much. If you need to, if you need further answers, feel free to drop in and, and ping us on Slack. Okay. So next one is clarifying that you have nothing to do with Oracle Talos, single point of talent. That's right. That's right. The namespace is crowded. We have not received any cease and desists from, from Oracle or Cisco or any of these other folks. And we're hoping to keep it that way, but there are only so many names out there. So that's where we're at, but no relation. All right. How mature is Talos to run production services in particular on bare metal? Sure. Yeah. So Talos is, as the operating system itself, you know, we've been around for, you know, two and a half, three years. We have a, we have a handful of folks who have, who have talked, you know, community members who have, who have been working with us for a while and they are running in production. It's a, you know, the operating system itself is, is fairly simple. There aren't that many moving parts and pieces. The, the code is all pretty, pretty straightforward, relatively speaking. So we feel as though Talos is, is ready for production. In terms of bare metal, I would say stay tuned. We're actually going to release something related to Talos and bare metal very soon that I think will, will be interesting. So, so stick around and watch for that release and hopefully it'll be useful for you. Okay. Is Talos available through any public cloud providers? Yeah, definitely. So the cloud providers were really kind of our first target. So we have support and some documentation for Amazon, for Azure, for, for Google's cloud platform. So, yeah, all of, all of the major cloud platforms are provided and we publish assets for, for every provider on every Talos release. So, yes. Okay. Any plans to maintain a Terraform provider for Talos? Yes, we would love that. It's been on our roadmap. It's one of the things, you know, one of the, one of the things that we know that we need to do is build integrations with existing system management tools, right? So maybe an Ansible module, maybe a Terraform provider. We, we have not ourselves had the, had the resources to work on that. But, you know, in terms of, you know, community contribution, if someone were to come to us and say they wanted to, to work on that, we would be happy to help and we can provide some, some guidance and some assistance. And I think that would be really valuable. So, Lucas, yes, if you're interested in, in, in this, in more detail, if you want to, you know, give us some opinions, if you want to maybe, maybe sketch something out, we'd be happy to hear from you. Okay, next from David. Can you run Windows containers workloads on Talos? Good question. I am not sure. I don't think, I don't think we've tested that. I think, you know, if, if you can run, if you can, if you could run these workloads in an, in an ordinary vanilla Kubernetes environment, you almost certainly could run them on, on Talos, you know, whatever Kubernetes and container D supports will support. But we have not, you know, it's not part of our test matrix. And it's not something we're targeting now. But if there's, you know, if there's community or customer interest, we'll, we'll make it happen. Okay. Someone's from PSI, from resiliency perspective. Talos provides, does Talos provide Bosch? How is it defined at Cloud Foundry? Yeah, if I understand the question correctly, I think that, that Bosch is a, is a deployment methodology for, for Kubernetes, if I remember right. I'm not sure, I'm not sure, Si, if I understand your question fully, but if you'd like us to, to dig in, you know, please feel free to send us an email or join our Slack and we'd be happy to talk. I haven't done a head-to-head comparison or anything like that, but we'd be happy to take a look. Right. Thanks, Si. And then what are typical approaches to investigating and responding to a security compromise without a shell access? Yeah, great question. So a couple different things there. This is something that we've talked a lot about internally, you know, how do we, how do we take advantage of, of kind of our unique architecture in that we don't have SSH, don't have console access. There's, there's a few different angles there. So, so basically when you're troubleshooting a Talos-based environment, whether it's responding to a security compromise or, you know, dealing with some other problem in, in the system, our API has been designed to replace the, the process of logging in and checking log files and, and, you know, running process lists and so on. So the, the OSCTL command line tool, which talks to the API, handles all that for you. So you can fetch log files, you can look at process lists, you can, you can, you know, list files, you can dump, dump files from the file system. You can do all that analysis work in, in a slightly different way. You can also stand up an administrative container alongside. Right. So I think that the Kubernetes world kind of calls these sidecars. So you might stand up an SSH sidecar to really get in there and take a look at things. We think that our architecture gives us some interesting opportunities to build tools for this sort of thing. So perhaps, you know, if we detect a security compromise or if the administrator sees something fishy, in the future, we hope we plan on building some tooling around that. So you'll be able to say, oops, stop this container, pause it, freeze it, move it to Amazon S3 or download the, the container image to your workstation, immediately take it out of rotation.  So yeah, I think, you know, we've talked a lot about this and we think that there's some really interesting capabilities that would be possible in this kind of architecture. So stay tuned and if you're interested in that sort of thing, we'd love your input. We do a couple things out of the box. We've begun working with the Linux IMA, Integrity Measurement Architecture, which actually, at a kernel level, watches for changes to files and can, you know, send some signal somewhere in case there's a suspicious activity. So we hope to expand the use of that and, and build out some tooling for, for that exact purpose, for that kind of, you know, forensics approach. Okay, next question from Lucas. I've had issues with minimal container-based OSes like Container Linux and Rancher OS, not supporting some low-level components slash drivers for things like NFS mounts. Are you aware of any similar limitations in Talos? Yeah, great question. I am sure that there are similar limitations in Talos, you know, here and there. We, we build our own kernel kind of out of the box, you know, we don't enable, we kind of, you know, we've chosen to minimize the number of kernel modules that we build in entirely. We do have some tooling so you can build your own kernel with your own configuration if you need to, to enable certain drivers or certain kernel features. We do have users who are using NFS and, and I think that that particular limitation is probably doesn't apply to Talos.  And we do have a path to build your own custom kernel and, and include those, include that driver support. Okay, we don't have any other open questions at the moment. Maybe we can give it a minute, see if anything else comes in before we close up. If you have any questions, anybody please enter in an Q&A right now. Yeah, and if you think of something after the fact, feel free to, you know, shoot me an email, tim at talos-systems.com. We'd be happy to have you join our Slack. We do, we do host a weekly community meeting that you can join kind of, you know, office hours with the team. Be happy to, happy to have a live discussion either, you know, on the, on the video call or just on Slack whenever we're, we're always there, always, always, always eager to talk to new users and people who are interested. Okay, here's another question from Axl. How about these drivers? What about GPU support on bare metal? That might be challenging, right? Yeah, yeah. So we've had a few people ask about this. We've done a little bit of investigative work to see what it would take. I think, you know, in general, the previous answer to the, you know, the kernel driver question applies. If you can, if you can build your own kernel with the options that you need enabled and set up the pass-throughs and so on within the Kubernetes and the container D infrastructure, then you're good to go. It's not something that's in our test matrix today, but I think it, you know, I think it will, it will be there at some point. Okay, well, we don't have any new questions. Right, Tim. I think that about wraps things up unless anything comes in the next 20 seconds. But thank you for a great presentation. There were such good engagement questions from the, from the crowd. Thanks so much everybody. Just like to thank everybody for joining us today and let you know that the webinar recording and the slides will be available later today at cncf.io slash webinars. So keep an eye out for that. And we look forward to seeing you all at a future CNCF webinar. Thanks everybody. Thanks, Tim. Thanks, Taylor. Thanks, everyone. Hope to see you around.


Justice Minister Abubakar Malami has directed for the immediate prosecution of offenders of the federal government's ban on Twitter operations in Nigeria. The other is contained in a statement issued by Dr. Omar Gwando, Malami's special assistant on media and public relations on Saturday in Abuja. Malami directed the director of public prosecution of the Federation of the Office of their Tony Generation, a Tony General, rather to swing interaction and commence in earnest the process of prosecution of violators of the federal government deactivation of operations of Twitter in Nigeria. He also directed the DPPF to liaise with the Minister of Communication and Digital Economy, the National Communication Commission, NCC and other relevant government agencies to ensure the speedy prosecution of offenders without any further delay. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.