I think we should start. All right. So, Huma Key is a developer advocate at Livepan, and today he will be talking about event-driven services as neural-synaptic services. Please go ahead. Good morning, everyone. So for, I'm a speaker, I speak at conferences a lot, and I've been talking about event-driven systems for some years now, because I really think they're very, very interesting. And I've been building systems using it as a developer advocate, so I've got decades of experiences at backend, like Java developer backend systems, so now I get to build systems for fun and demos and things like that. But I've been fascinated with event-driven, and over the last year, I've been using something else, and the hint is, it's called this, Kalex, but that's kind of changed my perspective on how to do software, and especially how to do event-driven software. So I think event-driven software is very, very interesting, and this approach that I'm gonna kind of show you as well I think is interesting also. So maybe you'll think I'm crazy, some of my colleagues think I'm crazy because I'm talking about this neural processing with event-driven, but we'll see, you'll be the judge. So what I'm gonna do is, I'll do review event-driven because I think a lot of people have heard about it if you're a backend developer or even full stack developer, but many people haven't used it.  I've been playing with a name, Neurosynaptic Services and Micromind, those types of things, but it's an interesting processing pattern. So with event-driven, one of the things I've been working on is a demo that's been a lot of fun for me, and it's called Ership. And the reason it's called Ership is that the user interface that I built is a map. It's like a Google map, so you can go anywhere on the earth and you can zoom in on it to a location on the map and then you can pick a circular region like you can see the circular regions here and then you can say, I wanna create a simulated number of orders. So this is like an e-commerce system, this is like the shopping cart demo is kind of like the hello world for backend applications. So I started with this simple shopping cart app but that got boring really, really fast, so I wanted to make this more interesting and I wanted to do more interesting things with the demo app but I always like to do this visual stuff so that you can, and this is on GitHub that I got QR codes where you can go grab the repo, but the idea is you zoom into a location on map, anywhere on the map you want, you could be in the middle of the ocean or it could be over a city or wherever you wanna do it and then you can specify how many orders do you wanna create, you wanna create some like a couple hundred orders, you wanna create a thousand orders, but what I wanted to do is kind of have this demo put a hurt on the backend system, already put a load on this system and another thing you could say is a rate, like orders per second type of a rate then one click goes in to this demo app and it starts generating a bunch of orders. So all these dots represent orders that were randomly distributed within the circular region. So this is a design diagram of this application, I'm not gonna be able to go into a lot of detail with it but I want it, it is event-driven. So these rectangular boxes represent basically event-driven microservices, but they're of a very specific type, they're probably not like maybe what you've seen with other event-driven before because they're very focused on what they do, but what I wanted to do was show kind of visually what happens. So you can see the bottom left, there's a client, I'm kind of highlighting a client and it's sending in a command to a shopping cart service and it's adding say items to a shopping cart so it's loading up a shopping cart but the interesting thing is when the command comes in from the client to say check out. So the service is emitting an event, check out. Now this is where things get interesting, that event is of interest downstream to another service which is to create an order, so that triggers the creation of an order. So this is event-driven flow that there's kind of a cascading sequence of events. Like I say, this got boring really fast so I wanted to plug in this UI. So the map in the top center there is sending in a single HTTP request to say, I create 500 orders at this geographic location within this circular region. So that goes into another service which starts triggering the creation of these orders. But you can see there's a flow like this generator got the initial request, it starts emitting events over and over to create these what I call geo-orders and these geo-orders are just orders at a specific geographic location. But here you can see the event coming out of geo-order is of interest to a couple other services and the point I want to make here is that each of these services, one of the things we've been talking about with microservices forever is they're supposed to be loosely coupled and these are loosely coupled services. These services have an API but they don't know who gave it send of commands. These services also emit events but they don't know who or what consumes those events. They don't care. So every single service is only concerned with its own functionings, not concerned with the rest of the system. So this geo-order service emits an event which happens to be of interest because of the wiring of the system to a couple of other services. But now we go deeper into the processing flow here and I'm not going to go through what's happening here but other than what this system does is allocate stock to the order. So it's doing something a little bit more interesting than just taking orders in. But you can see there's this kind of cascading flow of events that are coming in. I'll get rid of this thing here and what it is. Where the processing is occurring. So all these highlights are these cascading flows where commands come into one service, it emits an event, that event gets picked up by another one. So the logic of this system is built in, it's kind of decomposed down into these individual very focused little services that are wired together to orchestrate the whole processing of the system where it takes orders in, allocates stock to them and reports back to what the processing is. So this is the general flow of this system. This is a really different way to design a system. If you, like I said, I've built back in systems for decades and with regular databases and cred-like operations, multi-table transactions and all those types of things, this is not that type of behavior. So the design of this application was really interesting where I decompose a problem down into each of these services is solving a different part of the problem and it has this processing flow and including things like allocating stock to the order. So that's kind of what this demo does. It's a fun demo to run visually on the map when the orders are first shown, they're grayed out indicating, okay, the order's there but we haven't processed it yet and then just over a period of a few seconds when it runs, the orders will turn blue when they're allocated stock or they'll turn red when there's insufficient stock so the order's going to back order state but that triggers other behavior that's in this design which will order more stock and as that stock comes into the system that stock goes looking for back orders and fulfills those back orders so that all the orders get processed. Now, the other interesting thing is that there's three fundamental components that are used to build this system in this platform. There's the rectangular services, they're called entities and like I say, they have an API. It's a simple API. I'll show you the code in just a minute. They have logic. Basically, the service composes the state of a thing. It's like this shipping order item for example that's in the middle of this diagram. It's just a durable object basically that this thing is processing and then it's submitting events. Now as events driven, the events are for changing the state. Like with a shopping cart, you add an item to the cart. That's the command coming in. If the cart's not checked out, you run the business logic, if the cart's not checked out and it's a real product and everything looks good then the item gets added to the cart which means you're altering the state of the shopping cart by emitting an event. So the processing is a command comes in, it creates an event, the event gets persisted and once it's safely persisted then the event is used to alter the state of the object and that's the processing flow. So the big difference here is that these services are really simple, they don't reach out, they don't talk to databases directly. The state is recovered from the event journal. So like what is the current state of a shopping cart? Well it's got add item three times, change an item quantity, remove an item quantity, check out. The aggregate of all those events is the current state of the shopping cart. This is the basic behavior of these services. So let me show you a little bit of code. Here's the shopping cart service, it's in Java. Anybody Java spring developers here? So some of this should look familiar. The API is definitely using spring like annotations but you can see that on this line here, number 26, I just have a class and it extends a base class, it's called an event source entity, this is part of the software development kit here, this K-Lex software development kit. But then here you can see, here's how you define part of the API, like add an item to the cart. But the main thing I wanna show you here because there's a lot happening right on this line 49, you can see there's above, it's like effects dot emit event. So what that is, that's the code that emits a new event that needs to be persisted by the underlying system. So this is a high abstraction, I just gave a talk earlier this week at another conference about high abstraction platforms and K-Lex is one of them but I talked about four different ones. These are higher abstraction platforms that are abstracting away as much detail as possible. So this code does not connect to a database directly, it's connected to a database indirectly through the layer, the platform layer that's below it. So the goal is the simplified development. So here it's like, admit an event. I don't know where it goes, I don't know what the database is, I don't care, that's the problem of the platform. My problem is worrying about the business logic and the service. So it emits an event. The next interesting thing is this current state. The current state is an inherited method from the base class which composes, has the state of this thing, like in this case a shopping cart, as it is right now, retrieved from the database. This is also a stateful service, meaning that the state will be cached, kind of a very elegant caching mechanism where if the shopping cart's active and there's a user adding things to the cart, that cart's in memory somewhere in a cluster. There's one instance of it in memory somewhere in a cluster and instead of having to retrieve the state every single time you're gonna make a change, it has it in there. But all that's being managed by the platform as well. You access in our code the state from this current state. And then this event for is a method I wrote, it's just my style, but it's really simple. It's just taking the command in and creating the event back and returning that. So these APIs, you can see the API, like add an item, change an item, remove an item, check out. This is all I'm doing. And then the next thing here is these event handlers. So the event handlers are invoked once the event has been safely persisted to the database and we wanna now make the state change. So if you look at this method, you can see that it's returning the state. So that's the state of the shopping cart. Altered by the event. So like with the add item, I'll take a go into this on method. You can see this is just basic Java. All I'm, you know, the object here has a list of items in the shopping cart. So all I'm doing is streaming through the list and adding the new item to the shopping cart. Very simple, no database, nothing like that. So there's all these event handlers for the different events. And then the state itself, it's just a Java record. And I really love Java records. It's one of the best inventions come out on Java because they're immutable. So that means that I can't write code where I'm gonna screw up and change the state when I shouldn't have or something like that. So it's really quite nice. But I have these event, just kind of overloaded event for methods for all my different commands. And then I have the on methods, overloaded on methods for all the different events. And that's the basic flow of this class. So no database connections, no worrying about publishing messages, things like that. But you can see that there's messaging happening in the system because the event from a shopping cart, like a checkout, is being picked up and processed by downstream services. So the other component that I wanted to show you is the shopping cart to order. And here I just wanna show you really quickly so I can move on. This is called an action. And actions process the messages from upstream events. So the event comes into the action and the action translates, say, a checkout into a create order command. And it sends that create order command to the downstream service, the order service to create an order from the shopping cart. So the logic in here is really simple. I'm not involved in pulling or anything like that or talking to Kafka or whatever, high abstraction type of code to do that. So that's kind of the code. So I wanna get into the micromind piece here.  And when I was working on this design diagram and my goal for that design diagram where I showing kind of the cascading sequences of events but to help people visualize a flow and kind of a real event driven type of a system. And I thought they worked out pretty well but it took a lot of work. If you guys ever heard of Blender, Blender.org is a fantastic 3D tool and I love Blender and I'm gonna show you something else that I've done in Blender in just a minute that I love. But so this diagram I did in Blender but it was a lot of work to do all this animation of highlighting each step of the processing flow. But it's like when I was looking at it, it's like, okay, wait a minute. The way these services are working and the way there's this kind of cascading sequence of events that's occurring between the services, I started to think, wait, this is somewhat like the behavior of the neurons in biological brains. Neurons in biological brains have a state, they have like a charge. They get signals in from other neurons or other sensors like touch or pain or whatever. But the neuron itself doesn't know who sent it a signal, it knows it has received a signal. The neuron decides should it admit a signal. So it decides it admits a signal. It doesn't know or care who is wired up to receive that signal. All the neuron that knows is basic mechanics. That's kind of what these services are doing here. Really simple, really focused, not complicated, not doing database transactions, it's kind of a pure kind of adventure type of thing where signals come in and signals go out and there's some logic in the middle to decide when, you know, how to respond to incoming signals and how to emit outgoing signals. So that kind of blew my mind because, you know, like I said, the shipping order, it gets a command in, does a state chain, emits an event, that event gets picked up by an action because it's subscribed to it, that action takes that event and translates it into a command and sends it down to another service. There's your flow. So it kind of blew me away. So again, just kind of walking through it really quick, you can see this cascading sequence of events, you know, step by step, this is the exact flow of the way this application works. And what it's doing, it's not just taking in orders, like I said, it's allocating stock and what the system is doing with just these 13 simple services, it remembers exactly where every physical unit of stock went, to every single order, all that is being handled. It's also handling back orders, like when there's insufficient stock in the system and when stock comes into the system, it has logic to add stock to the back order orders, find them and allocate stock to them. All this behavior is in relatively simple service. So that's what's kind of cool here. Again, my colleagues think I'm not talking about this whole neural thing, but I don't know, I'm pretty psyched about it. The other thing that I thought was really interesting was that there were behaviors that were starting to emerge, recurring behaviors that I hadn't seen before in other code. So for example, there's like five or six recurring behaviors, I'll give you a summary at the end, but of the 13 services here, three of them kind of use the same kind of a hierarchical tree pattern, I call it a reduction tree. And once I figured out how to use a reduction tree in this kind of a processing, and this isn't unique to Calix, I think is unique to the style of doing event-driven, that these patterns started to emerge. There's all these different reusable patterns that are starting to emerge here. And that intrigued me even more that maybe there's something really interesting happening here, because I'm starting to think that the way our brains, you know, biological neural networks are put together is the same type of thing. There's these recurring patterns in the way all the things are wired together that are being used to do very, very powerful things. So it's this whole idea of, you know, services behaving more like neurons in other functions and behaving like synapses. By the way, this is all serverless as well on this platform. So these things are running in a cluster, they're getting allocated when needed, it scales up, it scales down, those types of things. So what I wanna show you is a another visualization of processing the services. So what I did was, before I play, cause I got time, I wrote specific log statements in the application so that I could scrape those log, specific log statements out of the log and use that as raw data to render the activity within the ERSHIP demo as it was processing orders. So what happens here is that that's one shopping cart getting processed and a stock getting allocated in the system. And then the two other shopping carts come in. But the fun part starts here, like, there it goes. Now 200 orders are coming in from the map. And what's happening here, every dot represents an instance of one of the services getting created. And the colors represent state changes that are occurring. So red indicates back orders. And yellow indicates things that haven't been processed yet. So it's processing and green indicates that, yeah, the stock's been allocated to the order and it's flowing back up. So you can see it's fighting, it's actually recognizing that the system's out of stock that triggers stock getting ordered coming into the system and getting processed. And you can see things turn from yellow to red to green and everything gets processed. And the lines that we're flashing represents the specific messages, like a vent came from a specific entity to another specific entity. So these lines aren't arbitrary, this is real data from the running of the application. And it's 3D, so this is the beauty of Blender. I wrote this code, by the way, it's Python, Blender has a massive Python API, but wanting to do this for years, I've been using Blender for a long time. But I used chatGPT to help me write the Python code to do this. So this was a blast. I was so happy when this video worked because what I think is really interesting here is I don't know if, if you watch videos on say the processing of neural networks or the processing, some videos are on YouTube, say like of insect brains or human brain activity and all those types of things, really awesome looking videos. And I wanted to have a video like this, but for a software system. So this is about 10,000 entities were created, about 15,000 events happened as processing all these 200 orders. And then I already know some optimizations that I can do to this to cut things down some more, but the other thing is that this system is, it's built to run in a fully distributed environment. Scale up, scale down, any part of this application can break and when it comes back up, the system picks up where it left off. It will not, it will keep running until it's finished its processing. Unlike AI systems, like large language models, which have you heard about it, it'll lie to you with confidence or that they hallucinate like chat, GPT and everything. I've seen it do it. You know, you ask it a question and it says, oh yeah, here's the answer. You look at it and go, no, that's not the answer.  The, this system is not trained. It's hardwired. Is it like it's got instinctive behavior, not trained behavior. So it's enterprise precise. You know, like we're using these patterns to do things like financial processing. With financial processing, it's like you don't go back to your management and to your business sponsors and say, oh, it works most of the time. Everyone's a while, we'll screw something up a little bit. No, no, that doesn't fly, right? So this is absolutely precise type of processing. And for example, decrementing stock. This knows exactly what's happening with stock. It won't make a mistake with stock. It won't make a mistake, say, aggregating financial transactions into merchant payments. That's another system that we worked on recently. Those types of things. So it's fully distributed, fully scalable, fully resilient and absolutely accurate. And the nice thing is, is that I write this code and I'm really kind of focused on the pattern of the code, not worried about resiliency, not worried about scale, not those types of things. I mean, of course, when you're doing IOs, you're always thinking about, can I, in my code, can I reduce the number of IOs that my code is doing? Those types of things. But it's a whole different way of thinking about building applications. So to kind of wrap up, what I like about this approach is that the microservices behave more like neurons. I've been talking about microservices since, pretty much, they came out. And some of the things we've been talking about, talking about, but not necessarily doing, is things like microservices, you do one thing and do it well. In this case, you're kind of forced to do it that way. And the behavior of the microservices, commands come in, commands emit events, events get persisted, and then the events trigger state changes. That's the flow. Those actions, those synaptic type actions, they're just consuming events, from upstream event producers. And for the most part, they're translating those events into downstream commands. They can do other things. They can do queries against views and things like that. Like, for example, when I'm allocating stock to the order, the action has to do a query to see if there's any available stock to be allocated, and it grabs that stock from the query. The messaging in the system, it's, I don't know or care how it's implemented. I do know, but I don't care, because I work for the company that built it, and I talk to the engineers that built the platform all the time. But that's not my concern as a developer. As a developer, I'm focused on wiring together the flows, defining the flows, designing the flows, in my software, at a very high level of abstraction. But the messaging is at least once. But that also means that you have to consider right in potency. And meaning that with at least once messaging, what that means is the consumer of the message will get a message at least once, and it might get that same message twice. So if you're doing things like decrementing the balance of an account, or taking inventory out of a stock, or something like that, you have to think about that. How are you going to handle the processing logic when those types of things happen? So I mentioned the patterns. And in the Earthship, relatively simple demo, there's two, it's called choreographed sagas. The choreographed sagas is the saga pattern without a controller. So that the saga happens just because of the way things are wired together. There's a lot of these actions that do one to many, meaning that one event comes into the action, but it emits multiple commands to multiple downstream services. That's a very common pattern. There's generation loops and generation trees. This reduction tree I mentioned earlier, there's three of them in the demo. And the reduction tree is really for that, is reducing something like, how many back orders do I have for a particular product? How many, with the stock orders they're in, how much inventory do I have for a particular product? The reduction of all that detailed data is being handled by these reduction trees, and they're really there for fried on potency. So like I said, this is a different way of thinking about doing event-driven. Maybe you don't have a platform that does it, but so that's okay, because I think event-driven itself is right there. So we did a review, looked at some of the event-driven architecture, looked at a bit of code, and went into this micro-mind. I'll leave you with two things. This quote from Alvin Toffler, the literate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn and relearn. I had to unlearn a ton of stuff to, and relearn new ways of doing things to use this new toolset. It was hard. I think we've discovered some patterns that will help people that follow us later on, but this is a skill set that I think every one of us needs to have if we're in IT. This one's relatively new. AI will not replace you, people using AI will. I use AI like all the time. Most of the images that are in my presentation are AI created. I've been using co-pilot for a year. Get a co-pilot in my IDE for a year. It's amazing. I'm sitting here, I'm working at home and I come out to my wife and it's like this thing's in my head, it just reads my mind. It's just dumping out blocks of code and I'm just moving. It's amazing. So thank you very much. This is the QR code for the slide deck, and I think we have a few minutes for some Q&A. If you guys have any questions. Yes, sir. That's a really good question. The only reason that particular service, the shopping cart service had bad validation checks in it was because there was a client, an external client talking to it. The other services, and I think what you're getting at is when you're cascading and wiring services together, a service just can't reject, right? So this is very insightful, excellent question. The service has to have like a compensating action. So instead of just saying, no, I can't process this message, it'll emit an event saying, no, I didn't process this message. And that event goes back, is wired to go back to say the event producer. That's a very common pattern that was in this type of a system. But so you have to be very careful about just rejecting incoming commands. And it's only really for services that are on the edge of the system where there's external clients that aren't a venture of inclinin and can handle the error response. Any error questions? No, it doesn't, not in this system. It's not yet at least. So you mean like training it with, yeah. So that's an important distinction that the service, the behavior that's in this system, I kind of refer to it as instinctual. You know, like when we're born and when other organisms are born with a neural network brain, there's a certain amount of pre-wired behavior in the system that doesn't need to be learned. And that's kind of what analogous to what this system's doing. It's just pre-wired behavior. But what's interesting about this, I think, is that how powerful that pre-wired behavior can be. Don't put your hand into a fire because when you feel pain, you immediately react. A lot of that is instinctual behavior. It's not learned behavior. Some of it is, but it's like insects, for example. They have some learning ability, but they have a ton of pre-wired behavior built into them. And that's kind of what this is. But it's extremely powerful. That's what's kind of surprising me. It's like, man, I'm doing all this stuff with just these 13 simple services. The code's not complicated. It's the wiring together that makes the behavior of the system really interesting. Yes, sir. There's a state of each instance. Like there's a state of a shopping cart, state of an order, state of stock, state of product, those types of things. And you have to think about that as you're designing the system. That's the fun part, the part I really enjoy of doing this is thinking through the design, thinking through the compensating actions. You kind of have a happy path where everything works just fine. That's kind of the easy path, but then there's the unhappy path. So things like there's services that, like an order comes in and the order's decomposed down into his individual physical units of stock that it needs and those physical units of stock that need, they go looking for individual units of stock in the inventory. And it'll send in a request to one that's supposed to be available and that one's not available. So it emits an event that says, sorry, I can't join with you because I'm unavailable. Some other order got me. So that event goes back to the requester which triggers it to just try and get something else. But maybe there's nowhere stock the next time it goes back. So you have to think through that. Like, so what happens when there's no stock? So every single step you have to think through it's not as bad as I feared it might be, especially when you're decomposing a problem into these small processing units. That's where I think it makes it simple. If you have a big bulky services getting, it's got that smell because it's getting too big, then you know you're on the wrong track. And often the solution when I ran into that was to decompose it down into smaller delegate to smaller services. But you have to think about it, make sure. I think we're out of time, so thank you very much for attending the talk. And I'll be around all day today, tomorrow. If you want to talk to me, I'd love.  This thing, where did you talk? You can hear me? Yeah, you can hear me. Every mix, silly thing I say you hear now. There are chocolates and there are puttman stickers here. That's maybe the most important. Belgian speciality. We want the chocolate off. Voila, another hand, good. Another one, right, or that's too far away. Who was it? You were? So we don't have a puttman desktop swag yet, so that's why. Chocolate and puttman stickers, that's what I can offer. I have to wait two minutes or can we start? Hello? All good. So, microphone working, everything okay? And people still coming in, so I will wait. You have two more minutes, so. Yeah, so I had to learn to use the windows to work on the project. That's very fun. You introduce. There is no more movement, you know. So, Fabrice, works at Crest Hubs, as a technical writer, and he will be talking about puttman. Desktop, puttman desktop. A little bit about puttman, but mainly about puttman desktop. So, that's the, so puttman desktop was announced as version 1.0 for just a few weeks ago. And it was, we were all sweating the week before because it has to work. So, it's a very young app, let's say that. We are building features like crazy in it, and when you build one feature, another one is broken very often, but it's start to be really usable to do nice things. So, why, why, why, why puttman desktop? It's, so how to explain that? It's that for production, you want to have your containers running on Kubernetes or OpenShift most often. And in development environments, people are more using Docker, Docker compose, and other things that are just, okay, they are containers, but they are not working the same way. And your container works on development and then it doesn't work on production. That's sad. That's a very nice picture explained by one colleague. So, that's the idea is, okay, Docker compose and Kubernetes-CML, that's not the same. Let's try to give a tool to developers to produce the Kubernetes-CML and make it simple to produce. And that's why it's not just Ponman, is that there is one part of Ponman desktop which is really using Ponman, and the second part which is bringing you from container on giant to Kubernetes and OpenShift. You too. I have a session this morning and one this afternoon. The main reason is that because you are two people applying for the sessions and the other person cannot come. So, I will focus this session more on what you can do with the container on giant. And this afternoon, I will not talk about the container on giant, but try to focus more on what kind of Kubernetes workload and features we have. Voila. So, one problem that we solved and that I didn't talk about is that containers work natively on Linux. I know it's C groups and so it's features that are really bound to the Linux kernel. And on Windows and Mac, you need to work in a Linux VM. And that makes it a little bit more complex to, so it's more complex to start with Ponman and on Mac OS and Windows. And the first thing that Ponman desktop brings to Windows and Mac OS user is it's easy to start with Ponman.  For Linux users, we don't care. So, it's not really a feature, it's more a non-feature. And for example, I have been using Linux for 20 something years. And now I do this presentation on Windows because some of the features are specially baked for Ponman running in a VM and not for Ponman running on the host. Meaning that when I was testing on Linux, I had things that didn't work and the application is really, really, really meant for Windows and Mac OS first. And we have very few Linux users. Yes, so container and giant. You see the little compose here? So you can install compose on it. So you can run, you can run compose. But the interesting thing is that then from the container that you started by running Docker compose, you can continue and do something that we hope is better. Yeah, so proxy connecting to registries and installing in restricted environments. That's feature that we implemented in December, January, February. So that's the first, it came even before connecting to Kubernetes this feature is that if you are running in a jail where you don't have access to internet, and we made this feature fast. So really for people who want to run Ponman desktop and Ponman in lockdown environment in enterprise. For most of the user who are working remote, it doesn't, it's not a feature, but for enterprise it is. And you have a little, so you can choose between your Kubernetes context.  So if you already have clusters, they will show up here and you can select in which context you are working. And now I have a lot of time for demos. Okay, so there are some demos that I don't do live because it takes too much time. So, but there are some that I want to do live. So this is how you start the Ponman machine in Ponman desktop. And I'm sorry, but if I do this demo live, it will take 10 minutes and I cannot show everything. So just it's easy. There are certain circumstances where you, so Ponman by default is running a little less. So it means that the Ponman is started with your, with a user, a basic user, not root. But in certain conditions, you want to be running a Ponman as root and there is an option for that. And we made that for kind on Windows. So to run kind on Windows, it works better if you have a rootful Ponman. Good, hopefully I can, I don't know, this afternoon, my rootful Ponman machine didn't work yesterday. So I will not show it this afternoon unless I can fix it. Pulling an image. Let's see if the network is okay to hear. So pulling an image, I will delete the redis image. Up, up, up, and I will try to put an image. And I have my problem with Docker not accessible today. That's wonderful. If I'm, I like the live demos, you know, when you break things. Okay, so it doesn't work. I'm super happy. Let's do, when the gods of registries are with you, pulling an image is super easy. So I've never had Docker registry refusing to download an image before this morning in this room here. So that's the big surprise. So, but yes, okay, so you can download image from multiple registries. And so when one registry is down, you can use another one. Quay is a code registry for that. Next, next, next. So pulling an image, starting a container. So I will not do the demo with the disk redis because I killed it, but let's take engineers. So to start a container from an image, you know from, sorry, I don't know. If you use the command line, you know, a lot of command line, you put that in a script, et cetera. Here you have a lot of options that are displayed in the starter container screen. And usually these are things that you define in the image. For example, the port to expose that's defined in the image and you just have to validate. You don't have to do anything and you should have something that works out of the box. And so now I have my engine is working good. And I can interact with my container. And when you go to the data list, it's always open the log first thing. There is a summary, which is not the most interesting thing. In the logs, if you click in the logs, no, control F doesn't work today. No, if you do in the inspect, if you click in the contents and you press control F, you can search for things inside. That's a nice hidden feature. So that's, you can inspect the container. The cube thing is one of the features I will talk more a little bit later. So from this container, you can create a pod. And basically the definition of a pod is a Kubernetes YAML. So when you create a container, you can directly have the YAML content to create a pod out of your container. And you can interact with the container. And last thing, no, not exactly last thing, you can, as the container is exposing a pod, you can directly visualize what is trending on this pod. When you're running Podman, that's easy because it's localized, the pod. When you are running in OpenShift or Kubernetes, that becomes a little bit more complex. And you can, you have here a button to deploy two Kubernetes. And the context here that you can see here, it's depending on the context that you activate here. Okay? So here I can decide to deploy to OpenShift local to develop our sandbox. On OpenShift local, I have two profiles and I have a micro shift instance that doesn't exist anymore, but I can still do it. Let's back to the slide. Starting a container, starting the nginx container, building an image from a container file. Yes, let's do this. And normally, today, I am cheating a little bit. So here you can put another name. So you first have to choose a container file or a docker file. So most often it's called a docker file, but sometimes container file. Then Podman Desktop will take your container file and all the directory that are around. So if you are copying files, your Podman Desktop has all the context to do the build. We'll see that in the logs. But as I'm cheating, it goes very fast. Okay? When you build for the first time, it's actually uploading the build context can take one minute or more because it's taking the docker file and the directory from your local file system. And it's copying that to the Podman machine, which is in OVM, and then the build is starting. And here I'm cheating because it's already built, so I'm using the cache. I'm not showing you. You want to see that? No, you don't want to. So registries. So that's important. What do I start? Yes, registries I will show in the app directly. So the registry, they are in the settings here. You have four registries that are pre-configured or you just have to add your username and password for Quay, in fact, it's username and a token, but. And if you want, you can add your own registry. So if you have a private registry just for yourself or for your company, you can do that. And when you have added one registry later on, you can, if the image name that you have, so if you built an image, if the name match the registry, then you can push it. For example, this one, my custom image, I can push it to a client cluster, but I cannot push it to my registry. But this one, which has a name that correspond to my Quay.io registry, I should be able to push to the registry. And then when I push to the registry, I have, I can select between tags. Obviously, I take the tag that correspond to the registry. And then I haven't tried this today. Does it work? Does the network work? Yeah, I'm happy. It did work. So configuring, you can configure a registry. How many times do I have still? Five minutes, oh yeah, yeah, yeah. You can push an image, we have done that. And next to pick five minutes to talk about pods. Oh yeah, yeah. That's live demo. Okay. I have my pod, I delete my pod. Hop, let's be crazy. So I have redis, I select two pods. So to explain pods, as I live in Belgium, I was thinking of a metaphor that is implying fries. You want it? It's, so in Belgium you have two categories for people.  But frites au safar, they call it. And pods, it's a little bit the same thing, is when you have container that are exposing pods, you want that the application is exposing the pods to the user, that's good. But the database, you don't want to expose the part of the database to the user. You want to keep it in an internal network. Same thing in your fries. You expose the fry to your mouth, but the sauce, it's just for the fries, not for your mouth. So, but you cannot do it with fries. So you select your pods, and then you select the pods that you want to expose externally. And the pods that are only required internally, you don't expose them. And, there he is. He's working. Perfect. In the logs, you see that you have the logs of your two containers, the piton app podified and the redis podified, okay? And, and can you search in the log? No, not today, okay. You have, you can inspect the pod, okay? And here you can search in it. And you can see that it's exposing pods 5000 and 8080. Good. And you have, you can copy and paste your, the YAML to create it from code somewhere else, okay? So that's already cool. And you have the possibility to deploy to Kubernetes. That's for this afternoon. And here, if you go, that's where the application is still young because if you want to connect so now, you want to see the app of your pod, you don't have access to it in the screen yet, okay? It will happen. But we have not done yet. But in the summary, you can, oh, I clicked on it too fast. In the summary, here you have the name of your containers. You have three containers. So there is one secret container that you didn't choose that happened here. And if you click on the container name, secret, then you are on the container. And this time you can open the browser and we have the application and the increment is stored in this database. So that's cool. And if you, so you have the, you have, you can see your logs. That's good. And you can see that the container, the container that I selected are now stopped. So I don't have any more container in the running outside of the pod. And my pod has been created and my pod has three containers. And the last of them is an infra container that's, that's rolling them. And for these logs, you don't see things. I can show you. You can interact with, with the, the reddit's database here. How many seconds? It's over, huh? So I will finish that and then questions. So question, you can select containers to run a pod. We've seen you can generate the YAML to run a pod. Questions or the last demo is you use the YAML and you start a pod. You can still use Pondman compose. You can still use Docker compose after that. Okay. So that's, that's not. You're just gonna work with the same stuff. Yeah. Okay. So when, when you, when you do the Docker compose, you cannot send the thing to Kubernetes. Kubernetes will not understand the code compose. So that's the, that's, that's the first point is that you have something that works on development. But then if you want to put it on production, the Docker compose is, is not working. So you have at some point, you have to do the translation from Docker compose to something that is Kubernetes YAML. And that's, that's where, that's where, what we are trying to do is that this translation is as, as painless as possible. So, and, and because, so you can use, you can use a Pondman desktop with Docker. I failed to install Docker this morning, but that's another question. You can, so you can use Pondman desktop with Docker and then you can use compose with the car and then use Pondman desktop to take the state of your containers to generate pods or to generate QBML because you cannot run pods on Docker, that's the problem. But you can, you can, you can generate the YAML and then run the YAML into Kubernetes. And, and so we support Pondman, we support Docker, we support Lima at the, at this moment. So three different container and giant. And for Kubernetes, so we, we support remote Kubernetes and giant that are in your Qube config. And we, we have more support to help you install OpenShift local, helps you install kind and connect to the developers on bugs. And that's what I will develop this afternoon, this kind of stuff. So the point is, do you want to deploy to Kubernetes after a while or not? If you don't want to deploy to Kubernetes, you don't need. If you need to deploy to Kubernetes and that's something that developer need to do and not the upstream, that's a very nice tool to make it easier. Well. Basically, so it's done. And Docker always wants root containers and what does want root less. And so, and for example, if you want to run, sorry, if you want to run the standard, so the Docker IO release or, or Nginx image on OpenShift for example, it doesn't work. So they are, OpenShift is putting constraints much, much higher than Docker. So, so containers are running fine in development and everything is okay. And then you, you push to production and suddenly the container is not working. So it's to have, to have this, this information, this feedback loop be closer to development. That's a, that's a nice tool. And that's something where the usage is different from the Docker or Kubernetes extension in VS God or in whatever editor you are using. It's not exactly the same usage. The, the, the Kubernetes extension is not something that brings you from, from container to Kubernetes. Yeah, that's a tool that's just, just to do this transition. To do this, I, I, I, I work on containers. I have everything that I, that I want on containers and that's what, what do I need to do to, to move to the, to the, to the next step. Well, good answer. Another question. New features that are coming. Stability. That's a, for me, that's the most important. So, so we have, we have been building features like crazy in the, the last month. So the, the project is a one year old since February or March. So it's very, very, very young and it was going very, very, very fast. So stability. So, and to end test to, to know that when, when we build a new feature, we don't break another one. So that's, for me, that's the most important one. And then for now, the open shift local extension, it's, it's the first iteration for the, for kind also. It's the, it's the first iteration of them. So that's a, we discover all the age cases that make things break at some point. And you know, and also things like now I have my pod running. I want to have the button to, to open my application directly. So to, to have things even more, more simple and less hidden. There are some features that now are just it's here, but you have to know. And I would like it to be, to be more, more, more simple. And that's quite a lot already. And, and yeah, and I know that what is coming now is the, the welcome page has been reworked because now it doesn't, if you, if you look at the welcome page, it doesn't fit in a screen, you know, I have one man that takes all the, all the room, but the rest is. So, okay, yeah, I'm, I'm a, I have a big scaling factor to display in a big screen, but this, this welcome page is, was okay when there was only one man. But now that we have more things. That's a, that's a problem. Another question. So the time, the terminal interface is there. So anyway, we install it. So you use it, you don't use it. It's your choice. I would not choose for people. I'm, I've been using the, the, the, the text interface for years and I learned things about containers by using this app, you know, because there are, there are things that are more displayed.  So in Kubernetes deployment is even more central than pods. And you don't, you don't see them. You don't see the, the secret or other things. So there is a selection and a simplification of what you need to know. I find it's really interesting to, yeah, to enter the machine and to say, okay, that's, that's the important point. I will focus on them and the rest we'll see, we'll see later. But I don't know, command line and windows, that's complicated. So I have command line user for years. And now I have cmd.exe, PowerShell, Git Bash, and the, the, some of the commands land in one of the, the terminal and not in the other. It's, it's, it's getting me crazy. So the, the, the terminal experience on windows is anyway, is a problem. Is a, is a, it's just, just, okay. Terminal, open a terminal, which one? Cmd, PowerShell, Git Bash, something else. I, I'm using this, this terminal that has all of them. But you see, it's, what, what do I, what do I want to, to, to use now? So that's, that's complex. So this afternoon, I will try to break things on Kubernetes, OpenShift local, developer sandbox, and all of that. And, and it will be fun. Ah, yes. And you can see our, hello, good morning everyone. I'm thrilled to be here today to talk about a topic that is crucial to the health and performance of Kubernetes applications and applications in general, testing observability features. As developers and DevOps engineers, we understand the importance of monitoring our applications, diagnosing issues, and ensuring that our alerts are accurate and timely. Today, we'll delve into how we can achieve these using Prometheus, alert managers, and various testing frameworks and libraries. A little bit about me. I'm John Villasa. I work at Red Hat. In specific, I'm contributing to the QBIRT project. I don't know if you had the chance, but in the first floor here, we have a small booth where we are giving some demos and giving some swag. So if you have the chance, please stop by. I started working on a tool of QBIRT named Hyperconverse Cluster Operator, but then slowly I started having more focus on monitoring and observability features across the deal components. So the outline for today. We'll start by setting up and seeing how to do a test environment, how to test metrics, then moving on to alert testing and how to ensure that they are actionable, relevant, and real alerts. And in the end, I'll show you a small demo of how I did everything together. So let's start with setting up the test environment. First, we need to understand what the test environment should look like. It needs to be obviously a controlled space where it can simulate the conditions of a production environment, obviously. But we don't want the risk of causing disruptions to our actual users. And in our tests, we'll need to create and delete a lot of resources, remove permissions, cause network problems. So this is where the concept of a disposable local cluster comes into play. A disposable cluster is nothing more than a temporary Kubernetes cluster that we just create for the purpose of testing. The beauty of this approach is that we can spin a new cluster whenever we want and whenever we need to run a test and we can just tear it down once we are finished with it. This ensures that every test of ours starts with a clean slate and we don't have to worry about any leftovers from previous tests. And weird permission changes that we might do to test our components. Spinning up a cluster can be used with a lot of tools. Some of them easy to use and available right now are, for example, KIND, which is Kubernetes in Docker, Minikube, MicroKaitos, or any other. On Qvirt, we actually have this cool tool already. I literally use this every day and I'm at work. And it already creates a cluster and has even a flag and sets up everything from it is related. So it's, I really like it. And in the automated tests nowadays we don't use it. We provision full clusters. But that's more like, I see, like a bonus for when the projects are more mature and the small projects that are starting, I don't think they really need it at the beginning. So, let's start now with metrics and events. Metrics are obviously the ways of observability, right? They give us insights on how the application is behaving and they help us diagnose the issues. And we'll explore how to test those metrics. I'm putting the question like, are unit tests enough?  Of course unit tests are not enough. But in the ideal world, everything will be fully tested, right? But then again, we can't have everything and we know time and workforce are limited. So in the end, I guess at least in the beginning, unit tests are kind of enough. In my opinion, it's more important to start with end-to-end tests for alerts. And usually a lot of our alerts already use metrics for the calculations. So if our alerts are working correctly, we have some degree of confidence that the metrics they are using are too working correctly, right? But even though we can start simple and most of the times like the simple things are the ones that end up saving as the most time in the future. And when we talk about metrics, it's very important to first validate that we follow the right naming conventions and we have the correct levels. And if possible, if checking if the metrics are prefixed with a component name, so that later on we can easily find out where the metric is being created and just more quickly trace it back. And have the unit tests for functions that update those metrics and in the end of the test validate that their value was correctly updated as we'll see later on in the demo. Now let's start with the alerts. One of the main concerns in this topic is that we don't want to be flooded with false alarms or miss critical alerts, right? So we'll discuss how to ensure our alerts are actionable, relevant, and they are real. And we'll also cover how to configure them correctly and ensure that they react to the appropriate triggering conditions. This is probably the most known quote in the area and if you worked with alerts, you already probably saw it that every time the page that goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued and every page should be actionable. This is taken from the site play with engineering book. So it's like one of the vivals of observability, right? And we have to be aware that an urgent alert might actually wake up someone in the middle of the night. And this is even more important when we are producing software for external clients. We don't know how many people on duty they will have. Actually worked on a small company before and you were two people managing infrastructure and we didn't have someone already ready at all hours of the day, right? And since we have clients on the other side of the globe, if we have an alert in the middle of the night, one of us will need to wake up and look at it. So it better be a real alert. So how should an alert be configured? It should have an owner, a contact person that it's basically able to quickly understand the problem that created an alert or the process it refers to. Sometimes it might be the developer of the feature, other times it might be someone from a monitoring team. So this is really important because sometimes we don't want to start with something that we don't know very well and knock your head off and then lose valuable time. And as in metrics, we should be able to quickly identify the component which the alert refers to. For example, if you have a Kubernetes operator, that is managing, for example, resources on IBM Cloud, we might want to have an alert like, IBM Cloud is not available. So we want to identify which component created the alert so we can quickly understand where it came from and go there to navigate the logs and try to understand in more detail the problem. Also a summary and a description, and I think those are actually pretty straightforward, but we should also have a link to a handbook which we'll see in more detail in the next slides. Usually for severity, some people use different severity levels, but some of you actually recommend these ones. And they are useful to distinguish like which actions we should perform for each alert and to sort the priorities on them, right? For critical alerts, those are usually the ones that will page people in the middle of the night. The warning alerts, we sometimes just want maybe to create a ticket that should be looking into the next day. They are usually useful for, like some component is reaching critical memory. If you don't do anything in a day or two, they will reach a critical state. And if we usually don't really perform any immediate action, we just also create a ticket that will go to the bottom of the line. About the alert handbooks, I think those are really, really the important part because they serve as a comprehensive guide for the cluster owners or the operators. And they should provide a step-by-step instructions on how to handle the specific alert. Like you all know that if you don't provide these handbooks, usually owners or operators, will need to go through a lot of the documentation pages or their personal notes to understand how they can debug and fix the problem. And this always leads to losing valuable time. Usually that time also means losing a lot of money, right? And even with standard, sometimes we force them to rely on memory or improvisation because they will handle some related issue in the past and that will also lead to mistakes and more delays. So, how can we test the alerts? In our tests, we should make sure that all the alerts include all the mandatory fields that we said before, that each handbook URL link is valid and the handbook actually exists, that the alert includes a reference to the instance or pod. It might be like the name of the component or really in a label about the pod or even in the description. And the alert is triggered when the expected conditions are met, right? And then again, most of these steps are actually very simple, but as I said, for metrics might just save us a lot of time and addicts in the future. So, now let's put this theory into practice and I'll try to show you a small demo that usually goes very badly. So, to start with, I have here the creation. I don't know if you can see if I should increase or do I want to increase? Can you see or is it better to increase? It didn't even start already, first issue. So, in our project, we want to create the metrics, right? Actually, I should start it. So, this is a simple operator for Kubernetes. I logically didn't have no logic at all for now, just some simple metrics and alerts to show you how you can start with, right? And from then, I used in this case, kind to create a new cluster and I just installed from this. I'm doing this locally, but obviously you can follow these steps on a cluster you have for testing. It's even possible to do it on GitHub actions or GitHub lava. So, really simple steps that you can easily do in five, 10 minutes. And after having the cluster, I then created my metric. I have here the control level to refer to the operator. We are now looking also, for example, to have stability levels that will allow us to deprecate metrics in the future. So, all these levels are really important for you to put some time into thinking about them and because these are the kind of things that will help you, your team and your customers. And then, for example, for these metrics, we have here the reconcile loop of the operator. For those of you that are not familiar with operators, it lets us create a custom resource on Kubernetes and then, for example, when I create that resource, it runs this reconcile loop for me to perform any actions. Imagine that here, as I said before, I want to work with IBM Cloud. I might want to create, like, a machine on IBM or a route, something like that and this is the place where I will be doing that. And for example, here, whatever the logic is, I want to increment this metric that will tell me how many times the reconcile loop was created. And as I said before, it's real here, we can start simple and write the test for our metrics. In the first step, we can start with easy validation that, for example, the metrics follows the Prometheus conventions. And we list all the metrics and then we'll link the metrics. Here, I'm using Promblin's tool which already brings a lot of validations that, for example, the metrics have all the necessary structure. For example, counters should not have the total keyword at the end. And as I said, this is really important to take a look in the beginning when we start adding metrics. In Q-Virt, we have a lot of developers adding metrics and without any validations. And in one component only, this is... So what ended up happening is that when we added the linter, we have all these issues. Non-Instagram, non-summary metrics should not have the console fix and you see those are a lot of errors in terms of units and so on. So, and now we saw before that we are adding stability levels. Why are we adding that? Because this project is used by clients. And which metrics are clients using? Those ones that we created before. So we can just simply go there and rename the metrics because they are using them and it will cause them a lot of trouble. So we are now thinking about we should deprecate these metrics for two versions, create new metrics with the correct names, warn the customers that these metrics will no longer be supported, then... Trying to see if they are already using those metrics, not causing them issues and this will take a lot of time. So, and you saw that really it will take like very few lines to call Prometheus Linter and save us from these kind of troubles. But that's life and that's why we are trying to now expose these issues that we have with the community and writing some best practices so these problems don't happen again, right? So moving forward, here it's the unit test for the metric, right? We get the initial reconcile value count and then we run the reconcile loop. As you saw, it was very simple so it's obviously updating the metrics but in the future, your logic will be much more complex but even though in the end you find it's up on this case expect the initial value to be added one that will be the final value, right? So, pretty simple stuff. From there, we move on to, for example, recording rules. So, this is where we are creating the, protecting the rules. So, for this project, I created two simple recording rules, right? The first one, the number of operator pods in the cluster which is simply a query for primitives which counts the number of pods up in the cluster of that type and also the number of heavy pods which is the sum of the pods in the cluster because this metric will have the value one if the pod is ready to be used, right? And also you can notice here that some of the, we are also already trying to use stability levels on the alerts, for example, alpha. Here is just an example but heavy is to show that this is not actually tested and it's still being worked on and other. From these recording rules, we are building our alerts and one of the most important alerts to start with is obviously testing if the operator is down or if the operator is not ready. We make use of the recording rules we saw before. If the number of pods, operator pods in the cluster are zero, we'll be triggering the alert and if the number of heavy pods is less than the number of pods, we'll give the alert test operator is not ready. For those, we have also some validations for recording rules. We are linking the metrics as linking the recording rules as we did for metrics because they should follow the same conventions as before. And then here, for example, I want all recording rules to be prefixed with the name of the operator. And also the same for alerts that should follow Prometheus conventions. And actually here, I also put the link here that I said before that we are trying to achieve adding the best practices for observability on operator SDK. So a lot of these validations come from there, right? Alerts must be in Pascal case format. They must have an expression. We are validating labels, validating annotations. So just following the recommendations there. And these are basically the unit tests. I could have handles just to see that as a good developer, I follow by the rules I created. This one for the metrics, it will be very funny that it fell. And this one for the rules, pretty simple stuff. So now moving forward for the end-to-end test, that's what we want to know, right? I already here have a cluster. Actually, I see that I have the resource created and the pod, but on running it will clear it all. I'll start running because it takes a few minutes, I think. So, and let's see. For metrics, we are doing the same thing. Here some setup deploying the operator, then port forwarding the Prometheus service so that we can access it locally and deleting any previous resources that exists. And our test says that we should increase the test operator reconcile count when the reconcile count is run. So we just get the initial value for the test operator reconcile count and we create a new resource. As we saw before in the reconcile loop, any resource that is created is supposed to update the metric. So in the end, we know that eventually when getting the metric, it should be equally the initial value plus one. This is a really simple test for metrics, but as I said before, it makes sure that everything is working as supposed. We saw that this is very similar to the unit tests, but in unit tests, it's easier for us to know that the metrics are updated. Here, it's more tricky because it has a lot of also other operations before because we need to make sure that Kubernetes is actually passing the right events that is being cut by our operator and then later executing the reconcile function correctly and we have reconcile count, but we can have number of resources created, number of resources deleted, anything we want, right? And for alerts, we first have the verification I mentioned for the handbook. We are checking that the handbook URL is available. I created just some gist on GitHub for the purpose of this demo and I can actually show because I copied it from one of our handbooks, cleaned up some stuff, but let's see because it's also useful to know what a handbook should look like. First, we have the meaning of the alert that supposedly these alert fires when no test operator pod is running the cluster, the impact that these alert has. Sometimes in QVirt, the operator might not have like a big of an impact because the operator is not responsible for virtual machines, but if the alert is like virt controller is down, then users might start to have a problem because virtual machines are running while in the cluster and nothing is controlling them. And then we have all the diagnosis steps and here we should have clear steps that in the end we understand very correctly what the problem is. So moving back to the tests, we are now following the same approach from our last format, deploying a new test operator and then we are making sure that the test operator is down, is thrown when we want it, right? And in this case, for example, we just scale the deployment down to zero and since we have no pods in the cluster, we verify that the alert is being triggered. In this case, pending is enough for me because you might want to have a delay on the alerts just throw the alert if the condition is met for more than five minutes, for example, but if I see if the alert is pending, like the condition was triggered, it's just waiting for those five minutes. I think the alert is working fine, so I'm okay with it. And for the operator not ready, I just come here and set a random image which might mean, for example, that the repository is not available as happened earlier in the other demo, so it's a problem that sometimes we think, oh, this never happens, but it happens more than, it happens a lot, yeah, that's what I want to say. And we then validate that the alert is then thrown. Once again, if it's firing or pending. And as the time goes by, we start adding a lot of alerts. For example, we might want to see if, oh, it failed. If the, so, we might want to see if the operator is creating the right resources on Kubernetes, and for that it needs permissions. One of the things that we can do is just go there, just delete the RBAC permissions and see if the correct alert is triggered, saying that this permission does not exist anymore and you should look into it, right? So that's one of the things that we test. We can test, for example, for the HTTP requests if they are failing, that's actually a metric that Prometheus already gave us. So the number of things you can alert for is up for your imagination, actually. And just to finish, I want to present like this really useful tool. Actually, we don't use it on QVirt, but I use it on my master thesis, which is ChaosMesh and it's really cool. It's really simple to use and it allows you to create a lot of problems in your cluster, for example, deleting pods, causing network issues that might be a lot of latency, requests being dropped. It can cause CPU and memory issues. It usually really take a look. It's simple to use, simple to configure and it has a lot of potential for unit testing. So to wrap up, I just want to say that it's really important to add observability features. It really help us. Usually when we start projects we tend to overlook those kind of things but they end up being very important to help us in the future. And also important that they are actually working fine. As I said, we don't want our DevOps and clients to be waking up in the middle of the night just to see that the alert was not real or we don't want to have bad problems happening in the cluster, but they are not alerted for and they are losing money and they are losing clients. So those are the main takeaways. So that's it. If you have any questions, feel free. Yeah, because those metrics are not up to standards, right? And the idea here is to do when we have the stability level deprecated we will have a flag in the help text and they are not. Because if we keep metrics that we want to update we'll have a lot of more things to manage in the future, right? And we really have already a lot of metrics and it's hard to keep up with them. We are also trying to add tools to generate documentation and to centralize the metric creations but there are so many of them and if we have 10, 20, it's manageable, right? But when we get to the hundreds we have a lot of components that becomes a problem. And maybe it's not a thing as you said to we cannot remove them maybe in a version or two or three because those problems happen but eventually in the future I think we need to end up removing them or else it will be unmanageable because even it's an open source project, right? And people come and go and if we don't do anything about them they will eventually be forgotten and nobody will know why do we have this metric and we have another one similar. So that's why it's really important to validate them in the first place. I don't know if you might not agree, right? But it's my only... So, any more questions? Please. Four? Yeah. Sorry? Ah, yeah. So the question was if how to create metrics for reconcile errors. So basically that example is very simple but imagine you have an error you usually exit the reconcile loop with an error, right? And there I have just this... I have this just to operate our metrics. So reconcile count. But I could create a new one like reconcile error count. And in our loop here I perform some operations that throws an error and I just create that new metric like increment reconcile error count. No, that was... Sorry? Yeah, yeah. But you can have create any metrics you want, right? You might want to have an error that was... You could not connect to an external provider. An error that's just... You can create any metrics for the granularity you want because metrics are really cheap and you should create the metrics you need to then help you debug, actually. But then when creating the alert based on those metrics we should be more careful, right? Some of the errors might not be worth to alert for because it might come from user configuration that it was not able to create the resource with the properties it shows. So that might be like a warning or an info alert for them. But yeah, my advice would be create all the metrics you want because if you think that information is valuable create the metric. The alert you should be a little bit more careful here. Any more questions? So I think this is it. Thank you everyone for being here. All right, yeah. Okay. So John works in Red Hat in the storage team which is... which has a static view on which is it in the dust and he was talking about the favorite language of every rust. Hi everyone, thanks so much for coming today. So I just wanted to give a little bit of background on why I put together this talk and essentially asynchronous programming is something that's emerging in Rust and when I was learning how to do asynchronous programming then I essentially bumped into a lot of corner cases where it became a little less apparent how to proceed and so while the ergonomics are quite good then I kind of wanted to put together a talk that would basically talk about how to get started from ground zero in asynchronous programming in Rust. So the overview of the talk today is we're going to start by talking about the state of asynchronous programming in Rust. So basically just like language level support what's there, what's in external libraries and that sort of thing. We're going to talk about asynchronous programming fundamentals so we'll talk a little bit about the architecture, syntax and why that matters and then we're going to do a quick case study of a modified read write lock that we implemented in storage for our use case. So the state of asynchronous programming in Rust. It's designed to handle use cases like asynchronous IO and network operations so in general anything that you would need to do asynchronously the framework is designed to handle that so long running blocking tasks in the background that sort of thing. I wanted to give a couple of definitions because in asynchronous Rust we talk a lot about tasks and futures and tasks are kind of like green threads so if you're familiar with Java or Go you're familiar with this concept it's basically multiple green threads can be served on a single OS thread in parallel and futures are basically the handle that you can use to pull and access the final computation similar to kind of JavaScript promises for those of you who are familiar with JavaScript. So one note is that Rust doesn't really have the same kind of level of runtime that Go does so it can't preempt threads like Go can in their asynchronous implementations so it requires an API design that builds in yield points any time that a green thread can't make progress or a task can't make progress and futures also require an executor to handle driving them to completion so in general you need some sort of executor to kind of drive the asynchronous work well so that's a lot of information but we'll dive into some of this in future slides. So the language level support is really just kind of async and await at this point so what async does is it's a keyword that you basically attach to a function and that turns that function into a future. Await basically allows you to await on a future to complete and while it has the same kind of semantics as blocking it's awaiting in a non-blocking way. In terms of the standard library support then there are a number of data structures that have been added in to the standard library and we're going to talk about some of these in more detail but the one I really want to call out is the future implementation so this is the trait that basically is the building block of asynchronous Rust and you basically implement this for all of your future needs. So in terms of external library support and then basically we have futures RS which provides utilities for working with futures so that's going to basically be combinators for basically handling multiple futures in parallel that sort of thing and you also have an executor which as we mentioned is required for actually executing the futures. Tokyo is basically a library that's built with more functionality built into it and it has a compatible executor with futures RS so the two can be used together. It provides a runtime for asynchronous programming and basically what this runtime includes is a bunch of async utilities like an IO driver, a scheduler, a network API synchronization locks so it's really kind of aimed to really cover a lot of the asynchronous use cases that you might bump into with asynchronous programming and then I figured I should also call out async standard but it does not have a compatible executor with futures RS in Tokyo so the two generally bump into problems when used together it basically aims to provide a similar API to the standard library but it's all asynchronous so this is becoming more popular for people that want to do kind of asynchronous standard library operations so with all of that out of the way we can talk a little bit about the fundamentals of asynchronous programming so first we're going to start with implementing a future and consistently this seems to be the thing that a lot of people struggle with the most in asynchronous programming I put up the definition of the trait for the future just so that we could go over it in a little bit more detail and the basic idea is that you just kind of define what output the future should give and then you have this pull method so what the pull method essentially does is it basically pulls over and over again as it's woken up and basically you return either that you're ready and you return the result or you return pull pending and it goes back to sleep until it's woken back up and it can make more progress so one note is that you should never block in pull so that can cause problems and we'll get into that a little bit more in future slides just why you should never block in pull so this is an example of kind of what happens when you're not ready so when you're not ready then you basically have to register this thing called a waker and so the waker was listed on a previous slide as something that's in the standard library support and what the waker essentially does is it's a way to signal to a sleeping future that it's ready to progress it's ready to make another pull invocation and so when you basically put the when you put the future to sleep then you essentially have to register the waker somewhere to be called later to wake up the future and basically then you return pull pending so here's an example of kind of when you would wake a future so here we have waker.wake it's pretty simple you basically call that when the future is ready to make more progress so something always needs to call wake when the future has reached a state where it can progress because otherwise the future is put to sleep indefinitely and will never wake back up and so one of the common misconceptions or difficulties learning asynchronous program is kind of where do you call wake and so I put a few examples of kind of where you could call wake it could be in a spawn thread in the background it could be in the drop method on the return data type from pull so say you return a handle indicating resource acquisition in the drop method then you could theoretically call wake to signal to other sleeping futures that they're ready to progress now the waker could also be sent using a channel across to another thread where you basically have a thread in the background listening for events and determining whether the sleeping tasks are ready to are ready to make progress so there are a lot of options and it's really up to the design of the developer so some important notes on futures are that creating a future does not start its execution a lot of people expect it to but it doesn't so it must be either pulled or awaited to kind of start execution however using Tokyo spawn will immediately begin execution in the background so that's one case where you can start it immediately pull also needs to handle wakeups where no progress can be made so in general you may expect that pull only needs to handle cases where waker.wake has been called explicitly but there are scenarios where it can be woken up due to side effects and where no progress can be made so your pull implementation always needs to be able to handle that we can now move on to kind of sort of the Tokyo runtime basics the runtime contains an executor and scheduler so that's where a lot of the IO code like kind of the functionality for E pull that you would get asynchronously in Tokyo is built into it has multiple runtime backends so you can choose either a single threaded back end where basically all of the tasks are interleaved with each other on a single thread or you can have a multi threaded back end for true parallelism and futures are spawned as tasks and multiple futures can make up a task multiple tasks can be served on a single thread so the end result is that blocking in a future can stop other futures from progressing and it probably makes a little bit more sense now why you should not block in either pull or in async function because that could theoretically stop another future that is ready to progress from progressing so there are two notes on the blocking aspect of things that I just want to call out which is that async functions can use blocking mutexes as long as they are not held across awaits so basically you cannot acquire a mutex then do an await and then release the mutex so as long as you're just acquiring a mutex for one single statement that is not across an await that is okay to use blocking mutexes and blocking mutexes are actually common in pull implementations but they can only be held for the execution time of pull because the general thought process is that if you acquire a mutex and then drop it as soon as pull completes pull is supposed to be very short running so if it's only acquired for that scope that is equivalent to non-blocking from the standpoint of asynchronous programming so there are two types of threads there are blocking threads and core threads so the core threads are the kind of OS threads that we've been talking about that can serve multiple asynchronous tasks on a single OS thread blocking threads are kind of a special case where they're used for areas where you cannot get around blocking and they basically assign a single task to the blocking thread so that it can block without causing any other task on that thread to kind of stop progressing and so that those blocking threads will get into a little bit more but that's one way that you can block in asynchronous programming I also just want to call out that the Tokyo documentation is very thorough so please take a look at the glossary and the more advanced topics in the tutorial because they'll give you more of a working knowledge of some of the terminology and kind of code that you can write with asynchronous programming so we're just going to quickly go through syntax and common usage we've already kind of covered async in a way but that's one thing that I just wanted to mention here we also have spawn which we referenced and so the way we can think of spawn is that basically we have this long network operation and so we call spawn on this asynchronous task and basically it spawns in the background and starts immediately executing we do a lot of other work that we have to do before we can handle the response then at the end of that we away handle the response and that's a great example of kind of a basic asynchronous workflow spawn blocking is very, very similar but basically it's for blocking operations and so this is going to spawn this task on a blocking thread so you can basically do all of your blocking here and so it's a similar kind of API you spawn the blocking network operation you get the handle, do other work and then away and handle the response so block on is kind of the inverse of spawn blocking where from that we're calling basically a blocking function from an asynchronous context block on allows you to basically call asynchronous code from a blocking context so in general here we have an example where we basically wanna make a blocking call but then after that we want to evaluate a future and this is assuming that we're calling mySync function from a context where it is safe to block so we can call the blocking call and then we can basically get the asynchronous API and then block on it so that we block until it's ready because you can't use await outside of an asynchronous function so you would need to block on the asynchronous code this would evaluate it asynchronously in the background blocking until it's complete join all is kind of a great example of spinning up a bunch of futures in parallel and waiting for all of them to complete so here we basically have a vector of futures so an arbitrary number of futures we pass that in to join all and await on it and then when they're all done completing then we can basically iterate through all of the results and handle them select is kind of turning that on its head where we basically have a future of a couple futures and we basically wait until the first one returns and when that first one finishes evaluating then what we do is we basically handle the result of whichever future evaluated first and we cancel all the rest so the previous one was kind of waiting for all of the futures to complete select is waiting until the first future completes so hopefully that gives you a little bit of kind of an overview of basic syntax usage kind of common functions that are used to handle futures and now we're going to dive into a case study of a modified read write lock that we had to implement on our team so the use case is basically that we had a table of data structures that all need to be locked independently and the IPC mechanism that we're using requires fetching read-only information from all data structures at once we initially considered locking each data structure individually but that scales linearly in terms of lock acquisition time and we began to notice a slow down when we were fetching these properties and so we ultimately decided that we needed to design a synchronization lock that handled our particular use case so we basically need to handle a single read lock on an individual element, a single write lock on an individual element in all read lock on all elements and in all write lock on all elements so we're going to actually dive into a little bit of code here so I've highlighted kind of the important sections here but basically you'll see we have our all or some lock and in that we basically have a lock record which is basically keeping the state of what is locked at any given moment and we also have this table inside of a mutex and that's keeping track of all of our individual data structures that we want to lock so we basically have an all read lock kind of field here and what this basic this is an integer because we can have an arbitrary number of all read locks at the same time as long as they're not conflicting with an all write lock for example we can only have one write lock all write lock at a time and we can have as many single read locks on an element at a time as we want mapped to which element is locked we can also have is we can have one write lock on each individual element so we keep track of which element is locked and have exactly a count of one for those so as you might notice if you're familiar with Rust this is upholding kind of the mutable immutable reference requirement so basically we can have all of them read locked as many times as we want to or all of them write locked exactly once so this should look very similar to references in Rust next we kind of have this implementation we're going to dive a little bit into acquiring a single read lock on an element and as you can see here we basically just kind of create a future and then we await on it so this is a great example of the kind of async await workflow we create a future and then await on it and that's really all that this does so it gets a little bit more interesting when we actually get to the future that we are awaiting and that's the sum read future so I've highlighted that this is the poll definition right here so we're actually going to see a more complicated example of implementing the poll method here for the future trait and I highlighted the acquisition of the mutex mainly because this is a great example we're using a blocking mutex here and because it's only acquired for the length of poll it is okay to use this next we kind of check whether we have a UUID or name that is registered in our table of data structures and if we determine we don't have anything with that name or UUID what we essentially do is we basically return that we're ready and we found nothing but we also above that call wake and one of the reasons this is important is because basically we've determined that we're done with our work but anything else that might conflict with this read lock needs to be woken because now it can make progress because we have finished our read acquisition so we have to wake up everything that's been put to sleep in the meantime if we do find that something has this name or UUID in our data structures but it conflicts with another lock that's already acquired as you can see in this next slide up on the top right then we basically add a waiter and what we do with that waiter is we store the waker so that we can later call wake on this for it to make future progress and then we return pull pending so this essentially puts the task to sleep and if we determine that there are no conflicting locks below in the ELTS block then what we can do is basically register that we have acquired a single read lock on this element we basically get a reference to that element using unsafe code and put it in this kind of some lock read guard which some of you might recognize if you've worked with New Texas before in Rust this is essentially a data structure indicating the lifetime of this lock acquisition and we can access the individual data structure from this guard so one thing that I just wanted to call out about futures is that when you drop them that is generally considered canceling a future and in this case we have to augment the drop method a little bit for the future mainly because if you're just canceling a future and there's no state stored in an intermediate place then you can just drop the future and it'll be canceled but in this case we're storing things like what's waiting for the lock in our all or some lock and we basically have to have this cancel method to essentially clear out all of the intermediate state if we drop the future because say we drop the future and then it doesn't clear out all of that intermediate state we could bump into a place where basically the lock can no longer service any future requests so always be careful about the intermediate state that you're leaving in your asynchronous infrastructure for the some lock read guard then this as I mentioned kind of indicates the lifetime of the lock acquisition and as I mentioned as an example of where we can call wake this is a great example of returning some lock read guard from the poll method and then calling wait in the drop method that was one example that I gave and what we do here is we remove our record of the read lock and then we call wake basically saying we're no longer holding onto the read lock and so wake up all of the lock tasks that conflict with this because they can now make progress and potentially acquire the lock so that was a lot of information and I just kind of wanted to go through a quick kind of series of closing thoughts about asynchronous rest because as you can see the design definitely requires some knowledge of the implementation to do it right? So some of the benefits are that the highly parallel nature of asynchronous rest allows very good performance on very few threads so in general if you are doing a similar sort of workflow without asynchronous rest you'll also you'll usually be required to have many more threads kind of handling that same workload and with asynchronous rest you can actually have those tasks kind of executing on fewer threads and it gives you kind of an ergonomic way to interface with that. It also has support from any types of combinators and asynchronous workflows that are already built into a lot of the libraries that exist so you don't really have to build everything yourself either. The design and documentation is actually very good and so in general I'm usually able to find answers to questions beyond that the community is very helpful and rust's kind of promise of fearless concurrency also extends to async code. You're going to get all of the benefits of type checking, lifetime checking all of that kind of stuff in asynchronous code as well. Some of the drawbacks that I just wanted to mention is that asynchronous programming requires care. So as you see it can be very, very detailed and it can take a lot of thinking about the architecture, where you're blocking when you're blocking. So an OS thread architecture can be used if requirements don't necessitate async. In general it's not bad to just use OS threads but if you're looking for performance then you know you probably want to go towards the asynchronous side of things. Lifetimes can also be harder to manage in asynchronous code. So asynchronous code has kind of the problem that we have multiple threads operating and so in general we assume that a lot of the time if you have a lifetime then you basically it has to be static and so it gets a little bit harder managing lifetimes and deadlocks and hanging can be a little bit harder to debug. There are cases where you have tools like Tokyo Trace to kind of help you out but it can be a little bit difficult. So with all of that said, thanks for attending. If you're interested in looking into the implementation a little bit more I've linked the repository where our modified read write lock lives and if you have a great rest of your time at the conference. So the question is whether I think asynchronous rust could help in kernel space to solve some IO problems that have popped up and so in general I've heard a lot of discussions I'm also involved with the effort to put Rust in the Linux kernel and in general I've heard a lot of people being very hopeful about asynchronous rust in the Linux kernel. One of the things that I think I've seen work happening on is basically building an executor that works in kernel space. So as I mentioned in the talk one of the major challenges is that you need an executor to actually handle the asynchronous tasks because of the architecture so there basically has to be some work on basically building an executor that works in kernel space and that may be merged by now I'm a little bit out of date on my information but that's one of the things that's popping up. So an executor can work without an operating system so in general the question is is there an executor that works without an operating system and in general there are executors that work without an operating system so I think I've heard of people implementing executors even in lower level situations than the kernels so it is possible, Rust is a pretty low level language so as long as you kind of satisfy the executor requirement and the trait requirement then you can basically implement your executor wherever you want to. The problem gets a little bit more complicated when you're dealing in environments that don't actually have an allocator and that's where you have to get a little bit smarter about how you're designing your executor because that can get a little bit. Okay, I think that's it. Thanks so much. Maybe this, this, this. One, two, yeah, it was muted here. Okay, so you should be able to hear me hopefully enough. So hello, my name is Martin Stepanko and I work in, not Quarkus team, but I work mostly on Quarkus at Red Hat. You can find me on these links and please do if you will have any questions about Quarkus after the talk. Usually I am responding on all of these channels so please reach out if you will have any questions. I want to point out this at the bottom. There is a code for 35% of any many products because I am writing a book which is called Quarkus in Action which you can already find in the early access program for six chapters which are published. So hopefully I know a thing or two about Quarkus so hopefully you will enjoy this talk. Please, I really don't have any slides. This is the closest to slides, what you will see today. I will open the terminal and I will start typing. So please, if you have any questions to what I'm doing, just shout. If you want to see something, just shout. We are here to make it interactive to give you the most of it as I can. So, and with that, I will show you this slide at the end so if you would like to see some things. So hopefully, tell me when this is big enough. Everyone can see even in the back. Okay, so the idea how to start with Quarkus. Of course, you probably know that there is Quarkus.io which is a site where you can find all the information related to Quarkus. Sorry that I don't see you throughout the monitor. You need to deal with it. Basically, if you want to start with Quarkus, the easiest thing, if you prefer clicking start that coding or code.quarkus.io, maybe I should ask, how many spring boot developers or spring developers? Raise your hand. Great, start that spring.io, code.quarkus.io. Exactly the same thing. Just here, you can see that in the spring boot you have starters in Quarkus, we call that extensions. You can see that the least, you probably cannot see how big the bar is, but I can click here, show more extensions and it's even smaller. I can show you it like this. It takes a while to scroll through this even on the computer that is around, I think, 1,000 extensions which are supported. Quarkus itself contains a lot of these extensions directly integrated into the Quarkus platform, however, because the community demand is bigger than we at Red Hat can maintain in one single GitHub repository. We created a so-called Quarkiverse organization which you can find on GitHub. GitHub, Quarky, if I can type it correctly. Yeah, Quarkiverse, which is basically another set of extensions which is basically done by community. Quarkus will give you a repository for your extension if you want to maintain your extension with the whole CI CD integration into the Quarkus platform, so with every Quarkus release you will be tested, you will still be sure that your extension compiles against Quarkus. However, it lives in your repository which is under this Quarkiverse organization and you maintain the code. So in this sense, if you have very cool library, you can get it integrated into the Quarkus ecosystem and still maintain the integration yourself. And as you can see, there is also a bunch of these repositories done by community because we just cannot, like, I don't want to spend any more time compiling Quarkus if I need to work on my extension because of your stuff, more or less. But Quarkus comes also with this command line interface which is basically just a CLI wrapping everything that you are doing in the command line with Quarkus. And one of these things is this create application. So if I want to create, because I don't like to click, create Quarkus application, I can just type, create app.io.exe.tepanq. Let's go with DevConf 2023. Very, sorry about that. So what this does is basically it will generate a Quarkus application with very basic stuff out of the box. Out of the box, we will get REST support, Maven wrapper, Maven config, basically some Docker files. And this is if you don't specify anything. With that create app, you can also specify any extension that you would like to add on the compilation to the project with minus X. So I can start typing name of the extensions here but I will not do this. The basic idea is that some of these extensions come with so called code starts which is basically some simple code to give you the idea of what you are basically integrating. But typically this is very good for people that don't know what they are doing. Let's open this in idea. But if you know what you are doing then typically it's just a stuff that you need to delete. So because out of the box it added REST easy reactive extension, why is this so slow? It, yeah, sorry about that. Give me a second. I need to remember the shortcut. I'm working on it. You just need to give me a time to remember the shortcut. So with these we will get a JAXAres resource generated with some sample string. It's just some simple code to remind you of the API that you are integrated. What that Quarkus extension is, is actually only a maven dependency from IoQuarkus, this one, IoQuarkus group ID and Quarkus dash name of the extension. Quarkus extension is basically integration layer of any library that you are putting into the Quarkus. Why would you want to use Quarkus extension and not directly the library? Because first thing, Quarkus is built around a few concepts which we don't have time to dive into in 30 minutes. So please catch me after the session if you are interested in this. Which you have access to inside the extension API. For instance, build time, slash runtime processing of application code, compilation to native, so adding some custom flex to Gravium, et cetera. So this is why if there is a Quarkus extension, please use that. Otherwise, in JVM mode, if you are running Quarkus on JVM, you can use anything that you like from Java ecosystem. In JVM it's still same old, but brand shiny new Java application. If you want to compile to native, which is a separate use case with Gravium, then you probably want to stick to extensions because that extensions will give you that guarantee that you will only type somewhere down here minus D or minus P native, and it will compile to Gravium native. Because that Quarkus extension is actually responsible to make this happen all the time. So if you have a custom library without that integration it might cause some problems. However, you still have access to some APIs in Quarkus itself to mitigate this if you need to. Okay. So first things that always you will do with the Quarkus project, if you will open it, is so-called Quarkus Dev. Just to mention, if you don't want to use this Quarkus CLI, in the background, all this does is call Quarkus colondane, which is Maven plugin, which is directly integrated here, somewhere, Quarkus Maven plugin. However, if you use this Quarkus to the benefit that it gives you is that it will shield you from the actual builds tool that you are using in the background. So you can also use Gradle, which is not that unpopular as many people think. So if I need to work sometimes on Maven and sometimes on Gradle, I am preferring to use Quarkus CLI because this will always run the correct invocation of command in the background. So what is Quarkus Dev or Quarkus is Dev mode. So basically this is single handedly the most productive feature that I got when I started with Quarkus development. What it does is basically it runs continuously your application in the continuous process and it dynamically recompiles any changes that you are making to your application. And if I am telling you any changes, big enough, let's just verify that I can call my application that generated hello endpoint that I got there. So now it's my application responding from the greeting resource hello from REST Easy Reactive.  I will just save the file if I repeat the same call. That class was dynamically recompiled in the background. It was exchanged with some call folders, et cetera, but in the general you have a new version of your application in a few milliseconds. Like in 800 milliseconds. So you can imagine that the workflow that you are doing is typically these terminals are on one monitor, ID is on another one, and I am just continuously typing the code, just refreshing the page on one terminal and my application is running for the whole day. Typically, really, if you are not doing something crazy, which, again, I'm working on Quarkus, so I need to do crazy, so I need to restart it a few times, but if you are writing your application, you don't need to start this for the whole day. And basically, this is the first really nice feature, so we have to go from here. Really, in 30 minutes, I don't have that much time to show you. So here in the bottom, you see that you can press a few buttons inside this terminal and something happens. So H will give me all the options that I can do inside this DevMode terminal. Now, probably second best feature that you will get in Quarkus is Continuous Testing. Since we are already recompiling your application, why we cannot run all the tests directly with it? So if I type R now inside my application, Quarkus will start Continuous Testing and then it will start running my test in the background and, of course, it failed because I changed something which that test is trying to assert. It is expecting help from REST Easy and I now have Ahoj REST Easy. So if I go into the test itself, and actually let me do it like this, I will move this terminal into the same workspace so you will see some magic. So inside that test, I will fix the test, Ahoj Z, and I will just save the file and you will see that DevMode automatically recognizes that I saved the test and it will rerun it and we see that now my test is passing. Nice. For me, this was impressive when I saw this first time. So you get the idea that you are basically just typing your stuff and you have this Continuous Testing enabled and by just saving the file, all your tests are automatically rerun in the background. So if I don't see on the periphery that something red is jumping inside the terminal, I'm not even looking at it. I'm just typing continuously. One heck that I learned myself is actually because if you are familiar with Spring, Spring Boot Test Annotation, this will still run a new instance of Quarkus. Quarkus test will run a new instance of Quarkus against your test. But with DevMode, this is still a runnable application which you can call normally. So if I'm not doing something which is really changing something inside the application which would interfere with what I'm doing inside my application in the DevMode, I will just comment out this Quarkus test application and I'm pinging really the application in DevMode because you see that this took like 300 milliseconds if I need to rerun the test. But I think that with R, I can rerun it and now you see that it only takes 18 milliseconds because this is just HTTP invocation against the DevMode. So in a few milliseconds, I can run hundreds of tests against the DevMode if I'm just testing something which is not changing something or if it's changing it in a way that I can still continue what I'm working on. But of course, then you will just uncomment the annotation and it will start the new instance against which you are testing. Okay, it takes 300 milliseconds which can bump up depending on the test, but you get the idea. So I put it back here and press H again. And what is now happening? Yeah, sorry. So what else you can do? With that continuous test things because definitely there will come a question like I have a bunch of tests and this takes time. You can play with these, you can only set to run the fail test, you can, I don't know. Oh, okay. Quarkus will try to do its best to guess what kind of code you are changing and it will try to not rerun all the tests all the time. It will try to rerun only the set of tests which are relevant to the code which you are changing actually on that iteration of DevMode. But of course, this is still software, so it's a guess, but typically it works pretty good or you can just toggle the broken only mode which will only rerun tests that fail. So we'll get there, but yeah. So the question is if this is only running unit tests or it can run integration tests too. Actually, this is integration tests because with that Quarkus test annotation you are running the new instance. So yes, this will also run Dev Services and we will get to it. So this will run everything in the background continuously. Okay, I don't want to jump. So we will get to the services definitely. Another cool feature that you can play with here is this DeVui which by pressing D inside this terminal you see that maybe you don't, but in the bottom my browser opened a new tab which is basically, oops, I didn't want to click. I want to put it down. It's basically a development clickable interface inside your Dev mode where each extension can add whatever they want. So Arc, for instance, which is our CDI container which is responsible for dependency injection, you can just click to list all the beans inside your application, like my greeting resource for instance. Or you can see these very nice removed beans or removed components because as I mentioned, Quarkus is doing a lot of stuff at build time. Of course, in Dev mode build time and run time is the same thing because we are already building the application at the same time, but if I would, maybe even clean package, so I would actually compile my application, Quarkus would remove 65 CDI beans which are never going to be used inside the final produced jar, in a final produced application. So this is like 65% less, 65 plus is less memory and less processing at runtime, just to give you the idea. Okay, extensions itself. We will get to it. Okay, so I am here in the right side terminal, I am in the same directory. So here I can run again that Quarkus X which is short for extension, which will print all the extensions that I have currently installed in my application, which is not totally true. Let me restart Quarkus here, so you don't think that I am telling you something wrong because you can see here that I have that CDI, Small Right Context Propagation Vertex, which are some extensions which are always included inside Quarkus because CDI, for instance, that Arc is integral component for Quarkus itself. Quarkus internals are wired through that Arc container too. Vertex, for instance, is a reactive set of libraries which is basically the core of Quarkus. Everything in Quarkus is reactive and we will get to it. So this is why we have only REST, easy reactive because this is the only optional one, which we edit on top of that when we generated the default application. If you type dash i here, which is installable, it will list you all the possible extensions which you can add to that Quarkus application, which is a very long list because we have only one extension. You can play with it, but we don't have time. There are categories you can search for particular keywords, et cetera, inside the terminal itself, or if you are lost, you can search inside that code that Quarkus.io or what I actually prefer because, sorry, I need to go on Quarkus.io because typically, if I don't know what I'm looking for, I need to find the documentation of how to use it. So for instance, when I was trying to learn cache, I can just search and learn documentation which is basically official Quarkus documentation that consists of set of guides which are step-by-step instructions ranging from typically 15 to 30 minutes, but some of them are even 60 minutes on a particular extension, how to edit to your project, how you can configure it, how to use it. So after that 15 minutes, typically, if I go into data cache, after these 15 minutes, I have the idea how to edit to my project, how to use it inside my project, somewhere, blah, blah, blah, somewhere, you get the idea. I personally, yeah, here, cache resolved. I personally really like this kind of documentation because in an interactive way, you will really create an application by in which you can continue experimenting after you are finished with this guide. So typically, for me, if I don't know what I'm looking for, I am going to the Quarkus guide. But you have the idea. So if I want to add an extension, you have directly a command here or you can just edit to your application, but since we are using the command line, I can use it here. So I will add, for instance, the one that I am maintaining which is more I helped and I want to show you a magic. You see that just by adding an extension, Quarkus also restarted and now I have new extension installed in my application. And actually, this extension comes also with some API, slash queue, slash help. So now I have exposed a new API. If I remove that, or before I remove it, I will show you both actually that Quarkus extension add is doing. It will basically just add the new Quarkus-smaller-I-help dependency into my POMIX ML. But this is enough to restart the dev mode and you have that extension installed. So again, not stopping the dev mode continuously running, I'm just experimenting any style. I can remove it and hopefully this will just remove it and after the dev mode starts, now I don't have that endpoint anymore because I removed the extension, which was ending. The funny part is that, let's take that cache one because I don't remember any extension. I will just copy paste, or maybe I will just copy paste this because I don't have that cache in the latest version, I think, locally on my computer. So yes, you see that even I don't have it on my computer and what Quarkus does, it will dynamically downloads it from Maven and it still works. So not stopping this really for the duration of what you are doing. Okay, so let's remove this. Some more fun stuff. One of the very good features of Quarkus, after this, what I was showing to you is framework for Hibernate ORM, for database access, which is actually called Hibernate Panache, which is a framework done by Hibernate Team, which is not in JPA, but it's something which is the preferred way how to do database access in Quarkus. And with this Panache, you have basically two options, how to implement the entities, active record pattern and the repository pattern. So if you are coming from Spring, you can do very similar repositories in Quarkus. But what I want to do is, because I can just add the dependencies directly into my POM and just save that file and Quarkus will still restart in the background, I want to show you that what I basically added is just that Hibernate ORM and I added a driver for Postgres. Just because, and this feature that I'm going to explain now is called the services, just because Quarkus sees that I added a driver for Postgres, I probably want to connect my application to Postgres, but I didn't configure any connection to Postgres. Quarkus goes ahead and starts a new Dev Services container for Postgres for me. And this actually started Postgres container in the background. And if I press C here, you will see that it started a new inspiring Gagarin container for me. And it automatically injected configuration that automatically connects to the container. So now my Dev Mode application is actually connected to a real Postgres database running in the container, which is tied into the run of the Dev Mode. If I kill now the Dev Mode, it will stop the database in the background. And this feature, Dev Services, it's something, and now you can start typing basically any codes, Avenger, Extents, Panache, and maybe I need to reload, maybe. Come on, Panache. Oh, come on, Panache, entity. This needs to be entity. This is what I'm typing here, is that active record pattern. So basically, we are encapsulating all database operation into the entity class itself. So I can just type my properties in a public fields because I cancel string name, public string, civil name, and public boolean snapped. And this is everything that I need to do for my entity. ID is generated because this Panache entity contains the generation. If I don't need to extend it, of course, if you extend the Panache entity base, you can customize it, but you don't need to. What is happening? Go back. And because Clarcus is doing a lot of processing at build time, we actually can, like, not force you, but we can tell you to use public fields because if you have private fields with public getters and setters which are just getting and setting, it's just public field encapsulated. What Clarcus does at compile time because it can also generate bytecode, it will first thing, generate getters and setters for you in this class, and of course, default constructor, like false hash code, et cetera. So it's a valid entity. And it will rewrite dynamically any accesses to this public fields in different classes to use that newly generated setters because you can dynamically transform bytecode.  Let's do it at path full, and this will be a list of, come on, list of Avengers. What is happening now, computer? Get all, and a lot of exception. Because everything is now encapsulated inside that entity itself, so select stuff from Avenger entity is just list all.  If I can type Avenger, this is not autocompleted. Of course, we need to still handle transaction, but what I can do is just on the Avenger itself, call some methods which will basically encapsulate the entity manager operation. So if I want to persist something, it's just Avenger that persists, and I can now return the Avenger. We don't need to play with this that much, but you can imagine that, for instance, that persis operation will actually do the persistence into the database, so you will get the ID generated, et cetera. I don't have that much time to show everything. So if I hopefully, and I need to move this because I want to have this on path slash, and we will move hello on path slash hello, so we will get, let's do Avenger. So if I now call 8080 slash Avenger, hopefully I should get a list of Avengers. Yes, I am getting a list of Avengers, which is nothing, but with Panache, all you can actually do inside the dev mode, create the file in certain domain resources called import SQL, and I hope that I have Avengers import, I have, where you can place any SQL that you want to get executed when the Panache is bootstrapping. So hopefully, if now it starts, yes, I am getting a bunch of Avengers, but I don't have the correct extension, and this is the one place that it sometimes break, so wish me luck, and it broke. So this is the one place that, because I am rewriting the extension, which is encapsulating in the background, thanks to the REST easy reactive extension, REST easy reactive Jackson is just adding Jackson support on top of REST easy reactive. I need to restart the dev mode, but at least I can show you that because I killed the dev mode, it also killed my Postgres database, and now a new Postgres database is starting for me because I am restarting the dev mode again, and now I have a new Postgres for the new run of the database. And this time, hopefully, I am getting back Avengers in Jace. Okay, so this is basically, the dev services is automatic management of test containers in the background, so basically this works out of the box, and because I don't need to remember this, I can just type here the services. This works out of the box for a bunch of stuff, of course, we are adding this continuously, so it works for all the databases, like Postgres, MySQL, I don't know what else is supported in the container itself, MongoDB, as you can see here, but it also works for Keyclog, for instance, if you are working with like OEDC, for instance, it will automatically start SQL for you, for dev mode, for IMQ or Kafka, it will start Red Panda, Broker in the background. So it's very streamlined development where you don't need to care about the infrastructure that is basically you are depending on, it is automatically managed in the background for you. And these dev services are not only limited to dev mode, but also if you write any tests that would need to access the database, actually that Quarkus test will start a new Postgres container just for that test execution. So every time that you are running tests, if I would run continuous testing where I would be persisting something to a database, it would run another Postgres database, which is just used for the testing. So no more like application development against Postgres or MySQL and testing with H2. Now you are testing with real database. Go ahead. Yeah, it would start a new Postgres just for that test execution. Okay, so one last thing, because we don't have that much time, but if you are interested, I have computer we can sit somewhere after the talk and I can show you all the stuff. One other stuff that I like personally is this REST data extension. Thank you. This is exactly what I can do in five minutes. So this is basically extension, which is generating REST resources based on your entities, which is, as I learned, 95% of application development in Java is crud over database. So what I can pick here is basically because I already have JDBC and REST is reactive JSON, I can just copy paste this ORM REST data panache extension and paste it into my application somewhere if I can find, save this. And all I can do now because I always forgot this when I'm talking, so I will just copy paste entity, panache entity resource, okay. I can just create a interface, call it Avenger resource and this will be interface that extends panache entity resource of my entity and of my type of my ID. Again, I need to give idea and hint that I have a new extension and that's it. I think that's it. So still nothing? Okay, now we work. I will on purpose remove this Avenger here and I will remove this here so we no longer have, okay. At least I can show you that if I save the file when it's still not compiling, the mode will not crash. It will just print you the error and give you an opportunity to fix it. So what did I miss? Oh, this looks good. Good? No. Avenger resource Java. Oh, I have, what am I missing? No, it shouldn't have needs for part, you know this one. This is like Java. Cannot find symbol.  This will probably not work because I don't have that extension downloaded because this is kind of on purpose that Quarkus will always do release like one day before I have this talk. And I will run into this kind of issues. So we will give you the second but basically what this does is basically it will create that CRAT operation rest resort over our entity that I will pass there out of the box. So now I should still have that Avenger but now it's coming from the generated resource and you don't need to trust me because, so this one's more open API. I will just add a new extension which I also don't have probably downloaded. Come on. My computer is somehow slow today. Sorry about that. Come on. Okay, any questions? Until my phone internet will go ahead. There is a separate extension for several extensions actually for database migration. We can talk about it later. Now it's taking a little bit of more time because it needs to restart the database if I'm adding a new extension because it's a broader, even I'm out of time. Okay, so thank you. I will just show the last thing. Just so you trust me, small open API, come on. Where are you? Where is open API? Oh, come on, I cannot find it. Oh, okay, I cannot see it here. Okay, let me do it one more. Okay, I don't know what is happening. So I will show it after the talk because we are out of time. But basically, you need to... Wait a second. Give me two minutes. You need to believe me that it basically generated a new Jaxarys resource with all CRUD operations over the database, especially as, exactly the same way I was doing it, that greeting resource. Just this is doing it out of the box. So basically, the CRUD over the database is two classes because I don't need a greeting resource. So now it's over. Thank you for your attention. Can you hear us? Perfect. Okay, hi everyone. Thank you for attending this talk that will be about securing Python project supply chain. So a few words about ourselves. My name is Maya. I'm a software engineer at Red Hat in the emerging technology security team. And hi, my name is Fridolin. I used to work at Red Hat. I used to work at Datadoc. I'm an entrepreneur now. And you can find more information about us on Twitter, Mastodon, or GitHub. Okay, so let us start this talk with a simple question, which is why protecting your supply chain actually matters. So if you follow open source security or maybe Python news, you might have seen this information pass, which is that not too long ago, PyPI maintainers decided to temporarily deactivate user registration on the index, and also the upload of new packages, of new projects, because they got overwhelmed with the volume of malicious packages that got uploaded on PyPI. So they couldn't handle it. Everything was very overwhelming for them. So they had to suspend all those uploads and maybe a little more, less recently in January, you might have seen that PyTorch, the famous machine learning library was compromised. Some nightly bit of PyTorch was a victim of a dependency confusion attack. So we'll see what a dependency confusion attack is later in this talk. But this is also a supply chain attack. And this is not a coincidence if you have seen all those articles passed recently, because a study showed that year to year, for the past three years, supply chain attacks have increased by more than 700%, which is very high growth rates. Supply chain attacks can cause a lot of damage to an organization or a project, can be both financial and also reputational. And if you have a weak supply chain, this can get you into a lot of legal and compliance issues as well. So some new regulations were put into place to try to secure software supply chains. And one of the most famous ones was issued in 2021 by President Biden. And this was the exe