Tessin, Tessin, one, two, okay. Should I be close or? Wait, just, you know, don't go too far. Oh, okay, okay, just stay, if I stay in this area, okay, yep, absolutely. Should I start? Yeah, you use my keyboard here. Ah, to point to me. Good afternoon. Good afternoon, folks. Welcome to the session. Our first talk is on lessons after two years of micro front end adoption by Ida Ignatovich, who is a principal software engineer in the middleware process automation group at Red Hat. Thank you everyone for coming. I'm pretty happy that there's people who are interested in like me on the web technologies and front end technologies.  That's why I joined was at the developer. But then again, interested also in technologies and what I'm gonna show you here is how my team develop. What I work, I work for this community that's called Key, is Knowledge is Everything, basically is one umbrella for a lot of Red Hat projects that behind the scenes, they have some sort of AI. Our base framework is pretty popular in Java community. And we have also JBPM and Optaplanner. And now we have the Kojito, there's new project that is basically take everything and put in a cloud native fashion. And also nowadays, since January, I think, I'm also getting a lot involved with serverless workflow specification. Basically the problem that my team is solving right now is when you have a lot of microservices and you want to orchestrate that Microsoft to the coordination. And so there's a lot of bunch of stuff. But as I said, I am mostly now a tech lead and architect for the tooling team. All this team are responsible for the architecture of the team. And what I started, because this is the funny part about Red Hat is that I was joined as a Java end and they say, I do have this graph API to do, to do the graphical things. Basically most of my work now is on graphical editors for BPMM, for GP, for BPMM is the thing at the front and then the back is the DMM. DMM is more for writing your rules of your application. For those that is not involved in the financial community, most of the banks runs this process on behind the scenes whatever and start something where we start a credit dispute or start to do a approval to buy your house, you're running inside a process. And I think that I do a do graphical, I used to do graphical editors to business users write their applications. So basically what the project does now, short way is to take this thing, graphical things that the business people develop and reconvert to Java code to run into drills. And as I told you now, I'm more on the service workstation, I workstation of micro services. But before this, I joined Red Hat almost 10 years ago, I started to write an application called business central. Business central was my, it's not my big it could become a gigantic, but I use it to call business center is a lot of conference as our beautiful monolith is like a gigantic monolith that we are really proud of to have idea on this talking about the part that of this front, this application that do a lot of altering for business uses a full suite. We have two millions, more than two millions of web code, just code that becomes JavaScript. We have five projects. We have more than 150 sub projects and these five projects. We have more than 30 developers involved. And what I was really more proud of is that this code base has more than 10 years old and was still working. Is it still working nowadays? The same code base that we are able to survive a lot of things. And the trick that I did to my team did to make this work because on the same web application, on the same gigantic monolith, we have applications that were written in Angular, that are written in Google Web Toolkit, that's the one that started 10 years ago, and React and everything runs on the same application, but was a gigantic monolith. I talk it in a lot of conference of how to do this. My motto in that moment is that how you can do your web application to survive more than 10 years, because one thing that for those is not so involved with the front end. People has this culture of, he writes everything every three years because there is a new framework, there is new trend, there is new technology. This is one thing that I love about front end is how fast is the pace of innovation. Some people doesn't look like much, but the thing is that there is a way to write applications and we did a lot of effort on this. But then the world changed. There was a avalanche of the software development with the cloud native. The everything that started with Kubernetes and when Kubernetes get really popular, put a lot of pressure over the Java. I am also a Java champion because I have a lot of Java background. So put a lot of pressure on the Java industry. There is any Java developers here?   So the Kubernetes puts Java back on the game. For the Kubernetes, Java runs pretty fast nowadays. But this also, because our projects are based on Java, this gives us our opportunity to revisit everything on the rules and JBPM platform to become more coordinated. Basically on Java, people means that to take all the engines that have 15, 20 years old to port them to Quarkus and change everything. For instance, instead of having one server that runs hundreds of thousands of process for a bank, now with new Kogito technology, every process become a microservice. So we change a lot of how we write backends. And that is moment that I told my boss, my Mark Proctor, that is the time to change also how we do web. So we try to think about how to modernize web development because we have all the architecture. So we started to think about how we can modernize how our platform, instead of monolith, what would be the next answer to survive ideally on the next 10 years. And we started to think about architecture because for me, architecture is a way for software. One of the things is that when you take a big problem that is really ugly, like it's complicated, we start to think how to put beauty and put on this. So we start to think about the problems. We have on the beginning of this business center that this monolith started me and my two more persons. Now 40 on my team. And so the technology also changed a lot. We used to have just with Google Web Toolkit on the back end, still have a lot of things there. But there's a lot of new frameworks, things that. How can I break up my front end monolith in a lot of smaller management pieces? How can I take a big monolith, there's no application and figure out how to break this down. And one quote that I have here is about, it's from Huff Johnson from the design patterns book. He's a writer of the design patterns book that architecture is about the important stuff, whatever it is. So what is the important thing in my application? My application was a web that give business users and now developers and servers workflow to write the application. So the most important thing was for me, the editor and the outer experience. So we started, I put two years but it's more, like three years ago, we started to think and we started to do something that the industry later called micro front ends. There is no micro front end movement yet because the marketing, it's not the marketing, or marketing following blog, they publish the coin this term of micro front end but we are doing, I think on the book of micro front end in action, there is the history of how the technology involves on this. And what is my micro front end for those that doesn't know? It's a architecture style. We are able to independently deliver where front ends application are composed in the great whole. So instead of having a big monolith of a single application, we break down multiple smaller applications that are combined. When the best way to explain this is when you have your old back end monolith application, now you have a lot of micro services that are collaborating for the great whole. We are doing the same, but for front end. What are the promise of the micro front end movement? It's not so easy, but what are the promises that they do? They promise incremental updates. So instead of having to rebuild the full application, you are able to do incremental upgrades in smaller pieces. You can have a simple and the coupled code base, each micro front end can run as a standalone. So your team can be more and more focused in just that thing. For instance, one problem that I have in my team in order to fix a bug inside our editor, specialized editor, they have to build and understand how the full application works. My promise for you independently releases and the things become more autonomous. So this is the way that I understand micro front end. The size of the components is up to you, is what depends on the need of, but based on how our sector is in one of the products that we have, the header is one of micro front end that's part of the foundation. Then we have a decision graph that is another application. Then we have the editor. This editor is the same one that's running for 10 years being developed. So there's a lot of old legacy technology running here. Have this part, it's called a container app. I prefer the word container, but on the micro front end community app shell. We have a app shell that for me is more than a container and you have a lot of micro front end that's running inside. And how this architecture evolve. On the left here, you have the monolithic web application where you have a data store, we have a backend and we have a front end. Then backend teams started to see oh, it's better to have more management, a better life. It's the same advantage that I put on the micro front end. They, in a lot of with a single API, that could be a BFF, that could be anything or it could be a single end. Have a more that is talked with a lot of micro service. What micro front end propose is on the right to have a lot of micro front ends, talking with the API that talk with the micro service. So they take in the micro service movement on the backend and put for the front end. But for us, what this means? Now imagine that that box is all the micro front end that just split for a better explanation that we have a container application that is what when we use it at this point, we will handle the front and the container application and the container application after that we will load all the micro front ends put on the right places and talk with a data store. Here I have a BFF that is a pattern to abstract the backend. It's called backend for front end is basically an adapter of a data store for the API from a micro front end. A rule like when you talk to people, they're really strict about my 10s that this is something that we can discuss later but it's not practical in my point of view that they say that a micro front end should never communicate directly with each other. This means that when a micro front end wants to talk, the A want to talk to B that you go for a database in your cluster and then the other micro front end react. This is what's made to make the micro front ends being the couple, okay? There are types of how to do your micro front end, how to build your micro front end applications. The first one is how people start to explain how the micro front end work. This is Microsoft and micro front end in the ideal world is on the runtime application or on time integration. So basically your team do some op-edating library, publish this library and then the container application fetch the source of this library or the component is like load the jQuery in the next of your page you go, do a HPP call, take all the source and send the component to a div. This is called a decline site integration. So after the container gets loaded in the browser you take all the source of the micro front ends. The pros for this is that we have the grow out of in the planned deployment. So when a container is one machine, the component is outside and external. So if the component get update, after you do a refresh and validate your cache, you get a new version of the application. So the teams can evolve and your application can evolve independently. What are the bad parts of this? The tooling, the setup is much more complicated because you are fetching remote resources, you are fetching different applications leaving in another teams and making this deployment independently requires a lot of care with your test CI. We are test your CI and your continuous delivery. For instance, how you manage your content running, I have three versions of the micro front end running some, which version should I fetch? So this requires a lot of discipline in your team and is much more mature. But this I put this first here because this is how people explain the advantage. But coming more for the real world, this is more how the status quo that I saw in the micro front end industry is to have a build time integration. So the applications leave isolated can be independent deployment. This is how my team does this right now. They leave in the plant, they deploy the plant, they leave alone and they don't know where the application is run. Each team developed the micro front end that is for, they don't know where to be running. After they want to do a release, they publish this on the NEPM. NEPM is like the repository of libraries for Java, for Java scripts, it's like the Maven central for Java applications. So when I'm a container application, when I'm building my application, I put the page and fetch the source code in the build time. This is much more easy because the setup is kind of other options does. So instead of having your monolithic JavaScript application, you're consuming the components of your application as a NEPM libraries. And this requires gives us isolations. The problem of this is that I don't have the live reload that one of the problems that people sell when they're talking about my end that when people do a change, I have a lever load, I have to rebuild the application. This is what we do. I have one container app and I have here in the lab is one editor developed by one team and that are doing. And why people doesn't that way? The runtime dependency is the grow nirvana of the microphone 10. Why people are still doing this? Because there is some concern of running, and I'm gonna talk about the bad parts a little, concerns when you go with a microphone architecture. One of the promise is that each team can run autonomously, it can run in isolation. So they can have a smaller, a quickly build. If someone here have a big web application knows that take a lot of time more than people imagine to build. Like the monolith application that I showed you take between 15 and 20 minutes to build the full application, the full suite that monolith on the thing. So when you go for the microphone 10, the DMN team can show you can build super fast. They are more focused on their problem they don't know about all the things that my application was running. They have less distraction and noise. What's the problem of having autonomous team? A lot of bugs only appears when the application is set up in full. So the team on isolation, things that is working but they can when you pack the application, the bugs appears. It's hard to run the complete experience in order to someone from QE or someone that is valid application. They want to build everything and test it hard because the team is working on isolation. It's hard to debug applications that bugs that go from one application to another because the applications leaves in isolation when a bug go from one to another one. For instance, one application publish a Kafka message and that Kafka message trigger a bug in the other application is hard to debug the scenarios. You can have in current experience because CSS can change, the wax can change. So you have to have more care when your team and autonomous to have a central point of focus. So there is some tricky issues. Another problem that is the big one on micro front end is regarding the styling, the CSS. Shadow comes, a lot of technology try to take the CSS. There is a lot of things that the modern JavaScript community does to solve this problem. Like CSS and JS library, there is a way to put context and only apply your styling for a specific component. But the CSS libraries, what they were designed to be global because they are custom style sheets. This means that if you take especially old is build not in the most modern JavaScript technology, there's a chance that your JavaScript CSS library or JavaScript framework will provide side effects over your don't tree. And what is the issue? The bugs, when developers develop an isolation, everything looks fine, but when he loads with another micro front end just in production, this micro front end has a CSS that will affect the other. Could be important, important. Could be something that is really changed, or could even be different CSS versions. For example, I had to use pattern flight. If you put pattern flight three and four in the same don't context, they will break. So CSS styling and isolation is a problem if you go with micro front ends. There is a way to fix this. The one do is to use CSS and JS in everything. The other one, if you really need to use a old JavaScript framework or old JavaScript CSS framework that is not CSS and JS, and if you are able to rebuild it, you can put scope in the CSS there. I did this in the past, but I don't recommend. The solution is true. One is the shadow DOM. There is a new application that is adopted for some browsers, but it's not the default everything. And my favorite one, that a lot. And when I start to collect two years ago, people throw eggs at this moment at me to use iFrames everywhere. iFrames is, in my point of view, the ugly duck of the web development. Learned like in 2002 and three, that if you are doing iFrames, you are doing wrong in web development. iFrames are the best tool to provide context isolation in the browser. So when you put the iFrame over application, you are sure that no context will leak everywhere. You are safe and it's the best way for you to have the couple of styles, to have global variables and to have a lot of things as a shadow DOM is not an option. And how this works, every time that you're going to render, this is the example that they do, they just, if you know Arching Fowler, I always do this as a pill of authority. It's a kind of a fallacy. Two people don't throw eggs at me because when they put Microphone 10s, in the first sample, they put a iFrame over there to protect. So in 2017 and 2018, you are handling the Microphone 10 inside iFrame to provide context isolation. And how we render my web application. Everything that you see later, how I do this, you imagine that it's wrapped by iFrame. So before handling every frontend, I put the iFrame. To give example here, a concrete example, I'm running pattern fly three inside the DMN editor here. And I running pattern fly four, that's more recent in outside outer application. The only way to make both works is to put a iFrame there. But another appeal of authority, I'm not alone, if you take a look on your Google chat, the Google chat there is actually, there is a iFrame protecting. And if you take a look on the website that you use, most of them, there is a iFrame here on there adding the iFrame is not a really good technology, until you have shot on don't adopt the bar ever came for the Microphone 10. If you don't put iFrames, you have surprising issues in production. This is the only true right now if you want to adopt a Microphone 10 technology. And one thing that to advise you, I'll give you my context later, my teams fix it a lot of issues that you face when you put the iFrame. For instance, the only way for one application, talk with the outer application via iFrame is with the postmasters. And this only passes strings, string, there we are request response model. We implemented a lot of our technology to fix this. So we basically create kind of reactive hooks so keeping contact with me, I can phone to you but a little advance for this talk. I was talking about it to where we develop or where I get contact with the microphone to understand everything. Why? Why do we are trying to actually build?  They don't do application. They want application. They want to put the editor that my team develop inside of application but they want to put the editor inside any web application. They want also to put the editors inside the visitor code. They want to put also in the desktop applications like Electron and inside the GitHub pull request. Have a visual graphical. So instead of having my microphone 10 running inside a single container, a single app shell. Remember that container is the one that handle all the microphone 10. My problem was a little bigger yet. I have to render my microphone 10 in multiple containers. It's not just one application like most of industry does right now. And this is why we created the multiplying act by us is a way that we do microphone 10 in my team in other teams at red hat is already using this. So basically what I did is every time that you see the word channel right now is when I talk about the container or abide up shell. The things that the main architecture difference of my approach microphone 10 that the blue, the BPM editor is the microphone 10 that's running in an outer application but I can change the outer application. I can change the yellow box. So I create in direction that is called a envelope. So for the microphone 10 he's be handed inside the envelope and the envelope is running inside the channel. So without a single line of code, remember the promise of the microphone 10. I was able to put the editor inside the GitHub because the channel, my app shell is the same. I was able to put inside the same microphone 10 that was developed there was developed inside the front end here supposed to put a video. I don't know why. Yeah, there was no internet, sorry. But it's just a video of running inside the business code. We don't think much what we do basically. I just put this to be in terms of the promise that I'm working on is that we have the app shell that is the iframe that I show you but the iframe doesn't know where it's running. So the iframe gives the envelope, gives the microphone 10, the ability to be handed everywhere on your GitHub repository or in staff.cubismarge.org if you're interested. This key tools is where it leaves all the code to have MonoApple there including the iframe technology. But coming back about the microphone 10, when you have to do a decision of going off microphone 10, basically you have two sides of the spectrum. If you go to strategic collaboration, your teams will agree on the tech stack. Your teams, you have a container that you render or handle all the API calls. You have a shared dump components that you reuse to keep the CSS and visuals in the same, in the same way. And you have a shared CICD, everything. On the other side of the spectrum, you go for the total independence that you go. Each team control the tech stack so you can have Angular, you can have React, you can have everything. Each team do their own decisions like the microservice. Each microphone 10 figure out how to do the API calls. The app is just composed in the runtimes and each microphone 10 has the own CICD. The right gives less fix, the right of strategic collaboration give the team independence but less flexibility. The total independence give the team less flexibility, developers happy, but you have more issues when you do the integration. And what is, what I use this to talk about the next trend in my 10 technology. Basically what I'm doing is a tool called Dash Builder. Basically Dash Builder is a way to do graphs, to create visuals applications. So when I talk about this for other people, I say, oh, is that too similar to what Grafana does, what Prometheus does to write your graphicals. But in the end, what my team builds is that I have this way of compose microphone 10s to put microphones on everywhere. So goes for the next question. Why I cannot create in a YAML a declarative way to put microphone 10s over a web page? So basically every box of this image is a microphone 10 that are composed via YAML. So basically I'm creating a descriptor to be able to create new web applications based on the component. So I put the microphone 10s for the next level. What are my use case for this? Imagine that you're going to write your own dashboard with the editor with your logo of your company. A lot of my clients wants this. So you basically take YAML, reuse our microphone and put in your, and this there is a website dashbuilder.org that has this. But the problem is, remember when I talk that I do only build time on my projects that my components know when they are everything that are on the application and build the application as a single bundle. Even if your application is developed independently. I cannot do this because this is running inside the browser. Basically I take all the declaration of a YAML creating a container page and fetching the resource remotely. What is this? For instance, that graphical on the middle, there was three different microphone 10s that use each charts, okay? Each one has an eye frame and each one was loaded in runtime. How many times fetch it, hand it and put each chart in the DOM? Three, right? Because every component lives in isolation. So in order to handle the first one, I have to unload the charts and put the charts there. The second one, again, as is correct because I put the eye frame to doesn't break everything is not shared, everything build the isolation. This is a problem of run times with eye frame because I have duplication of my library loading. In build time, we know how to do this. You put as a dependency on the application and the webpack you figure out and you only put react in your application even if you put react everywhere. But the problem is when I have this, this is my container application and I have three microphone 10s handed there and those microphone 10s are created in react, I will download parts and put in the memory of the browser react three times and this put a big penalty in any web application. So there is new technology of webpack that is called federated models. This is the last trend of what modern web developed. Which problem webpack solves? Webpack is a way to build your other script application but in version five, they provide a federated model. There is a way to do runtime integrations of microphone 10s in build time. So when you build your microphone 10, webpacks put the source there in the dev mode.  This means that react is on the window. He will not reload in the bundle again. So it's a way to load only once. So this means that if I build my microservices in isolation and run webpack, you identify in production, there is no react and then we will fact-react lazily. But if I put in a container that is already has react in the memory, we will refatch the react again. So basically what we are doing is this is a demo that I do to do my exercise. We have a home page and we have a little deep inside webpack right now. So hold it to me. I can, it's too many to explain how this works. Instead of do here on the line four, five, six, I'm doing the reports, but see the imports using a special lazy nomenclature. I'm using online 17 and 13 a suspense to only fetch the components when the components are loaded. This is until so far is everything react. I'm just doing lazy loading of my react to doesn't load everything once. On my webpack, I put online three, the faded plugin that this is the magic of the new webpack. And here I put my dependencies with the plugin on line 16 and line 19 and 21. I'm not fetching a resource, I'm not putting a dependency. When I hit them the point of a remote called the marketing, I will fetch marketing at a domain. This is a variable to be able to meet you change between local hosting and production and then a remote entry point. So instead of putting the dependencies, I'm putting reference for the dependencies. This means that now I will show you how one microphone 10 is handling. Microphone 10 have also the import of my model federation in line two and see online 17. It exposes a app and expose a bootstrap. So when I put here a remote entry on my container application, I'm put a pointing for this expose of a plugin where a plugin is kind of a manifest, a plugin tell remote federated models that I'm serving these assets and when you need you can fetch these assets. When I'm running a app, I see that I load here my container. Instead of loading my login page there, I put in a remote suspense. So you see here a load just for the remote entry. So my app learns that when he needs to fetch that resource, he should look at remote entry. As soon as I click on the application, you fetch the remote entry, see that you need to take the resource and you don't load the restores for another part of that resource is hosted. Basically what Webpack 5Ds with federated models is to put in production a lazy way to fetch scripts. But what is the magic? Remember the problem that I have one micro front end and another ones that share the same library and I have to load the library a lot of times. See the difference here. Here is that I'm running on the right the micro front end only in isolation of the off. See the number of libraries that it loaded. When I'm loading the same application in the container of the micro front end with the host, do it with the federated models, identify that all that libraries is already loaded. So there is no way to do a fetch, no way to do a typical to fetch the resource, in no way to parse, making my application much faster and all this works, the heat for you is this 19. 19 is that here is from Webpack out, that I am sharing all the libraries from my package JSON. Here's the package JSON of the file that I'm loading in line three, required all the packages of the root. So basically I'm telling the federated model that I can share all the libraries with my host application. Now hold your hand. This is how the architecture of this works. You take a container and you take a provided models here. I think I can show you the mouse here. When a provided mouse needs to be fetched by container, it go to the heifers, go to the model, check to the version, instead of fetching from the back on the version, if there is a shared scope, it doesn't do that. So coming back from a problem, I'm able to have a lot of micro frontends develop independently and each one of them and with federated models, federated models is the, in my point of view, is the solution for runtime integration, solution of to refetch a lot of application and make bloated. And the beauty is that each team can run and deploy the application isolation because if the correct is not there, you fetch, you can split all servers and we finally are able to do the real, the, always there is the cons, okay? What are the cons? This is the bleeding edge of web development. If you, my pull off of guessing ball, my guess ball, if you take all the JavaScript community developer, only 10% or 20%, really know how web packs works. Because it's complicated. Usually it was one thing that a person, that is more senior in your team, when it started the project, did all the setup and most of the developers never told you that again, because it's like for the Java word, it's like we change the phone file. Just some people know how this works. And this put a lot of dependency off your architecture and a web pack and people need to understand better how web pack works. It's complexity, it's a lot of push pressure on your CI CD in order to have the runtime. You have to have a good CI CD, can break sometimes your boundaries if you have multiple loads of different applications and also doesn't fix the CSS isolation. Remember that the CSS isolation that I told that's one of the big problems that you put iframes, web pack and federated mode doesn't fix this problem, yes. This problem is either fixed right now with iframes or with ShadowDom that is not implemented yet. So on micro front-end architecture, when you are deciding to do micro front-end architecture, you can, part of the problems are solved, but this problem of CSS isolation is not solved yet. Remember with my talk that root front-end development is really hard, it's not so simple because applications are really complex right now. And the most important thing that I learned in my career is that your favorite web framework will not be here forever. If you take a look at the internet, I already told that Google Web 2 Kit was the best framework, then Angular is the best framework, then I told everyone that React is the best framework and now I think I am like in love with the next GS. So we have four trends already in like 10 years that work in the front-end. So you need to be able to design your architecture in a way that you will know that if your application will survive more than three years, probably you need to have multiple versions and frameworks running inside web applications. That's why my team is so invested in micro front-end because it's one way that the JavaScript community is thinking about how to have multiple dependencies, multiple small applications running together. Come back again, if you have to do this decision, remember of this slide, you have two ways of do micro front-end architecture. One you go with teams doing strategic collaboration when your teams are friendly, they live in the same building, they live going the same calls, where they can live in the pendulum, they agree in tech stack, but progress alone and then we pack everything together that built to the Nirvana that there's still some issues, but it's a way that this technology and architecture use really see the potential some years for now. Thank you so much, and thank you so much for your time. This talk is really dense, is a lot of content for 15 minutes. And I am on Twitter is at IGN, this is the thing of Red Hat, but you can find me on Twitter. The repository where you live is, let me go a little slides, this is the most funny thing to reach us, is keygroup.keytools. This is the rub repository where our tools are developed. And thank you so much. Questions? Yeah, so talking about like security perspective, did you face a challenge in making sure like every micro front end reaches a certain benchmark of security so that it doesn't become an entry point to attacks? This is a big problem because it's a problem of microservice because when you put everything in the micro front end, your latency of website will be always more or less the latest of your slowest micro front end. One thing that I do in my teams is to put something called fitness functions. Fitness functions is something like we do in a verification of the poor that every time that you go a request for it's in our editors, a big file in our editor, part of the CI CD and the way to load in less than five seconds. If you are not able to load, we will break your poor quest and make it work. So fitness functions is a way to you protect but the problem is those fitness functions only works in when you have a good CI CD pipeline. So if you go in micro front ends, you really need to know and have put a lot of investment in your CI CD to make this automatically. The second challenge of micro front ends is to have observability in production. When you are doing the build time and put everything in the same bundle works amazing. But if you are focused on this issue for you, for this you need observability in production. What we do with both fitness functions endpoints so we watch what's going on with the web servers with ping people in the back. And if you are front end people off back and they are better off doing this. So what do like for my people that run Java big applications in production I ask them how you monitor things because they are the same problem that we will face right now with the micro front end. So what to do in summary? We monitor in production. I don't have this problem yet in my production because everything is a build time. I don't have the latest but I will have with the dash builder right now. It's in production but not production. It's just for the community. But on the micro measure the timing of one micro front end and now so I put integration test in my repository that makes sure to test at least one happy path of the whole application integrated. So if something breaks and measure, I can have at least a heads up what's going on. Thank you for your time. Thank you for watching me on Friday. Have a nice weekend. There is a hummer here too. To stop the question. Thank you. We have a USB or USB-C. Check, check, check, check. Okay. Check, check, one, two. Welcome folks to the next session. Our next talk in this session. This talk is building a REST API from the ground up. The speaker is Steven Nemo who is a staff specialist solution architect for it. Correct. Greetings. Sorry for that. No worries, no worries. Welcome everyone. As he said, I'm here today to talk about REST APIs. And we're going to take a little bit of a different tact. We're going to build one from the ground up. So just to cover the agenda real quick. Really what we're going to do is a really quick overview of Porcus. I think we've had enough Porcus overviews today, but we're really going to start from the bottom up. So we're going to talk about the repository layer, moving up through the business layer, the service layer. And then we're going to finish up at the resource layer and then we'll talk a little bit about testing. I have 25 minutes. So we'll see. So a lot about me. I'm an energy trading guy, energy space. So if you all want to talk energy, come and talk to me. I've also had some very interesting experience. I also worked in low latency trading, most notably with JP Morgan in their direct market access stuff. So I've done some really fun stuff with low latency stuff. So if you want to talk about that, we can. Or if you want to talk about golf, I can talk about golf too. All right, so why Corkus? Corkus is actually what brought me to Red Hat. I was a spring boot guy for the longest time. I found Corkus and I was like, this is really neat, really innovative way of doing things. I reached out to a Red Hatter local in Houston and asked him a couple of questions. And at the end he was like, hey, what are you doing? Want to come work for Red Hat? And I was like, absolutely. So more than anything else is throw the Corkus stuff out. This is a completely micro-profile compliant application framework, right? So all the goodness that you get with the standardization of all the micro-profile stuff, you get inherently in Corkus. So it's not really learning something new. It's just a framework. I really think about it as just a different framework or different application platform for a micro-profile compliant application. It actually mixes both imperative and reactive styles and you can kind of run them both. One of the most notable things about Corkus is even if you're writing some stuff in an imperative, it actually executes in reactive framework. So there's a lot more coming with Project Loom that's gonna help with all kinds of us getting back out of the reactive framework and the callback hell that you can get sometimes from that programming model and back into a more groggable imperative style that most developers are used to. Anyway, but that's coming. And the other big takeaway is like the huge memory and performance gains, quite literally even just running in a JVM mode, it's about half of both CPU and memory consumption of a traditional spring boot app. So on your application, that doesn't make a big deal, but in an enterprise with 10,000 apps, that's a lot of money, right? In operational compute costs and things like that. So really think about it from a scale standpoint. And if you talk about native image compilation, that actually shrinks it down to like 10% or 17% is actually the formal documentation, but it's a fantastic way to do it. And what we're gonna talk about today is kind of starting from the bottom. We're gonna talk about Postgres, we're gonna talk about the repository layer, move up in the service layer and then finally hit the resource layer at the very top. So starting with the database. So most people say, why would you do that? Don't you wanna start from the very top, build a mock and push it out there at everything else? If you're building a really complex API in which you need like lots of feedback from your community before you do any of the implementation work, absolutely, I'm 100% for that. If you have a fairly simple API interface that you're trying to build like a simple CRUD app just to get something out there and you don't wanna, if you don't really need that feedback cycle, just start cranking. And that's what I'm doing here because we're just building a customer and it's really, really simple. So I'm really not gonna talk about core orchestrating the rest of this. So it's really gonna be talking about all the libraries and tricks of the application layering and things like that along the way. So the first one is Flyway, right? There's probably two, well, there's two main ways to do database schema migrations in a source-controlled manner. My favorite is Flyway. It allows you to be able to again do all of your database schemas, manage them in source control, have them versioned and they can actually version up and run in layers and it makes it really, really easy to manage your database schema. If your database schema is not in source control, you're doing something wrong. Put your database schema in source control today. That will save you a lot of headaches going forward. We're also gonna be using the Hibernate ORM with Panache. What is Panache?  So when you do this repository class at the very bottom that you're seeing, when you extend that, that customer repository gives you all of the finder methods that you can think of just out of the box, ready to go. So really at this point, once you have this in place, all of your CRUD methods are done, now you're just building your super inner joins or other custom logic that you want, that you need to do at the database layer. And then of course, it's just JPA, the basic JPA implementations. So really here, there's a couple of strong opinions. Number one, I do not like the active record model. I feel like that's a converging of concerns. I like having a separate repository and then a separate entity model. That's maybe just an opinion, but it's a strong opinion, loosely held of course. We can always argue about that. And then the other thing is I like to kind of put a lot of, I like to put a lot of text in here. I like to explicitly name columns. I like to explicitly name tables in my entity. If you don't put any of this stuff, it will be inferred for you in that anything when you start getting to inferences and assumptions, that's what breaks in production. 99% of the time it's an assumption or an inference that breaks a production, not something that you're coding. And it doesn't take long to do all of this stuff. The other thing is I like to name my things that are related to where they're coming from. So if you ever look at any of my projects, anything that's a JPA object, I have with an entity suffix on there. It's easy to find, it's easy to know what it is and it kind of takes away the idea of what this thing is. And this thing is tightly coupled to the database table, right? So there is no other use for this thing. It's an entity, that's the way it should be. Hibernate validators in here as well. So you'll see some not MDs and that type of thing. This is a really, really easy way. And I actually have this on multiple layers. So it's at my, two is one, one is none. So I have this at my JPA layer, but I also have it in my domain layer. And if I have a view layer, I'll put it in there too. Why not? It doesn't really cost me anything. So I have that all there. So once we get down from the bottom, now we're moving up the stack a little bit. As you move up the stack, keep in mind that the reason that the architecture, that kind of the three layer architecture is in a traditional kind of web app thing is to reduce your amount of, you reduce your coupling, right? Is to reduce the exposure of the lower level details to the higher order kind of value objects, right? So when you start talking about the service layer, really that's where you start building your domain, right? That's where you have your domain objects, meaning the structure of the way that you think about it from a business perspective, right? And then same thing from the service object, right? The service object is kind of what you're doing, but you wanna do it at a business context, right? This doesn't think about like this interface and this interface really is more related to the business and it obfuscates the underlying implementation of the database, right? Doing this allows me to then, if I need to make a database change later on, I can kind of isolate it down to the layer below it, right? So this has no references to any layer above it at all. The only thing it knows about is the thing below it. So I'm insulating the layers from change. So if I make a database change or if I wanna change the table structure to be optimized at a database level, I don't have to change all my view. I don't have to change my REST API infrastructure, anything else. I just have to change my service layer and it's all insulated. So it represents the domain object. Domain objects in, domain objects out. That's the rule, right? So you're not passing, you're not returning entity objects up. That's a leaky abstraction, right? And I don't want anything except for domain objects coming in. This is actually really, really cool too, because if later on you wanna pull this out into an actual jar itself, which I've done in multiple projects, and repackage it in a later time or use that as just like a business type of jar that you wanna then repackage and reuse it in an event-driven architecture versus whatever. There's a little bit, it's real easy to kind of yank this stuff out and put it in a different context. If it's built around the protocols, and I mean both protocols, incoming and protocols from a persistence layer, it's just all, it's not very fungible, right? You can't just swap it out and move it around. This is also where all my transactional control, context control is there. So anything that I need to do from a transactional context is the entry point and exit point for creating the transaction and coming back is at the service method layer call, right? So this is where the transactions happen. I use, we're gonna use MapStruct a lot for all of this stuff. If anybody's aware of MapStruct, MapStruct is a fantastic mapping library where you basically provide it with some interface, you provide it with some interfaces and some very loose definitions and say map this object to this object and it automatically generates code that does the mapping for you. And unlike regular kind of generated code, this is actually kind of readable type of stuff. I've gone in and like looked at it. Anybody who did like old like EJBC, you know, container managed persistent beans and stuff like that, that code generation was garbage and you couldn't read it. MapStruct is actually really good. So shout out to Gunner for that and his team. So we're gonna be mapping those entity objects to domain objects and domain objects back to entity objects. You'll see down here, right? We're doing some persistence and then we're doing a bunch of mapping, mapping back and forth as we pass in and out, right? And then the valid annotations also are very interesting. So if you look at the incoming method call, right? This valid annotation, which is a part of the hybrid validation framework, this will fire as part of the stack as it's coming into call this method. So if this customer object, right, has an empty first name, before it even gets to here, it will fire off an exception saying that this is empty, right? So this valid really kind of protects you as the objects are coming in to that they're valid before you even get to any of this kind of logic here. Yeah, and again, just got the repository in there and just got the mapper in there and that's it. So then once our kind of service layer is done, they were moving back up the stack, right? Now we're at the resource layer. In this, the resource layer, in my opinion, is the interface between JSON and your domain, right? It's what is your JSON or what is your REST API structure look like on the outside versus what your business domain looks like on the inside, right? This is the insulation as it comes through. But the interesting portion about this is, first of all, it's all Jaxrs, right? So it's all micro profile compliance stuff, right? For all of the get tags and the path and everything else, it's all just Jaxrs, there's no magic here. But the neat thing is the micro profile open API annotation. So within these annotations, if you look at the tags here, right? And then down here when you're talking about the API response, I can define a lot of what my swagger API will look like, what my open API contract will look like. Some people like to write to the open API contract and then do the implementation, right? Have the code generated, do the implementation. I find a lot of value here because it's self-documenting with it really close to the code. As a developer, it's sometimes tough to have to switch back and forth and do documentation in one place and code in another. I kind of like to have it all together. So it's a different way of doing it if your team kind of agrees to all of this stuff. It's maybe a different way of doing things. Again, the Hibernate validator helps us out as the incoming posts, right? We can say not null at valid. So by the time we get this customer domain object coming in that's being serialized from the JSON, we know that it's valid and we know that it's not null. So you don't have to do a bunch of the object not null logic and everything else that you normally have to do there. And yeah, I guess the other detail about this that you kind of want to take a look at is, right? I've gone through in making sure I'm defining all my API responses because I'm having to build out my, I wanna build out my open API spec to really be fully developed, fully defined across it all. A lot of people don't go through the details of defining exactly what are all the things, the known errors that can happen from a 400 perspective. And this helps with documentation, right? It helps the developer. I'm always thinking about when I'm developing an API as the developer, the other developer's experience, I want them to be able to pull up my documentation and easily be able to flow through and see how things work and see all of the error conditions along with that. It's not only about the happy path, it's all of those little sad paths that define the bigger picture for what a good application looks like. And then the last thing kind of about Quarkus and about building this is if you saw Daniel's presentation earlier, testing, testing, testing, testing. I'm a test-first type of person. I like test-driven development. I think it goes a long way. But with this, you can do test containers. And if anybody doesn't know what the test containers project is, it's a fantastic way to dynamically spin up services. Quarkus took it a step further and literally looks at your Maven Palm and says, if there's something, if there's a library on your Maven Palm that needs a database, right? If it has the Postgres JDBC library extension on there, it will automatically spin up a Postgres database for you and fire off all this other stuff. So it's really neat and it manages all of those things. The WI-2 is actually really neat. The thing that I use in the WI more than anything else is the config editor. I like to go in because it not only allows you to mess around with stuff, but it actually gives you the runtime values for all the configuration, even the defaults, right? Again, most of the time, we do get tripped up in production because something is inferred or assumed, right? This gives you a really good view of this is exactly what all the configs look like. So it's pretty powerful. So I do have a little bit of code to show, but more than anything else, this whole talk is based on a blog post that I published back in February about from the ground up. I worked really hard on this to make sure that as you're going through the blog post, it literally is like cut and paste. You can kind of build it as you go. So it's a pretty good resource, but if you just Google cork is ground up too, I'm out there and of course, follow me on Twitter. So yeah, so this is what the project looks like. And from here, if anybody has any questions, go ahead and fire them off. I'll just kind of talk a little bit about the project setup and things like that. And as we go, right. Okay, so the question was protecting others from breaking changes on your API spec. Oh, no, no, it doesn't matter. It's still a breaking change, inadvertent or not, right? And that's where actual service registry comes in. So I like to build a, you know, Red Hat service registry that's out there and that will actually give you feedback on breaking changes when you apply a new version out there. So as part of your CI CD pipeline, I would execute this, I would take my swagger and part of my CI CD pipeline, I would push that out to service registry as a new version. And you can set that schema validation on the service registry to say, everything has to be backwards compatible. And if I get feedback from that, that it's saying that it's not backwards compatible, I would break the build, right? Yeah, yeah, API definitions. When you start talking about introduction of breaking changes into a large ecosystem, especially if you put an API out there and you have a hundred different consumers across six different business lines, all with different prioritizations and you wanna introduce a breaking change, good luck. Right, there's a whole another talk I can do on deprecation schedules and organizational behavior with that. So this is a horizontal only scalable kind of approach, right?  In this type of behavior, it's just did the database transaction work or did it not. And if it didn't, then 503, or not 503, but, you know, 500 or whatever, catch that, give them a good, you know, good error and stuff like that. But there's not much else other than that. Stephanie, the app valid, I just speak up. The app valid notation, when it's not valid, or how do you define what is valid in that notation there? Yeah, so the app valid is basically saying, go look at all of the other hibernate validator annotations that are on that object and run and fire those, right? See if anything's valid or not. So if you looked at customer, right? If first name is empty, it'll fire, it'll fire a thing. And there's a, if you go look at the, so if you go back to the GitHub, right? You can go look at the, I have a whole bunch of exception mapper and there's a follow on blog post to that that actually talks about how to build a universal exception mapping kind of interface with a really good error structure, right? How do you pass those error structures back to the users in a way that allows them to be able to consume them in a normalized fashion? Error handling is murky at best. So having a really good, not only an individual team, but that's the type of thing that you go back to the global organization and say, what does our error structure look like? And let's all conform that all APIs will return these structures of JSON. That way we don't have to, A, and re-emit the wheel, but B, as a, as an API consumer, if you've ever consumed four different APIs all at the same time, because you're trying to build something and you're having to manage four different sets of error structures, it's a pain, it's a total pain. And it's so much easier just to kind of build it once and extend. Yeah, so regarding the difference of the objects and the entities and the business domain, but you were saying that you want to decouple the service layer from the resource layers, but we saw that it was the same customer. So do we expect to have like JSON annotations in that same object and share that between the resource and the service? Or you're allocating for having two different types of customers, the one that it's handling the JSON part and also having the other one with the validation and annotations and domain? Hugo caught me in something, right? That's a very, very strong inconsistency in this implementation, right? If I wanted to take it to its logical extent, I would have customer view objects that represent the JSON. The customer would represent the domain and the database would represent something else. From my experience, the domain and the JSON tend to be more coupled in lockstep versus the database changing because of optimization needs from a query standpoint, right? So it was a choice to have the database for a demo, but yeah, and that's where you kind of get into that like, I only have a minute left, but there's a bunch of development monikers out there, like Yagmi and Dry and single responsibility principles, those are all dichotomies, right? Don't take one and say, oh, it's gotta be, you aren't gonna need it, right?  Do it up front, that way you don't have to do large refactorings in a short timeframe in a future state, right? But then again, from a Yagmi standpoint, like, do I need to build the view object that completely matches the, yeah, I could have, it's just kind of a, it's a very subjective choice. And I wouldn't see anything wrong with doing it from a consistency standpoint on the other way. 25, all out of time. I'll be around for the rest of the afternoon. If anybody wants to talk about this stuff or wants to talk about energy trading or anything else that you wanna talk about, I'll be here, so thank you very much for coming to the talk. Test, whoa, hello, that's definitely a game on. No, that's not loud enough, is it? Is that better? We're getting, we're getting microphone? Yeah, oh yeah, there it is. Yeah, I hear it, sweet, right, we'll turn this off so I can have a private, am I getting reinforced by the headphones, microphone speakers? No, maybe a little, it's a little, I think. Sounds like my voice, thanks, man. Here, we'll test, we'll go out in the hallway where you definitely can't hear me. So is that working? I don't know, is it still working? What about now? What about when I'm out here? Can you hear me in there? Eric is dumb, could you hear that? I just wanna make sure it was working. Oh, is it so people can hear me? Oh, oh well, well hopefully not so many people are watching. Are you the AV person? Hello, okay. So what happens when it talks and I talk, science fiction is all around us? Let's find a bit where it actually talks. 1130, hello, you can hear me while I talk. So it sounds like, it sounds like 1130 in this. Can you hear me? Yeah, that's going on, right? Yeah, there we go. All right, welcome all to the next talk in this session. The talk is 10 design tips for microservices developers. Our speaker is Jim Tyrell. Jim is a senior principal solutions architect at Red Hat. Hey, thank you for that introduction. Good morning and afternoon everybody, anybody watching the live stream? My name is Jim Tyrell. I've been a Java developer for 25 plus years starting in 1995. A couple of years ago, I wrapped up a design thinking degree from what I would call the hardest design school around here around basically it's a master's only program. There's only two of those on the planet. It's not designed like art colors and fonts. I couldn't draw to save my life, but it really is all about strategy, innovation and leadership. And it's a lot of thinking about how we can make the world a better and easier place for us poor humans that have to live within it. So I'm kind of putting those things together in this talk and I've been a huge advocate and passionate about the intersection of humans and software for a very, very long time. A few brilliant books in this space. Primers, if you will, that I absolutely love Don Norman's book on the left hand side. If you've ever had a chance to listen to Don talk, it's absolutely outstanding. If you didn't know Don Norman, you probably use his term all the time, user experience or UX. He invented that when he was at Apple in the late 80s, early 90s. The next book, Clive Thompson writes for Wired Magazine. Coders, absolutely fantastic history of the last decade and a half or so of software development. A lot of insights in this hit me and the biggest one is that we asked developers to be designers in the sense of what I learned in my master's program. We don't really give them any training to think about that. And I've been thinking about what that means and blogging and writing a little bit about that as time goes forward. And then the seminal work by Alan Cooper. You don't know Alan Cooper probably by name but you do know of .net. And some of the work that he did at Microsoft, again in the late 90s, early 2000s to upend the way in which we create software. These things I really use these as inspiration to think about what it is we are doing as software developers and architects to make our systems easier and simpler to use. And then right up for this talk, I talked about the Death Star from Netflix and the other one that I can't think of right now. But you've seen a picture that looks a lot like this except I'm funding with you right now and that this is actually biases.  And the one that's you probably can't read it in here but the one that I find fascinating and the most interesting for us as software creators is something called the endowment effect.   That manifests in software like the switching costs of picking up new software is really painful for us humans something to think about. But it's also something that I use to think about how I'm creating software to make it a little bit easier to have users uptake it and use it and leverage it. You've probably seen these pictures Netflix and Twitter from 2015, right? They had these Death Star pictures a big graph of a circle of all these web endpoints all the different permutations of how all those things connected to and with each other. And the interesting thing to think about is just as you look at this chart just the amount of permutations of things that go around this, right? And if everything would talk to everything else you might get a 500 minus one on that. So 500 times 499 divided by two communication paths between all these different things almost untestable. If you go to the point of like any of these things could be called in any order with anything else you get a 500 factorial problem, right? The traveling salesman problem if you have a computer science background the mathematics of software development I would argue is almost, it's not almost it has eclipsed our ability of our wetwear and our brains to eclipse it. We can't keep up with the changes we can't keep up with all the permutations of the code we are writing. And this talk kind of pivots on that a little bit and asks us to think about how we upend 27 or so years of Java development to make our lives a little bit easier. So the Java being spec how many of you are Java developers? Like have been Java developers, few people. So Java, I started doing it in 1995 and 96 this Java being spec came out. It was this idea of a no argument constructor public getter and setter methods and then this idea of private member variables. For the last 20 plus years we as Java developers and even C++ and other language developers doing object oriented development have never really questioned this idea of public getter and setter and other methods along with private member variables has good data encapsulation that works for us really, really well. The challenge that I would put out there for you though is that by having those public methods and public classes the challenge of this communication path problem is multiplied with our Java objects and we'll get to that in just one second. So speaking about this, right? If you have 10 classes that might communicate with each other, right? It's 10 times 10 minus one divided by two for the number of communication pathways in this. So there are two, four, six, eight, 10 of us in the room. If we wanted to communicate with every one of us, you know, pointing in every direction there are 45 communication paths to each of us to each and every other person in this room, right? Do you go a little bit further with this? If you go to the idea that there are five methods or like we might communicate five different ways with each other, the mathematics of all the permutations of communication paths in this get a lot bigger, right? North of 1200 in this case. And then it gets even worse if these communication paths of us are actually to everybody else, to everybody else in any order, you get pretty quickly a 50 factorial problem and 50 factorial is 10 to the 64th. As I talk about these big numbers, the atoms in the universe from memory is like 10 to the 80th or 70th, I forget which. You are probably 10 to the 17th worth of atoms. In other words, 10 to the 64th is a really big number and a lot of permutations and I would argue probably untestable. It gets even better. And I've talked about this with a lot of people over the years that if you have all these permutations but only one path is correct, your odds are probably better to win the lottery than they are to have a successful completion with all these permutations. So a little bit of a statistics primer, if you will. And I would just ask this as we're running software to think about all these communication paths that exist, all the ways in which your software probably fails yet, there is maybe only one or two ways that it actually runs successfully. So what does that have to do with this? I would argue that we probably want to get rid of these public getters and setters and public methods in general. And that's kind of our first primer as we think about this and I'll interject two books at this point. Design, development, anybody heard of that book or this term? A few people are right. It's this idea that we could take our software domain problems, put them around a named thing like user management or an order or a store and we put all the methods around that. On the right hand side, the book switch, they talk about how do we do behavior change another great book, Atomic Habits recently published. But they all speak to this idea of changing the environment to basically make it such that you can't fail. In other words, putting that toothbrush by your sink might be a way to ensure that you brush your teeth at night. My wife still tries to put bags that are supposed to go back to my car by the front door. I just happily walk over those so she hasn't quite hacked me on that yet. But that's the idea, right? That we change the environment to do nothing but success for ourselves. So as Java developers, we've probably done this forever, right? We have controller domain repository and service packages sitting in a folder. These folders have probably hundreds of files in them. As a 16 year person traveling for Red Hat, I've watched people scroll for minutes looking for four files in four different folders. It is a really big headache. If you're really unlucky, they all have the same name across those four folders. Hopefully we haven't done that more than a decade or two, but I still occasionally seek code that looks like this. But you end up with in this case, maybe 400 files you might need to navigate and go across those four different things. And with model view controller architectures, that's what we did. So instead of coming up with those different packages, what if we just did the domain driven development, things like user registration settings, what other domains are? We create that and have that look like that. And then all of our Java objects, especially with Quarkus, we have something that speaks of JSON speaking front end. We need to marshal those JSON objects into this API. We care about writing things to a database and what does it look like? And in Java, they're gonna have annotations around them to inject that into the database. And then you need some code to probably execute this object to make things work. And those are almost the same problem as that user problem of earlier. So I don't really think those names are great, but what we can do is things like double up that user domain name in these structures to help you figure out where it is that these things are. When you're looking at them in the IDE, your life is a little bit easier to know where you are to navigate to what's going on here. And so this naming convention helps, but it still doesn't solve that public private problem, right? We still are stuck with those four objects times, all of your domains, if everything's public, could communicate to everything else in the math. Mathematics of failure gets to be almost insurmountable with that. So, and this is what we've done in Java forever, right? A private method, a public class. There are public getters and setters in this. So there's a getter and a setter for this first name object that's marked here as private, all public, but not necessarily the best way to go about that. So what I'm starting to, or what I'm asking you to think a little bit more about is that you get rid of these private member variables and make them unmodified, at least in Java. A lot of other languages have this. And if your head's being burnt up just a little bit, just work with me here for a few more slides. And instead of having this public class because that kind of gives you that same problem, we actually can get rid of that modifier on that. So we end up with a no modifier on the class, no modifier on our member variables such that these things are basically only available to the package or the folder where these objects are hanging out. So going a little bit further on with this is that instead of a lot of words and a lot of things that you're juggling, you now just have the definition, in this case of the JSON object and the values that you are looking to pass through that. And they are encapsulated, if you will, at the package level such that you're not gonna be able to bleed things outside of that package level to allow you to basically have success because your environment of what you have to live with is a lot smaller and a lot shorter than what you're dealing with. So rather than having private member variables, we have unmodified member variables as we see on the right-hand side of this. And the idea of this is that what's happening is your client, whether that's Postman or whatever, speaks to this endpoint that speaks JSON. This thing holds the JSON request. This holds the annotations around persisting that information to the database because inevitably these two things have different annotations with them. And the thing that I had as I was running Quarkus code is that I would, it was hard for me to keep my headspace as to what I was looking at. Was I dealing with the JSON annotations or was I dealing with the database annotations? And like 80% of the cases, they're all the same, but it's that 20% of things that shouldn't be written to the database, like API access keys or a password or just a one-time use token that's being passed around. You'd have to remember to mark it as being unused. And I just didn't like the way that you were messing with two different domains in one class file. And so often our examples put all four of those classes into one object, and I just don't think that's a good abstraction or good use of our time and our precious brainpower. And the thing that happens with this by using that unmodifier on the class and on the member variables, you basically are creating, except for this endpoint that speaks JSON, a firewall enforced by the JVM around your services that you're delivering. Giving yourself a lot less blast area that you as a developer have to worry about. You still have the problem of the desktop of those communication endpoints, but that's intentional, that those JSON speaking endpoints are now able to be orchestrated. And you've now created well-composable services that can be reused in your organization. And you've hopefully slowed down the amount of cross bleeding of that user object that can be used everywhere that drives microservices deployments and development a little bit crazy. So a few other things around these tips in the last few minutes of our talk, right? I like the idea of having a two string on these objects. It is so much easier often when you're writing software to just have a log output of what's going on with your objects just to see if things are set correctly or not. In an IDE, it's so super simple and easy to do that with a right click. You know, this is a VS code screenshot of that. As we look at six and seven, this API user endpoint, some things to think about when you're writing those things, it's thinking about a response code. Certainly the HTTP response code, you know, 400, 303, et cetera, would probably be important. But also giving people a textual string about what's worth being successful. Do you wanna think about how you're helping that user consume that endpoint? Do you give them a link to the documentation in that response so they know where to go, right? It isn't always possible that terminal access for these services are gonna be available for your end users and what could you do to help them with that? I'm of the opinion, never ever should you return a white screen of death on a 2006 struts code that just displays nothing to an end user. If somebody gives you something at an endpoint that you do not expect, you should give them an next best action, a message of what it looks like to give them information, bring information in that web service, or at least a link to documentation. To just give people a dead endpoint, I think is completely unfair as fellow humans in this world. Other things that I like to think about with that is maybe a health check, not just a health check of my upper running, but a health check of do the underlying services I'm looking at, are they all up and available, right? Is the database available? Is the message queue available? What kind of things are going on with that? I might also think about the last build date or a version number. So I know that I'm current. No one has to admit it, but I know we've all probably been there where you are running something, you're thinking the results aren't right, you're like, well, did we deploy it? Is it the right version? And often nobody really knows until you log into a terminal and look at it, and even then you can't tell. Why doesn't the endpoint just have a method well known by all of us to let you know what's going on with that? A few other, my other big one that I like to do, and the case here isn't the best, but this model user, this is a variable passed in it. If you're not familiar with the marshalling and corkis, basically this JSON object is gonna be hydrated for you automatically with that request. All the member variables are gonna be set, provided the case is matching. And then that gets passed in as a method signature on this. And then I like to return those same things. A lot of guides I've seen around this have a different object for in and a different object for out. My attitude kind of going back to the one just prior to this, where I like to ensure that people can see what is the expected input. I just like to give people the output and they can fill in whatever values they want and I either write them out or overwrite them or not. So in other words, maybe there's a user ID field that I'm gonna pass in. But if I just call that endpoint with anything, I get back that a user ID was blank and it might be zero and all the other fields are empty strings, right? And that way it gives your end user at least a fighting chance that maybe they need to create a JSON object that looks like that with the right values to get some information out of that endpoint. And I think that really can help you deal with this and it cuts down on one of the classes that a lot of people talk about with this. As far as testing goes, I think these two holder objects in the middle aren't anything that really need to be validated or tested. All your testing needs to be around the JSON request and then writing to the database and by an organic side effect, which I know from my favorite professor from college would say a side effect is not a good thing in computer science, but I think in this case, it's probably an okay way to go about this. I also don't think you should allow nulls in these. You should check for empty strings. What's the string null in a JSON object versus an empty string, right? That's really unfair for your clients. We really should have good things around initialization. And then the last one I'll talk about is telemetry. It's very easy with a micrometer project to put in some telemetry into your Java code. Instead of corkis, you get a pretty little web URL to look at to see how many things have been invoked, what's going on with your service and just in general, where and how things are going on, count-wise, elapsed time, et cetera. And there's a lot of support within this. So we did make almost 20 minutes, 25 minutes. So we have a few minutes for questions. If you wanna hang out, I'm here on LinkedIn and this code isn't quite, the examples aren't quite there at GitHub, but they will be here in another week or two. But with that, do you guys have any questions or thoughts about upending 25 years of Java orthodoxy with throwing away public getters and setters and public classes and using the package scope? Essentially, the package scope now becomes your object. That folder of four to five or six objects are now the object because we are doing things that are so complicated in Java these days, wanting to write to endpoints, writing to a database and writing things to and from JSON or databases with annotations that the throw all that into a single object is just too overwhelming for us mere mortals and wouldn't it be nice to keep us in a single domain as we're writing software? So anyway, any questions or thoughts? Nope, mind blown, maybe a little. All right, well with that, thank you for the second and last talk of the day, right? Two more people after me, all right. Yeah, thanks. We should turn off this microphone so no one has to listen to me. No, okay. Okay, now it's working, okay, thank you. So everyone, to the next talk in this session, the talk is Memory Barriers 101, the Linux Kernel Edition. Our speaker today is Lander Costa. Lander is a senior software engineer at Red Hat. He works in the Kernel security team, mainly handling CVEs for the Rails and to a stream kernel. Thank you. Hi, everyone. This is a beginner talk. Although it's from the Linux kernel point of view, many of the concepts are applied for other environments and programming languages. Our agenda, we'll begin with some definitions, then we'll move to compiler barriers, then processor barriers. And if time permits, we are gonna talk a bit about automics. Now, beginning with some definitions. When you're talking about memory order, we can say that from some point of view. The first one is search code order. That is the order that appears in your search program. Then you have program order, that is how the compiler translates that to the binary. And the example in the original search code, I have the assignment of A first and then the assignment of B. But when the compiler generated the binary code, it inverted those assignments. The third one is execution order. When the processor executes the code out of the original order. And the last one is observed order, in which a CPU A might execute the code in the original order. But a CPU B C in different order. Now I see that I made a mistake here in the slides. It's in the CPU A executes, it's supposed to be stored in A first. Sorry about that. Let's see the concept of sequence before now. Imagine that I have a code that starts with a segment of a variable A, then assign zero to B. I declare variable C. I increment A, now A has developed four. I increment B, now B has developed one. And I divide A by the increment of B. Clearly the final value of C is two. Then I have this other example, we're assigning one to A. And then I do this strange assignment of A over the increment of A. What the value of B will be? To understand that we have to understand the concept of sequence before. The expression A, sequence before the expression B, if the memory side effects of A are visible by B, sorry. Here in the first example, on line six, the evaluation of the two operands are sequence before the division operation. That means that the evaluation of A and the evaluation of the increment of B will happen before the division operation. But the evaluation of operands are not sequence before each other. That means that the compiler can generate the code to evaluate the operands and each other. It thinks better. So here it might evaluate the first operand first or the second operand first. And because of that, this is undefined behavior because depends on who is evaluated for us, the value of B will be different. Now let's move on to multi-core. Imagine that I have these two PSF codes. One is executing in CPU zero and the other one is executing CPU one. Here the assignment of data is sequence before the store flag. And the load of flag is sequence before the printf statement. Store and load are implemented in a way that they establish the synchronized wave relationship. That means that every memory assignment that's before the store are visible to other CPUs after the store. And for that to happen is the other CPUs. To other CPUs, see that side effect. It has the load that it's implemented in a way that everything that comes after it is visible to the CPU. And with that, we have the happens before relationship between the assignment of data and the printf statement. Although they are executing parallel, we can establish a temporal order between the assignment of data and the printf statement. For most of the code, for most of the example that you're gonna see, it will be some variation of this code. You have three variables, AB and flag. And flag indicates if there is valid values to read from A and B. And then in one CPU, I assign to A and B and then I set flag. And the other CPU, I wait for flag to be true. And then I use the load the values of A and B and use it whatever way I want. Okay, let's move on to compiler barriers. So I have this simple example here in which I assign to three variables and then print the values. And the compiler, it's free to reorder this code, whatever it wants. As long as it's preserved the correct result. If you see here, this code is invalid because the compiler moved the assignment of B after the printf. So the final result will not be the same as the original code. And how the compiler does that, it builds a data structure called the data dependence graph. Let's see here, for the assignment in line three, I create a node for A, then I create a node for B. And SC depends on A and B, then I create a new node for C and edges from A and from B to C. In the line eight, I create a node for this expression and also edges from C and B to D. And finally, I create an edge from D to the printchef statement. The compiler then inserts what's called a barrier between those statements prohibiting it from reordering these statements across the barriers. In this example, the only reorder that the compiler could do is between line three and four. But when the compiler is processing the code, more specifically the optimizer, it thinks that that code is executing only one CPU. There is nothing executing parallel. So if you see here in CPU zero from compiler point of view, it's perfect value, it moved the assignment to flag to before the assignment of A and B. But that would introduce a bug to lower code. The same way it could reorder the while to the bottom of the code in the CPU we want. And to prevent that, you have to manually insert a barrier by using this kernel function called barrier, which now you prevent these invalid reorders to happen. Very actually defined as a macro that is in line asmi. The volatile keyword it says to the compiler to not optimize this statement away. The assembly code is empty since a compiler time thing. This is the output operands is also empty as well as the input operands. And the last one is the clobber list. And this memory clobber says to the compiler that there are memory side effects in this assembly code that you are not aware. So you should not do fun business. It will cause the compiler to generate code that will flush any temporal value that is in a register before the barrier and reload all the values after the barrier. So are you done? Not quite yet. The optimizer may prove that nobody's using flag and then removes it from the final binary. The optimizer may prove that flag is always true here in the while and then replace it by an infinite loop. Or it can generate this code that it reads flag once, cache in a register and then ends up in an infinite loop again by just comparing the register with value zero which will be always be and then get stuck here. Here are three t-shows example. Notes here, I have a reader function that's called from man if flag zero. But before that I have this function I start writer thread that will run my writer function in another thread. If you pay attention, you only enter in the reader function if flag zero. So the compiler might say, okay, I know that this function won't be called one flag zero. So I know that this while loop will be an infinite loop because I already know that flag zero. And the trick here is volatile. Volatile is a keyword in this language that prevents the compiler do some optimizations that will cause the side effects that you saw before. Let's see an example here. Here I have a local variable called i. I look that we'll assign zero to i incremented until it reads then. It's pretty clear here that the final result, final value of i will be then. And that's what the compiler's concludes to. So it replaced this for loop just moving a then to the output registering in return. But if i change i to a volatile now the compiler is obligated to do what they call says it must do, it must do then assignments of i in a loop and it generates the code to do that. The Linux kernel provides two macros, regions and right ones. What they will do is take a variable, take its address, cast it to a file, volatile pointer, and the reference it. And in case of the right ones, it will assign to this reference memory. And all you need to do is replace our assignments and our loads with right zones and regions as it's here. Question, do you think that you needed the compiler barrier? The answer is no, because the compiler doesn't be our volatile access respective to each other. Let's see some examples of optimizations that the volatile keyword might prevent. The first one is load fusing when the compiler he reused a previous loaded value. Let's see here, I have three variables A full and bar and assign from Y to full and to bar. And if you see in the generated code, it reads to a register and then reuse this value to assign to full and bar. But if I use the regions to read from I, now I have two loads here in the final binary. Likewise, I have store fusing. Here I have two assignments of I, but as the first one, the compiler says that usually nobody is using the results of this assignment so it's optimized away from the final code. But if I use the right ones, now it will do the two assignments. Code reorder, you already saw that in the previous examples. Invented loads. Here I have a local variable PTR that reads from global PTR. The compiler might do that, eliminates this local variable segment and reads directly from the global pointer, potentially reading from the memory three times and this value may change between each of these reads and cause trouble. And to prevent that, I use right ones in PTR and then I force the compiler to do what I'm saying to do. They're also invented stores. Imagine that I have this original code. The compiler might do that. It might speak to really assigned to A and test the condition. And if the condition is false, it rolls back the value and then executes the else branch. And to prevent that, use the right ones here again. That code elimination. Here I have a global variable Y in which I initialize it with zero. And then inside the function, I assign the value zero again. The compiler say, okay, I don't need this. I already know that Y is zero. So it just put a return value here. But if I use right ones, now it will assign. If you wanna see more examples you see more details on that. I recommended the blog post who is afraid of the big bad compiler to miser from Paul McKinney, okay. Anything not clear so far? For processor barriers, it's quite simple. As the not only the compiler but also the processor is free to reorder the code. You have to use another function called SMP-MEB that you generate a processor barrier instruction. And just I need to replace my barrier bar SMP-MEB. And now I am good. By the way, I removed here the markers, read-ons and write-ons just for clarity. I also have variations for only for writes and for reads. SMP-WMEB is only concerned about stores. It will only prevent reorders of stores. And like a wise SMP-RMEB, we only prevent reorders between loads. The way that is implemented in a uniprocessor kernel, they just are defined to be compiler barriers. Because in uniprocessor machine, you don't need to worry about that. But on SMP machines, they are defined by their counterparts, by default by their counterparts that don't have the SMP underlying prefix. Why do we have these other barriers without the prefix? It's because they are needed even in uniprocessor. When you are using MMIO, if I need to write to a register that's mapped in memory and then read from another register in this order, I might need to put a barrier there to prevent the compiler or the processor to reorder these instructions. In SMP processors, you have specific functions for MMIO barriers. So this first definition here might not be exactly that. A little quiz now. Do you think that a full memory barrier might be equivalent to a load memory barrier followed by a write memory barrier or vice versa? What do you think? Any guesses? Let's see. Let's see this example. I have a store A, then a couple of barriers, and a load. But as SMP, RMEB are only concerned about loads then I can reorder the store here. Likewise, I can reorder WMB with the load of B. But now there are no barriers between them, and then I am free to reorder the load of B and the store of A. So the answer is no. The links kernel also has this function SMP store Mb that it does the equivalent of that, assign the value to the variable and then execute a full memory barrier. So I can replace this true statement but a call to this function. Now you're gonna see the release acquire memory consistency. SMP store release, it store to the variable and executes a barrier that will allow I reorder everything that's after it can be reordered before it. As in the example, I reordered the C without concern. However, it cannot reorder anything that's before it to after it. And the idea here is that I am releasing the data to the other CPU. So it's okay, I assign to all my variables and then I use release to say, hey, okay, you are free to read. So moving any things for the bottom to up cause no harm. And the counter part of it is SMP load acquire. Load acquire does differently. It can move anything that's before it to after it but it cannot move anything that's after it before it. And with that we can use store release and load acquire to establish my happens before relationship. Here I assign to A and B and then I set flag to say, okay, you can read now A and B and I guarantee that A and B cannot be reordered after the release. In the other part of the code in the other CPU I use load acquire to avoid anything that's after the acquire to move before it. I guarantee that if flag is true, I know that A and B has valid values. Questions so far? So let's move to Atomics. Say I have this assignment here to weigh. The code that the compiler regenerates is something like that in a load store architecture. I load the contents to a register and they store the register to my variable. If I increments the value of A, then the code will be a little bit different. I load the register, add one to it and then store the result. And this is called the read, modify, write operation. I load, I modify and then I store. Let's imagine that you have two CPUs here executing the increment of A at the same time. We would expect the final value of A to be two. But if you analyze the generated code, imagine that they are executing parallel. I load A to the register of each CPU. I do one to the register and store the final result to A. At that point, someone will store first because it get exclusive access to that memory address temporarily, store the value and then the other one will get the exclusive access and store the value. Anyway, final result of A will be one. That's because the increment of A is not atomic operation. So when to use it? If you are executing one atomic, no read, modify, write operation, which means loading stores, you can use the read, run, write, run, load, acquire, store, release, or you can use the atomic. But if you are just using load stores, atomic is just a waste of time. But for read, modify operations that you want atomically, then you have to use the atomic type. It comes in three variations. Atomic T is inch. Atomic long T is long inch. The atomic 64 is for 64-bit variables. They are all sign in it. But you can safely cast them to unsign it because the compiler uses this, the Linux kind of passage, the compiler just flag that will cause sign in values to be wrapped around. So, atomics for memory order point of view. Any load store operation in atomics have no memory ordering error. It's like I am assigned to a local variable. Read, modify, write operations that do not return a value have no ordering as well by default. And read, modify, or press the return, a value are fully ordering. And I have modifiers, relax it, acquire, and release to change the default behavior. Relaxed here means no memory bear at all. Here, our read, previously example, now with atomic version. I declare A, B, and flag as atomic and use this atomic in each macro to initialize then. In my right, I use atomic set and atomic odd. They do not return a value. So, they are not ordered at all. But when I'm starting to flag, I have to use the atomic set, release variant to guarantee a release memory order. Here, on the other side, I have to use atomic load acquire because load by default does not have any order. And then I use atomic load that has no order, but I don't need them. Here, I'm just using add on these two variables, same as before. But here, now I use invariants that return values. And as you saw in the previous slide, they impose full memory barriers, but that's a waste of time because the only barrier that interests me here is the release barrier. So, I have a better performance if I use the relaxed variant of these functions. The fetch variant, it returns the original value of the variable and the return variant will return the final result. This is something that you often don't have to bother, but the way that the code of the atomic system structure, it has these abstractions with maybe before atomic, maybe after atomic, that they are the memory barriers that should come before an atomic and the memory barrier that should be after the atomic. So, a full memory version of fetch edge is equivalent of calling the barrier before atomic, calling the relaxed version, and then calling the barrier after the atomic. And that's it. Any questions? One of the audience does not have questions.  There is no middle term. I am afraid to ask. So, thank you. Do you like it? Yeah. Much better than the other one, right? It was a bit longer the other way. Because also, I remember that first one, we had trouble with the connection a little bit. Yeah, yeah, yeah. But this was good. I thought that would be a souvenir. We welcome folks to the last talk of the session. This talk is Stratus, Integrate Device Mapper in Early Boot. Our speaker today is John Buppets, who is a senior software engineer at Red Hat. Thanks so much for coming today. Today we're going to be talking about Stratus root file system support. And what this really boils down to is kind of how you integrate a device mapper stack into Early Boot and this has broader applications for volume managers in general and just storage that requires more of a user space component. So, this is what we're going to go through today. First, we're going to have an introduction to Stratus. Stratus is kind of the case study that we're looking at in this case. And we're going to go over a little bit about what are the various components of Stratus that we're looking to integrate into Early Boot. We're going to go over a brief overview of the boot process just to kind of give everyone a base level familiarity with kind of the components that we're working with. There have been a lot of changes in the boot process over the past 10 years or so. So we're going to touch on these right here. Then we're going to talk about support for Stratus as the root file system. I've broken this down primarily into kind of the support work that went into it and also the usability constraints that we want to talk about when kind of designing easy to use software. And then we're going to briefly touch on current methods for setup in case you want to test it out. So let's start with the introduction to Stratus. So what is Stratus? Stratus is a managed storage solution. And at a base level, it's a device mapper stack that gives you the ability to put an XFS file system on top of it. It provides features like encryption, then provisioning automatic file system expansions, snapshots, and really what it's aiming to do is kind of provide a all-in-one experience that you see with some of the more modern file systems that have come out that are in the kernel. And so we're aiming to kind of build that with existing functionality in device mapper and provide a good user experience and helpful error messages for users. So the API allows the creation of kind of two main resources. We have pools and file systems. File systems are just going to be our XFS file systems. Pools are basically composed of block devices. And they're essentially a pool of storage that you can on an as needed basis allocate from for file systems. And together this gives you a little bit of increased flexibility where you don't have to worry in the same way about partitioning schemes like you would in traditional storage solutions. So why did we come up with Stratus? The basic idea is that we wanted to make storage easier. So there are definitely some very robust storage solutions out there that provide lots and lots of tunables. And what we really wanted to focus on was having kind of an opinionated approach to storage that gave the best setup for most users and focused on helpful error messages, user experience, and essentially had an easy path for things like encryption and growing file systems which it does automatically. So Stratus is really focused on kind of providing an all-in-one packaged kind of high-level solution that may not fit everybody's needs but is meant for the general case to make it as easy as possible. So Stratus' architecture is broken down into two components. We have the daemon and the CLI. The daemon has an API that users interact with and it handles most of the storage stack modification operations. So the daemon is doing all of the heavy lifting for storage and the CLI is really there to generate high-level error messages and issue debus requests and then report the success or failure of the operation. So let's go over a brief overview of the boot process from a show of hands who's familiar with kind of the basic idea of how the boot process works currently. So we've got a number of you but we're just going to kind of go over the basic idea of the boot process just for those of you that might not be as familiar. So we have Grubb, which is the bootloader and at a very basic level it is loaded by the firmware on PowerOn and essentially handles transitioning control to the kernel for continuing to boot. We have the InetRAM FS and this is essentially used to mount the root file system. So the goal of the InetRAM FS is to mount the root file system and in cases where you have a user space component the InetRAM FS is very important because not all of that code is housed in the kernel. So we need an additional image to kind of contain all of the user space component tools that might be needed to set up that file system. We on Fedora use Drake it for generating the InetRAM FS so that's kind of what we're going to be focusing on in this talk. And then we have SystemD so SystemD is PID 1, the Inet process you've probably worked with this before and it's used both InetRAM FS and once the root file system has been mounted it handles service management and is really meant to be heavily parallelized so it has a lot of dependency driven kind of concepts to parallelize as much as possible. There's also a notion of targets which you can think of as checkpoints during system initialization to basically say I want my service to start before any file systems are mounted. So we're going to go a little bit into now support for Stratus what we did at each of these layers to kind of make this as easy as possible. So for Grubb we didn't really have to make any support modifications for Grubb in particular but what we did have to focus on was usability. So Grubb's interface is the kernel command line that's primarily the way that we're interacting with the InetRAM FS so we kind of needed to come up with a way to identify which pool housed the root file system at boot time. So it really could be on any pool and the pool might not be set up yet so the root parameter for the InetRAM FS can't be used alone as that path might not exist yet. So the goal was to kind of come up with a usable kernel command line interface that basically made it as simple as possible for users to activate our functionality. And this is our end result here at the bottom. It identifies a single pool to be set up and other pools can be ignored for performance reasons and the identified pool is guaranteed to be set up so by the time the root parameter kicks in that path will be there and will continue boot successfully. How it works is the root parameter and Stratas pool UUID parameters are essentially associated so we're guaranteed to have all file systems on that pool set up. One of those will be the root file system which will pass to the root parameter and then system D and Drake it handle everything else internally for an easy user experience. For InetRAM FS support, we had one major limitation that was debus is not supported in the InetRAM FS and that's what we use for most of our IPC. So we basically had to think up something that might meet our requirements for an alternate IPC mechanism. So it had to be supported in the InetRAM FS and we had to have the ability to send file descriptors between processes. We use that specific functionality for our key API to essentially ensure that we're not sending plain text passphrases across a network API. So in the end we ended up landing on JSON RPC in UNIX sockets so JSON RPC already has a significant amount of tooling available in the REST ecosystem so we were able to lean pretty heavily on some other existing libraries. And UNIX sockets seem to be the best approach for the transport layer because it essentially allows you to send file descriptors and we only support communicating on the same system so we didn't need anything like internet sockets or anything like that. In terms of usability, we really wanted to make sure that our user experience was similar to that of crypt setup. So crypt setup generally has a pretty good user experience for boot time setup. You're prompted for a password, you type it in and everything just kind of comes up. So we wanted to mimic that as closely as possible. The components that we've kind of put in place to make that happen is a system degenerator which parses the kernel command line. This will generate a system de-service that handles setup and basically we use Plymouth to prompt the user in early boot at the console level or in a graphical prompt depending on how the system is configured. We also include all of the necessary tools in the init ram FS for easy recovery. So if the boot doesn't go as planned then we're able to recover from that using existing tools to rescue that boot. System de-support was probably the most challenging. We had two primary kind of overarching themes in our support. One was timing and one was device detection. So for timing, we had to make sure that StratasD would start at the right point in the boot process. So this really needs to be after Udev has started so that we're able to query the Udev database for devices that are available on the system but before file systems are mounted because StratasD might be responsible for setting up some of those file systems and if they're not present by that time then the Etsy FS tab handling can fail. So what we found is there's this sys init target and we chose that for our particular starting point for StratasD. There's a very helpful diagram for people that are doing this kind of work that system de-provides basically outlining all of the boot ordering of targets and that's a very helpful resource for determining where in the dependency graph of targets you fall in where you need to start your service. We also needed to make sure that IPC was reporting when it came up because originally we had a simple type service for system de and this essentially just reports success starting the service immediately after the process has started. This caused problems because our IPC hadn't always initialized by that point. So we switched to type notify where we could actually tell system de. Okay now we are ready for other services that depend on this service to start and consume our API and this solved our race condition problems in terms of consuming our API. We also had some race conditions between our daemon processing the devices and Udev processing the devices. Udev is going to do certain checks on a device but we do some additional checking on the super block to pull out additional information and it's not always guaranteed that the devices would be fully processed in our daemon by the time they showed up in Udev. So we added in some weight sleep retries internally to essentially ensure that by the time we started actually performing our setup operations they had been registered in our daemon. For device detection our primary challenge was SimLinks. The root parameter in the init ramfs expects a Udev device and previously our SimLinks in the format pool name file system name were manually managed. So they weren't actually recognized by Udev. This meant that when we went and booted and pointed to these easy to use SimLinks then the boot process would just time out because it would just sit there and it was not aware of the SimLink anywhere and so the boot would just time out. So we actually had to move over to setting our SimLinks in our Udev rules and what this allowed us to do is basically get detection by the root file system parameter but it also allowed us to send some additional information to our Udev rules to appropriately set them up and handle things like pool name changes and file system name changes. In terms of usability the main usability point that we focused on in system D was the case where something fails and so you'll see up here to system D directives and what this essentially does is drops you to the emergency console. So we chose this specifically because previously without these directives what could happen is lingering jobs could cause interference or unnecessary prompts during recovery and so this isolate job mode basically allows us to cancel all pending jobs that may cause interference during recovery and one of the major cons of this is that you're required to start off all of the services again once you drop into the emergency console and that might be a little bit confusing to users but the pros definitely seemed to outweigh the cons. So let's just talk briefly about current methods for setup. Currently Anaconda does not recognize and cannot create stratus pools and file systems in the graphical mode so we have a little bit of a work around for testing this on our team. What we essentially do is create the pools and file systems in the terminal mode of Anaconda and then we mount it, point to the mounted hierarchy with deer install in text mode and then the partitioning step is skipped by the installer. This has some gotcha points because after you've installed you do have to manually write at CFS tab you have to install the bootloader and write the config and do a native RAMFS generation. So this can be a little bit complicated and is not recommended for most users but this is an option. The option that I personally use for testing is R-Sync so what this boils down to is copying over an existing install with R-Sync and then the gotchas here are you have to reset the SE Linux context for the entire file system that you've copied over otherwise you can get login errors and then you have to modify all of the files that store information about the devices. So for example, you'll have to modify grub to point to the right boot partition. You have to modify at CFS tab to have the new devices that you're using and possibly Cryptab if the install that you're copying from was encrypted. So thank you so much for attending. Hopefully this gives you a little bit of a general overview of how you might go about approaching supporting a new volume manager in early boot and I'm happy to take any questions that anyone has. Hi, is full Anaconda integration something that you guys are trying to work with the Anaconda team on or is that something that is probably not gonna go forward from a, like where are you at with that, I guess? Oh, sorry, what was that? Just where are you at with Anaconda integration in general? So we're actually working on Anaconda integration upstream now. So our goal is to collaborate with the Anaconda team to eventually get that supported. So we're working in Fedora to basically make that a reality and we've already had some initial discussions about some of the dependencies of Anaconda. We've gotten support for Stratus in those dependencies. So hopefully it won't be too much more work to get Anaconda supported as well. Okay, one follow-on question if I can. Once that lands Fedora and Rell have long kind of defaulted to LVM partitioning. Do you see this replacing that or do you see it as something is augmenting it on the side? So we primarily see this as an option. So we want to give users the option of opting into Stratus. There are certain aspects of it that we basically want to make sure users are aware of, like thin provisioning. And so it probably wouldn't immediately be the default rather something that people can opt into and take advantage of if this seems to meet their use cases of ease of use and in opinionated approach to storage. Okay, thank you. So you mentioned supporting Stratus in the NetRunFS. That's just for like mounting storage that's already been created or are you talking about actually possibly creating like various high level storage pieces in there as well? I think we've had some people discuss actually creating a pool and file system in the NetRunFS. So we don't currently provide any functionality to support that. But for people that have worked with Drake it and kind of know how to roll their own in NetRunFS, we have talked with people, particularly I think in CoreOS who are interested in creating a pool and file system in the NetRunFS because of the nature of CoreOS. And so that is actually possible and doable with our current architecture. You would have to use the alternate IPC mechanism but it is doable. Cool, yeah, I work in CoreOS. I know those guys, Jonathan and a couple others in particular have worked really hard to get it to work and have overcome a lot of things. Yeah, it's not an easy thing to do. Yeah, thank you so much everyone.