something how do we make it more efficient, things like that so that's kind of how we came up with these different ways to consolidate these models so if you want to learn more about that I would encourage you to look at the AI for CI repo and you'll find all of these resources there like the notebooks the dashboards we've built the data sources and so on so I don't do that and these are again some more references to all of our repositories and with that I would like to stop here but thank you all for attending and if you have any questions we have a couple of minutes to take them yes yes yes yeah yeah that's a good question repeating the question again the question was how well would this work for a smaller github repository so if you have a smaller repo a less mature project I'm assuming where you have smaller unit tests how accurate this would be, is that your question yeah how much data do you need so I would say for any machine learning model the generic answer is more data is better so I don't want to default to that answer but I would definitely say that it might not be fairly accurate but at least over time if we keep trying and seeing that it's predicting at somewhat near accuracy then it would be a good start but of course if that particular check or that particular test is also found in other repos we can also use that as our training data set so it doesn't have to be coming just for your particular repo but if it's a test that is very specific to your repo then maybe the training data sets cannot be as large as expected but I would say it's definitely generic enough that we can try to find you know data sets from other repositories which might have similar checks and maybe use that to better train the model so that would be one way to approach it yeah yeah I think just to add to that I think the more we retrain and the more feedback we get from the repositories and the more number of tests that we run that's the feedback that we look for in any sort of machine learning model so that we know how well the model is predicting and we can probably just retrain it on the new data that is being made available by new PRs and tests being run so I think that would be useful does that answer your question yes yes so repeating the question it was so if you are taking tests from different repos and using that to train your model how do you kind of map and figure out which if the data is accurate right so yeah so some of these checks like for example in that screenshot you saw there was a check for file inting or there was a check for precomit check so those are more generic so those kind of are standardized to some point to some extent so even if you're collecting from different repos we can have some confidence that these data is going to look similar but yes some of the other checks are more customized maybe for a particular repository so it might not work well for the other so in those situations we might have to eliminate those kind of data just so that the model is not training on the wrong thing so it's kind of like an experiment of you know checking which data sets is it able to learn better versus which one it's not so that's something we might have to look at further yeah yes yeah that's definitely a good question I'm going to repeat it please correct me if I understand it wrong so you mean to ask that now that at this point we're making predictions for long running tests but what about the shorter running tests that is it even accurate that it finished so early right is that the question short running tests and false negatives that's a good question but as of now we are just looking at the long running tests but that is one of the use cases that we would also like to address but it's a part of proud logs because if you don't get any information from run duration the best place to go look for more info is proud right like where do you go look for your logs particularly for these tests do you have logs that are being generated for these tests like is there a place that you go to ok ok yeah but for most of the tests especially for open shift repositories they are linked back to proud and often times when they don't get an answer the engineers are working on these projects they go back to proud look at the logs and try to understand why particular thing was happening in certain ways and that's where like we also have like a very initial project that we did on proud log classifier which would help you understand why certain things are happening because it mainly just classifies all these logs into categories as to what category they belong to in terms of failures and passing and what could be the possible reason if they are behaving a certain way so that's another use case but this use case is mainly focused on the long running test go for it yeah thank you for that plug in oindrila also to add on to that at this point it's a work in progress but what we aim to do here with this initiative is that anybody who wants to use this tool can have a customize file where you can just put in what you want from this tool to do so starting from what is the threshold that you would like to specify it could be anywhere between 75 to 95 whichever thing you want to specify and also the next step here would be to have a tool that will automatically terminate all these tests but this is only something that we can do if the user or the repository owner lets us do so this is something that we would want to add to this tool but definitely something that the owner has to take a call for so you can specify all these parameters and once you specify it the GitHub action will automatically run on all the incoming pull requests in the future so that's the plan but most of the things are still work in progress but we are happy to just present whatever we had so far looks like we are out of time so thank you all again for joining us please feel free to reach out for questions even later during the conference thank you thank you so today we'll be talking about Ansible and more specifically how to automate some tips and tricks that we have discussed over the course of the last few years around how to think about automation in a way that I think it's a little bit more sensible or sane so I've been roughly 20 years in IT at this point then I have been using Ansible for 10 years and we'll be talking a little bit about the project that it was actually my first project so it's a 10 years old project but still I think that we have done some things right and I had a lot of learning from it and now I work in reddit as a specialised ocean architect for Ansible so the project that started my experience with Ansible has been a project that was a little bit messy let's put it this way so I arrived in this company as a consultant and basically they started to describe me the complex situation they had and they did not really felt that it was a problem but it felt that every time they tried to do something it was a little bit too much complex for some reason so what was the situation first of all we're talking about a Java environment where they had roughly 300 Glassfish installations Glassfish is an application server where basically you deploy your application on top the way they did their business was that basically they had one application server per customer so they had roughly 300 customers which means for this specific application so they had roughly 300 servers and they were not really the same version of Glassfish because they started with 4.0 and then some customer had some issues with 4.0 so they started to route one to some customers I mean obviously to the customer that actually complained about the issue and then they found other issues so they started to roll out 4.1.1 but even there only to the customer that actually have issues kind of the same application in the sense that the application was the same or at least that was what they told me but not the same version so you know and they had 250 of those machines of those applications running on EC2 roughly 10 per machine and then 50 on bare metal in a complete different situation on Centos 5, the others were Centos 6 so you know different version of stuff and 300 instances of MySQL because you know why not you have one application server per customer why not also having a MySQL per customer now interestingly they were installing MySQL from RPM and they never updated any machine so this man that basically they had probably 60 different version of MySQL running and to cover it all they had 5 scripts per instance to actually manage the instance research and those were kind of the same but you know kind of until you start to replace them and you discover that actually they were not exactly the same so as you can see it's not a very linear situation but I am sure that with some slight differences but still same concept you can find in your organization exactly the same situation I have been a consultant for many years I have worked in Reddit as well for many years as a consultant as a presale person and I have seen a lot of different cases where this is roughly the same situation everywhere so we started to search for a solution for this problem and we were looking for an automation system that was the only thing that was clear at the beginning and very easily we started to add requirement the first one that had to be simple because you know why add more complexity to something that is already kind of complex second it has to co-exist with whatever is already running because we are talking about a business that is in production all those customers are running these applications in production and they are running their whole company based on this application because this is a mission critical application for the customers so everything needs to be smoothly move to automation without any interruption or issues for the end customers it does need to keep the same security model because we are talking about PCI DSS applications applications that manage credit card numbers therefore if you change the security model you have to re-audit everything which might be a little bit more complex than we would like to and it has to be kind of self-documenting in the sense that at least we don't we have to have just one thing not two code and documentation so it has to be together code and documentation so that effectively we can be sure that over time we don't have a drift between the documentation and the code and this was a requirement that I added for the customer in the sense that it was not a requirement for them but it became clear to me that that was a very important point it had to be edempotent now what do we mean by edempotency I think that edempotency is a great concept very absurd sometimes but very useful so it's a property of certain operations that allows you to ensure that only the first time the operation actually changes something from the second time onward it will not be changed with some masters so some examples of edempotency x equal 100 no matter of the version of the value of x at the beginning after the first run it will be 100 and from that moment onward it will always be 100 so effectively it will not change another one is x equal x at the zero it can be any number then it becomes one and then stays one so going into the IT sector kind of thing equal a string to a file replacing the whole file so only one major sign it's edempotent because no matter on the content of the file after the first execution it would be stable and it would be just test now so what is non edempotent an example would be x equal x multiplied by two because the first time is let's say one and then become two and four eight and so on so effectively it's not edempotent and another example is very similar to the previous one but we have two signs of major which means that effectively we are just to the file which means that every time we execute the file will have one more line than the previous one now there are some major cases here to be aware of about edempotency and the first one is YAM update YAM update is not strictly edempotent because it relies on the repositories which is the one that you are fetching your RPM from and if your repository actually changes the content so if there is a new version available of your RPM then you will have a change so I would not exactly define this as edempotent but obviously edempotency is in math is everything is easy is either right or wrong in the real world it's a little bit more mixed for the same reason YAM install something can be edempotent or not based on the state of the repository now all these can be fixed in a way or mitigated if you manage your own repositories so you don't use fredes repositories but you use something like satellite for instance that would allow you to actually check and be the one that creates the changes in the repositories so even then it's not edempotent but still it's something that you can control and manage wget same thing if you wget you can have different issues with edempotency the first one is that the source changes which might not be ideal and the second is that a corruption wget should fix it or fail but still it can be a little bit tricky to speak about edempotency in a situation where you wget from an unknown source if you wget from your own server then it's easier and also if you are managing your own server and you put in the file name also the version then it becomes way easier to ensure edempotency another example of something that can be thought as edempotent but actually it's not we only have one major sign so effectively in theory should be edempotent but we are dumping a variable so that variable value can change I have seen servers with terabytes of data being completely wiped out due to this issue and not with Ansible with bash but the thing was the same because removing variable slash rm slash dash rf variable slash asterisk if variable is empty so be very aware that variables can be very useful but also very dangerous so Ansible was the solution that we found because otherwise it would not be installed but still why, first of all it was agentless agentless means that we didn't had to install anything on the single machines but just creating a new service that would handle some complexity second, it connects to the machines bssh, it can also connect with other protocols to machines but in our case it was only so effectively ssh was obviously the choice we went for and does not care about the state of the rest of the machine which means that if you tell Ansible that it needs to handle a specific file for instance it will only care about that file all the rest of the machine who cares this really helped us to do that slow migration to automation and a big advantage is that it applies changes in a sequential way which means that if you look at a playbook first thing that you see is the first thing that Ansible will try to do and then the second, the third and so on this makes the whole thing very simple and it can seem obvious that something executes in a linear way but actually in a lot of automation systems this is not the case because they try to optimize by parallelize operations and this also creates a way more complex situation and it has a very gentle learning curve as we see a little bit of YAML code YAML is fairly straightforward I would argue that Ansible is not configuration as code is configuration as data because I would not really define the YAML as code and Ansible playbook can be easily read by even non-technical people so like auditors we are in a PCI DSS environment so we have auditors every three to six months in asking questions and be able to tell to an auditor this is exactly what we are doing can have auditors but at least to some and it's very simple to set up and another big advantage that it has and it was a deciding factor in the end is that it's a Swiss life tool effectively you can use it to provision new systems, you can use it for deployment for configuration for whatever you want so the initial setup is simple but how exactly simple you first create SSH keys if you don't already have them you should already have them but you can maybe create one specific for Ansible that could be a good idea and then you start to distribute those SSH keys create a git repository where you will put your Ansible code into yes you can do it without git you really don't want to do that you create the first inventory where you list all your machines and you are done you are ready to actually write playbooks now which playbook should you write now because we have the whole environment ready now what so you need to select a process that you want to automate your first process to automate how to select it first of all it should not be a critical operation if you are trying to do something critical as first thing it will probably not go too well try to with something very easy it should be a very well understood process hopefully by you or by whoever is writing the automation because if you understand the process it's very easy to automate it if you don't understand the process it's going to be more complex also because while if you do the operation yourself it's interactive so even if you do a mistake you know the console will tell you that's wrong parameter wrong whatever with automation it's a little bit more indirect so if you don't know exactly the process it can become much more longer the effort to automate it and it should be easy to test so whatever thing your process you select it should be a process where you have a clear understanding of the process and how to prove that the process actually was successful or not successful it should be or it could be at least a boring thing that you should be doing and say you know I don't really want to do this but I can automate it which can be fun and you should have a little bit of time a few times the time to automate the process is longer than the time of executing the process should not be like millions of time longer but still if it's a process that you usually run in 5 minutes probably you will take it will take you 30, 60, 90 minutes to actually automate it so if you are in a rush it's not pro a good time so let's say the first process that we did was actually the provisioning of those application servers because we had new customers in the pipeline that had the requirement of having new server deployed so the first part is understand what is the process this was not the process the process when I ask which is the process that you are following it was basically a 30 step 30 steps long process we worked first on the process changed the process made it sensible and at that point we automated the sensible version of the process it's much easier much simpler, much quicker so you install java from rpm in that case create the glassfish user because for reasons they wanted to have a different user for that download pajara which comes in a zip file and that's why we had installed unzip unachive pajara set pajara file ownership to be owned by glassfish and create the system the unit to start it this is the process that we use for bare metal servers for sure servers was slightly different but still same idea so how do you automate it with Ansible you create a playbook such as this one and as you can see with very little changes in wording you can find the previous steps after the name so ensure we have java installed ensure that we have glassfish user ensure we have unzip installed ensure pajara installer is present and so on and as you can see then we have a few lines in between that are the actual Ansible code and the two are very similar I mean obviously it has a module it has some parameters and so on but still if you read one or if you read one or the other it kind of feels like the same and if you have both available it's way easier to understand them both and this is the second part of the same process as you can see we had 7 points here 4 steps here 3 steps here 7 again so we started from a process that had 7 steps we end up with 7 steps of automation this is a good thing if you start with the process that has 3 steps and then you end up with more than 3 steps in the automation your process was wrongly mapped you have to map your processes very well to ensure that your automation will follow exactly as you predict some consideration around this first of all if allowed redesign your process before you automate or while you are automating it because it's going to be easier second, simpler is better so if there are steps that are no longer required try to remove them if they are still required and another thing that we worked on was the automation of users so the left cycle of a user in every company is slightly different but roughly is like this you create a user on certain machines you add the user to certain groups you add SSH keys for this user to actually begin and then time passes and at a certain point you will delete the user on all machine that it has access to because that person maybe does not work there anymore or for other reasons so same here as the previous one we can very simply automate the creation of the user we simply have the built in user module we pass the name the shell, the groups already here as they present ensure SSH keys are available we copy the SSH keys to the target machines those are the public SSH keys not the private ones and then we can call it in this weird way which is basically passing a bunch of extra variables to explicit which is the user which are the groups which UID we want and which are the target host where this user should be present and we can remove users in a very similar way now some consideration around this it's very easy with Ansible to create distributed batch scripts which is what exactly we have shown now but that is not edempotent so it's very good to know that you can do it it's very good to do it but be aware that that is not edempotent that is not something you should aim to but it could be a good step in stone will improve the consistency of your environment even if using non edempotent way but you are not leveraging the whole tool and then automation is a step by step process so don't try to automate like this try to automate like in the second way which is basically try to create something easy that solves your first issue and then over time you will improve your scripts your code to make it even better but don't try to big bang the whole environment at the same time because it will not really work so we wrote the second version of the user automation the first one was not edempotent so it was okay at the beginning but not really in the long term so we decided to create a file yaml file that describes the user that we want to have in our environment so we have the user file and the user admin for instance and we could have many more of them and then we changed the first script that we have seen to use with items and iterate basically on that file this allows us to have a effectively edempotent way of creating users and ensuring that all users are always available in the right machines now this also does not cover the delete part of users that was something that we continue to handle in a non edempotent way because it's a little bit more complex and there are some security concerns around removing users so that was not changed over time but still this was a good example of something that started as a script a distributed script written in Ansible that then became something edempotent so wrapping up Ansible can provide you with simple way to automate distributed processes if that is your stage of automation start with some low hanging fruits and then build on top of that if possible rethink processes don't try to automate whatever process someone tells you that that is the way we are doing things here try to understand what exactly they are trying to achieve and then create a good process to achieve that result aim for complete life cycle automation but work over time don't rush it and in a way try to work for a complete company automation so effectively to automate all processes within a company or organization group whatever but same thing as before don't rush it it's an overtime process automation has a huge impact on people even more than processes so it's critical to explain to people what will change how it will change what will be the impact on them and repeat this kind of things many many times because a lot of times people are scared by changed and when we are talking about automation that is even more scary for people because the first thing they think about is great I'm now going to be replaced by this thing and be fired which is not usually the case obviously there are some cases where that is the case but 99.9% of times in my experience automation such as sensible does not mean people being fired simply means that people will change the role will change a little bit how they work will change their processes but over time everything is fine, no one is going to be 5 due to this and there is a big point that is critical which is be intelligent be specific on what you are automating if you have a process a long process and that process has a certain speed that speed is probably the speed of the slowest component or the slowest step of the process if you automate or increase the speed of anything before or after that step it will be just a waste you have, if you want to increase the speed of a process or the efficiency of a process you have to identify the weak link and then work on that part not the rest of the process otherwise it's just a waste so thank you and if there are any questions we have still some minutes so the question is how big should roles and play books and so on be to be useful so not too big, not too small well my answer is there is no right answer to this in the sense that it's absolutely okay and that is usually how it works that you start to create a role and over time it becomes bigger and bigger and at a certain point is like okay that's too big let's play that out but that is exactly the iterative process that should be going through because there is no right answer of you know oh it's 57 lines because it really depends I have seen some roles that made sense and they were written below 20 lines others that were more than 1000 lines still kind of made sense but still it really depends on your specific environment my rule of thumb is a role should be as big as possible but it should be able to run exactly the same in different environments without any change so that means that if for instance you are installing Linux and MySQL and you are doing it in the same role probably does not really work well because there will be some cases where you only want Linux or MySQL but if it's a very big role that is configuring maybe a nautical rock then it's fine as long as every time you install nautical you are installing a nautical rock so that is a little bit my thumb rule yes so the question is around which text editor to use around YAML this is absolutely a personal choice my personal choice is VI I think it's the best tool ever but I do understand the fact that 90% of people that use Ansible or YAML code does not use VI and is very well off without it so I personally find VI for instance very comfortable because I can grab stuff on the command line on in VI I can open multiple tabs multiple windows at the same time and I mean this is my workflow I think that over time you have to find your workflow what works for you and it could be VS code it could be VI, it could be something else it really depends on how you are used to behave and interact with it yes so I think the question is about what is the best way to handle a lot of play books that are automating a big environment and I think the real point is it depends on which stage you are in automation if we are talking about an early stage automation where usually you are running play books yourself having multiple play books can be okay I think that over time when you reach to a point where effectively you can nuke your whole environment and recreate it from scratch just with automation and you can run automation over and over because it is completely damp and so on I would expect you not to have many play books I would expect you to have maybe many play books because you want to do something specific in some cases but roughly your environment should be describing roles more than play books and then I would usually have all play book that simply calls every single role for every single environment and that would be also part of my CICD system where every time I commit something it then triggers that play book to be a rerun and therefore my environment to be aligned to my last change into the git repo but those are very different situations based on the level of maturity of the automation within that environment yes so there is no easy way because it's basically a problem of proving the impossible of something or the non-existence of something so it's a little bit tricky the way we have handled that in this case was basically that in yeah don't worry so the how we solved it was very simply in a very short amount of time we had one user per machine and it was the answerable user that was the only user that was allowed to connect to any production system we have decided at a certain point that was the good strategy before we had for a certain amount of time a number of user that were non-root or non-sudo users and it's an overtime process I think that ultimately the only way to really solve the users issue is not having users at all on machines yes and that is a big problem so if for instance is the absence of something within a file then I would use a template for the file and then replace the whole file with exactly the content I want with the users in theory you can do the same because users are justify entries in the the users file so effectively in theory you could do the same with that but handling users can be critically problematic in if something goes wrong or in other situation like this so the question is about get structures for ansible playbooks and roles if I recommend to stick with version or not my answer is yes I do recommend to start with the recommended layout and then start from there now those are not strict rules some of those are so for instance the folders within a role have to be named certain way and so forth but roughly the whole organizational files is not a strict rule and there is a reason for that you have to be able to change it if it does not fit you now if you want to move out of the standard read the standard very well try to understand why they did what they did and understand if that really does not apply to you or should apply to you and what is your situation but if for you it makes sense to have a different structure definitely have a different structure so thank you and with this we are out of time but I will be here and probably just outside the door thank you let's talk about Linux system roles thank you hello everyone welcome my name is Fernando I am a senior software engineer at Rehat I work for the Networking Services team mainly focus on network management and today we are going to talk about Linux system roles right so let's go to the important part what will you learn from this talk so basically when you walk out of the room you should have learned what is Linux system roles what is NMS state how we can mix them navigating the API of NMS state and the world role and then the new features of NMS state so ideally this should help you to configure or your networking using Ansible and using the role so let's go with the first part alright, what is Linux system role so Linux system role is a set of roles that aim to configure new Linux components for different subsystems it is a set of roles but it is also available as an Ansible collection so if you want to use Ansible collection feel free to do it we must say that it tried always to use native libraries subsystems sorry, native libraries from the interactive system instead of using CLI commands so all the time that we try to do it we use the libraries directly and it support a lot of those subsystems like Portman, Selinux Storage Network SSH FreeEPA many of them but today we are going to focus on network role ok, the network role so the network role supports two providers one is network manager and the other provider is Initscripts we are going to forget about Initscripts let's focus on network manager and it supports a lot of features for example a lot of interface types like Ethernet, B-Lambond Bridges a lot of them IPv4 and IPv6 address configuration DNS configuration routes routing policy DNS option configuration and some other utilities like ATH2 or authentication wireless yeah, plenty of them let's check first some examples of Linux system role configuration file the main thing that we are going to focus here is that when using the network manager provider we are going to focus on connections so as you know network manager has their own configuration files which are called connections so we define a variable which is network connections and in this variable we define the properties in the end what we are doing is translating a key file to a jammer file so we have the name of the connection the state the type, the interface name and some other configuration that we would like to use for example IP address or bond options like the mode or some other options like remoon or whatever and for example in this specific example we are defining a bonding interface and attaching an Ethernet interface to it so we can see like we have the controller property here and we need to define the ATH1 connection and then we are going to define the bonding connection I am showing this because my idea of this is show you then later the difference with the NMS state configuration in the other side we have Ethernet with a b-line in top of it so we have our Ethernet with the main connection port and IP Ethernet, auto connect the state app the interface name here we are using just a body upload from the top we could place whatever we would like without the ACP connection sorry without the ACP configuration and then we have Bilan configuration with Bilan ID 100 it's quite simple because I wanted to use simple examples to see better the difference so basically this is the network role when using the NM provider and when you see the network connections body app now oops let's talk a little bit about NMS state so what is NMS state? NMS state is a library with an accompanying command line tool that manage the host network in configuration and it do it in a declarative manner so it is written in REST and we provide a native REST library but we also have plenty of bindings to see Golan, Python and others one important thing and the difference is that when defining NMS state configuration files we focus on the device we are not going to focus on the network manager details we are not going to focus on how the specific properties are going to be written we only want to focus on what is the what the what the user want to be configured so the user can define for example I'm the user and I want a bond interface with type bond to state app we define the different bond properties like the mode the options like MIMON and the port and this has the same effect than this so this is the main difference here we are talking about connection name interface name here that doesn't exist we only have the interface name it's what is interesting to us and also NMS state here we try to help the user so if the user want ATH1 and ATH2 configure and the interfaces are present on the system NMS state will automatically create a Ethernet connection for them and it's going to manage them and it's going to attach them automatically to the bond so you don't need to care about how Ethernet is going to be configured obviously if you want to configure some specific thing on the Ethernet interfaces you will need to define it but if not you don't need to care here you need to define it because you need to define that is the controller the controller is a bond so more difference for example if we create a bilan the difference is that we create a bilan ATH1.100 with the type bilan STATE UP, IPv4, Enable True and the others so what will happen is for example this bilan already exist and have something configure or this bond for example in NMS state it will keep existing properties and we only replace the one that the user specified so for example if the user specified minimum 100 NMS state is going to set minimum 100 but the other properties are going to be consider and they are not going to be replaced here it is different and in this case it will replace the other properties because in the end it is rewriting the whole configuration so this is done because how NMS state works is that it fetch first the existing state so it fetch the existing configuration translated to a general file similar to what we use mesh them and apply them so it allow us to do what we call partial editing so that's quite great but currently NMS state by itself it can only be used on your host and we wanted to have a tool that can use NMS state across multiple hosts at the same time and then we thought the net will work alright so let's mix them we introduce the network state variable so when using the nm provider we I mentioned before the network connections variable and now we are going to have something different the network state variable how it's going to work in the network state the user will define the complete network state that wants to be applied and it's going to do it using the NMS state schema so in the end if the user is already familiar with the NMS state API they don't need to lend something new, they can use the same schema and apply it as I mentioned before thanks to this using NMS state partial editing is possible and now it won't replace the existing connections on the system not mentioned here but also very important NMS state has an extra feature and it does verification checkpoints and dropbox so when you define an state we call it desired state first NMS state is going to check the current state and say ok this is what we have configured using the network manager feature of checkpoints is going to save a checkpoint then is going to try to apply the settings if something goes wrong it's going to revert it but if it went well according to Kerna but then NMS state fetch again the existing configuration and compare it with the desired one and it doesn't match means that something was not configured properly so it's going to revert back obviously this can be configured to be disabled but it's a really really good feature because if you misconfigure something and you revert your connectivity it's going to revert back to the state before and therefore the matching is going to have connectivity again hopefully and also it's important that it simplifies a lot the API for the user because we will have we are going to focus on the interface instead of the NMS connection in the other hand it is true that if you are willing to configure various specific NMS details this is not going to be able to be done by using NMS state because it's not the scope that we want to call alright, so let's see some examples of combining them as you can see this is basically an ATH this FNN interface configuration and here we are going to define NMS state and we define the whole state that we want to configure in this case it's a FNN configure with IPv4 and IPv6 and yeah and that's all so it's in my opinion it's quite clear and just by looking at it even if you are not familiar with the API and it's the first time that you look at it you should be able to recognize what is this doing then we can go to a a little bit more complex state and in this case we are configuring routes and an ATH1 interface so the same thing here we don't define the routes in a specific network manager profile we just say I want this route to configure and NMS state will try to find out how to do it so the user never need to care about how it is being done basically need to care about what is going to be a configure alright, so this is a sample of that route then we have another example which is for example next wednesday and we have the DNS we can configure the search and the server names so again this none need to be configured per profile it's going to be configured in the whole networking and this is quite good because in the past for example if you configure using this using this schema if you configure DNS in one interface and for some reason you need to drop this interface the DNS is going to be dropped as well and this way you don't need to do it when you drop an interface as I say partial editing is one of the main features of NMS state NMS state will recognize that in the past the DNS was configured so if the DNS configuration is stored in one interface it's going to move it to another one and it's going to move it to a different place or a global conflict in case there is no a suitable interface so it is quite great because you don't need to keep moving the DNS or routes or routing policy configuration from one connection to another you just need to configure it once and forget about it until you want to drop it and alright so how to navigate the API the NMS state API is declarative and we try to make it as much intuitive as possible so it is documented at nmsstate.io and it use inclusive language I recommend you to take a look to our web page we have documented the Jamel API we also support JSON and we also support some libraries like Python Rust etc and it's also documented so you can check the libraries documentation alright and one big benefit that we get on the network role when supporting NMS state is that we don't need to implement things twice imagine that we have a situation where network manager have a new feature and NMS state if you want to use it for network role it's immediately supported if it's supported on NMS state because it uses directly so for the developers it's quite cool because we don't need to implement things again and again which is quite common when you have a diamond and a set of tools that interact with that diamond also it's beneficial for the user because when we fix NMS state it's immediately fixed in network role just say that I recommend you to try it out and I would like to hear from your opinion we have mainly list the project is on github NMS state and network role there should be some more okay so I would like to hear from you some feedback, questions any comment at all yes you can use Ansible I mean you can use this as an Ansible role so in case you are configuring or managing your infrastructure using Ansible this could be part of your tooling and apply this directly with Ansible and it's going to be applied to all the hosts that you are managing with Ansible so could you speak up, sorry yes yes, but as far as I know for example if you use yes, NMS state is basically a client of network manager so alright, okay the question for the stream the question was if you need to install NMS state as you install NMS CLI in your host to make it work so yes but as far as I know the network role is able to install the necessary tools when needed so when you try to use it it's going to install NMS state obviously this is not supported on not support the distributions so I could check the list but I think that supported distributions are REL, CentOS, Fedora and I think some more but not completely sure it's fine thank you more questions something no, alright so I hope you enjoy it very much who was there this morning we have the last session for today for Convention on Computing for the host and Christophe hello so my name is Christophe de Dinshin try to say that three times fast I hope you learn since this morning and so I'm working for Red Hat nobody knows what I do there because I don't know myself vaguely working on computational computing to try to show how to go from confidential computing host all the way to the workload so what we are going to cover today first a quick overview of confidential computing that's a repeat for folks who wear faithful and weather this morning then we are going to show how to access a guest memory to steal secrets we are going to set up a confidential host which is extremely simple as you'll see then we'll try to set up a confidential VM and hopefully everything happens as we expect then we'll discuss some limits of memory production we'll talk very quickly about confidential VMs for instance on Azure and other places but Vitaly explained everything about that this morning so there's a better talk and things like confidential clusters as well as examples of SLS concelación and continuation containers I'll probably have to skip for time reasons and there is a workshop on Sunday where we'll show you how to do that in practice so you're welcome to join the workshop for containers so the problem statement is how to protect data in use that is data in memory and we use so confidentiality is really the essence of trust and it's broken right now in the cloud because your infrastructure essentially sees your data so the software that you have runs on the cloud also known as someone else's computer and the hardware resources there are owned by that host you don't really buy for this memory or whatever you just front it you run various workloads on it so that's typically some things like containers it can be virtual machines can be other things and we have many mechanisms in Linux collected memory fruits all these kind of things are designed to protect the host from the workloads the other way around nobody really cared about who has root access on the host can fully look at the files for your containers look inside the memory and so on and so that means that if you have competitors on the same hardware they tend to not be very at ease to put really confidential stuff there because nothing tells them that their competitor did not pay the CIS admin more than they did and so in the end maybe their data is being shared so we have solutions for data on disk or data in transits networking etc so that the host essentially cannot really know what you have when you use an encrypted volume for instance normally you're safe for memory it's a different problem and there's tons of secrets that run there and these secrets well they may be transient that may be moving fast but you can still extract pretty much anything from it if you know how to look and so what continental computing first added was a way to encrypt memory and the encryption is not necessarily super strong which is why I'm not using a super strong encryption there either but it's very fast it happens on every write to memory and so it really offers some serious projection anyway and then can I take that question at the end that's the next question the short answer to that is that the encryption the encryption is really normally not visible to the host at all and dependent on the page dependent on the location and other aspects that make it in practice hard to break what could work well would be replay attacks on a given page for instance that would work better because that encryption mechanism would be the same for that page but we can take that later that's a very good question and when I say weak it's not super weak either it's still relatively serious so there is also integrity production for the CPU state essentially to make sure that the hypervisor cannot make your workload do whatever they want by changing the address of execution or some registers in the CPU state and one thing that I covered extensively this morning was proving that what you run is what you want through mechanisms known as attestation and that give you some cryptographic counties about what you run ok so let's go to some real me so this morning was sort of a joke presentation today we are going to look at a real system and I would have like to do that live but I explain in a moment why I did not do it live so I'm going to comment what I recorded so we have a FETO R38 system it has a relatively recent Linux 6 to 14 and it's it's running with 16 CPUs so that's an epic 16 core so that's a relatively serious machine with a CV and a CVSNP available on it that's what I'm going to use for this this demo today so in terms of so in terms of what the machine looks like when you remote connect so that really depends on the machine but you have typically web access like this and you can look for instance at bios settings when it starts that's the good old fashioned way of doing it so I'm going to skip a little forward in the setup you can access the bios the old ways with hitting the F12 or F2 or whatever it depends on the machine what matters there is that we are going to look at the processor settings and check that we have secure nested paging enabled so you can see that at the bottom so that's essentially the thing that re-matters in your configuration there and of course the way you do that depends on the machine you use so the rest is not specifically interesting you just have to be relatively up to date with firmware because these technologies are relatively new and so older firmware may have values issues ok so nothing really really special otherwise we can boot the machine and look at the virtual console the virtual console and you get just your regular text only mode Linux this also support graphical mode so the more modern we are doing that is to go through the web interface like this so you have as always the place depends on where you are but you have system settings and you can see again that you have your secure nested paging somewhere in the middle there I scrolled too fast so it's a bit above so you see it's a secure memory encryption secure nested paging at the bottom of the page ok so I think that's about for this page so we can check that we have some host level SCV support by looking at the message log and that's essentially all we can do without additional software ok so in order to be able to do some kind of demo I'll have to use a little bit of extra extra software and I'm trying to give a sort of shout out to KCLI which is a relatively convenient tool to do various things so you enable a special copper for that and then you install what you need and I forgot one of the packages there which is Lidlot so nothing really complicated but you need essentially a hypervisor, a manager for that so you're not very surprised so far I guess so now we can check with self color we can check a little more about what our machine has inside so it has support for SCV SCV ES and SCV SMP and now you all know what this means we also need to record these numbers that we have there, the number of physical bits reduction, the cbit location and stuff like that we'll need that later so let's just remember for now 50.15 we can also check that we have a kernel that has enabled the AMD support and we can check that our Libvert also has support for this kind of features so the DOM capabilities will tell us if there is an SCV section in there then your Libvert also has support for that so this is essentially a pre-flight check to be sure that your machine has what you need and again we have this reduced face bits and the 51 and of course you can now notice that the number of reduced face bits is not the same between the two so starting to scratch your head and say ok something in there does not cook well which one will I use we'll see ok doesn't so KCLI I assume most of you don't really know how this what KCLI is it's a sort of a quick user interface to create VMs, create clusters and it has voice backends it can use Libvert so here I'm going to use Libvert and so I create a default pool for that I essentially reset that machine initially before doing all that except I have I forgot to reset the network so let me just delete the network, reset, create so that you can see how you get started setting up a special network for your machines and let's click a little and now I have no VM and these are the images I played with one interesting thing with KCLI I can list all the available images and you see there is a number and that's for me the best tool for instance to install things like Arcos or Rel compared to the others saves you a few steps in terms of being able to deploy this kind of operating system so this is essentially the basics for what you can do with it so we are going to in the first run create a regular VM and steal data from it so you use the KCLI create VM the dash I is I want the Fedora 38 image I want 4 CPUs and I want 4 gigs of memory and I'm going to pass a root password which is one of the secrets I'm going to look for so please remember that secret we'll use it later so now I connect to my console and I check that the machine boots so far it looks good, I'm happy with it right and you see that there is some cloud in it magic happening behind my back I log in, I tap Strumpf which is the French for Strumpf and KCLI already sets up for me SSH and stuff like that so that means instead of going through the serial console I can use a more serious tool and use SSH because now you we really care about secrecy and stuff like that so I'm going to SSH into my test VM and you see that I'm user fedora so KCLI has this idea that depending on the image it can actually create a user for you and do not necessarily log in as root but you can also log as root and we are going to do that to check that there is no ACV inside so no ACV support in this VM the way I created it just a regular thing and let's try to now inject a typical C program in there so that's high quality C code QE told me you'd better run that in a VM please they told me there is a bug or two in there I don't know looks good to me when I compile there's barely a little one or two let me copy that to my VM so I have the ACP command as well that's fine and I connect to it and then I'm going to run and of course it's going to run perfectly well like any C program so it's not exactly what I expected that's not the message I put in there but close enough ship it ok so now I have this this program and I'm going to do something slightly more complicated so this one I have to look up the documentation every time I want to use it and I'm going to done the memory of the guest so that's why there's a pause there because I'm thinking ok, where is my documentation where do I copy paste this stuff so and also because I discovered at this time that reinstalling everything caused me to lose my completions that's also because so I'm going to use the QMU monitor command and execute a dumb guest memory and the arguments are it's very user friendly it has all the quotes and backslash and curly braces you want and I'm going to use a protocol which is file colon something to dump to a file which why not just file colon but it's essentially I'm dumping the memory of my guest into this place you can also do something like decore if you do that you're going to have some other stuff than the guest memory so that's why it's better to do it that way and you see this disconnected from QMU system doesn't look really good en realidad that's probably why my machine ended up being a bit screwed but we'll see later so now I can drip for my secret that was in my C program and you see I have a match so let me check with Emacs what I see inside my guest memory Emacs even completely designed to open 4 gig files so it takes a few seconds to open it does want me, you know 4 gig is big do you really want to do that when it displays the stuff the way it should be displayed tons of zeros control ads and now I look for secret and I find my hey really secret stuff right in there so my C program does not emit that message but at least I see it in the memory so I know I did load the right program now what if I search for my strumpf password well of course I find this as well and I I find it in this nice little snippet of script so hey that's perfectly unsafe I'm happy because that's sort of what I wanted to demonstrate right so success for some definition of success of course so let's try to fix that and you'll see that it's really really super easy don't worry so let's first try to make I'm compatible with what you need for for continental computing so unfortunately KCLI still creates all style VMs so I have to change the hardware to say I want a more recent Q35 stuff so let me do that I replace PC PC 440 effects with QMU and then it complies because over now I need ok so let me fix that too so I need to fix my devices the good thing about Libert is that it checks tons of stuff for you and then it complies about the addresses being wrong so I just remove everything it's going to recompute a set of addresses that match usually works and and the next thing I need to do is replace my SATA disk and normally I should be more or less the good shape after that so what I'm doing here is only trying to make my machine compatible with the kind of hardware I need for virtual machine so ok now I forgot something and it's I mentioned the SCSI stuff and I did not do it so let me fix that and that's because the devices you are going to pass in your VM need to be producted somewhere so you need to have IOMMU active everywhere and to have IOMMU active you essentially need to convert to virtaios crazy and stuff like that so that's what I'm doing here converting to virtaios crazy and everything should be pitchy ok so let's start my VM and check that it's still alive come on don't do typos my AI is typing that's why it's sometimes making typos ok so the VM looks good and I can still connect and of course I still don't have any kind of ACV I did not activate that yet but at least the VM has the right kind of hardware so I need to do tons of VM edits like this and it's well documented in Libert on the Libert.org page so you see that for instance that's why I took 35 idea and things like that but it also tells me I need to change the bootloader and I need to do some adjustments to the memory so we are going to do that let me scheme a bit quickly that's how you need to boot with OVMF so UEFI and you need to do some adjustments to the memory you need to do some adjustments to the network so it's all relatively well documented let's keep relatively quickly to this because it's not necessarily the most interesting part of the talk but it tells you what I'm spending my days on right who doesn't love that ok so I follow these instructions and I keep turning and turning and I try to start it again and that's why I got my first Libert Hang so forgot that restart restart some demons and I can start my VM again and now it boots in UEFI so that step looks good I'm still not completely there of course I still have not enabled ACV so you're starting to think maybe this is taking a bit longer maybe this is not the best way so before I can actually activate ACV I need to activate it on the host and endorse it so to do that first let me give you a bit of terminology because AMG is very light on acronyms so ARK stands for AMD root key ASK stands not for ask but for AMD ACV key CE key is a chip endorsement key OCA is the owner's certificate authority PEK is the platform endorsement key PDS is the platform Diffie Helman you need to know all of this and there will be a question at the end I ask question at the end oh and there are two more like the tick and the tech and you can read so let me do my host setup let me first reset everything and let's verify the host and you see that I have this nice chain so this morning I taught you about the root of trust and maintaining the chain of trust you got a good example there with this sign this and and certify the etc and if I run it twice I get the same results for PDH, PK and OCA which you remember what it means if I do a reset though I'm going to get different numbers this time this has changed and transparently SCVCTL is actually talking to somewhere at AMD which I learned the hard way when their server was no longer responding to the requests so notice also that the AMD root key fortunately did not change when I did a reprovisioning of the system so that's what you do when you reprovision the system ok so that looks good now I'm going to do the guest setup so the guest setup is primarily adding a long security step to certify the virtual machine as confidential and you'll see it's really easy so it's all well explained on the document here that's the xml you need to put so I need to find a place to put it in my code and that's that's why I'm going to put that stuff and that's over this is where my cbposs and stuff matter I need to put them there and I need to select whether I want to put one or five frontly both work so there was some discussions on what exactly was supposed to be there so I've got my long security section but it's not sufficient because I need to have actual keys in there so in order to do that I need to create a security session so first I will export my pdh which is the ok everyone platform if you have mania and then I'm going to just check that it works with this setup and then I'm going to generate the files that describe a given session for a VM so now that's related to that specific VM you will use the hostname there because you can transport this data to some other place and I'm going to keep it simple just use host.pdh and do everything local normally you can do that remotely and the policy is helpfully explained in chapter 3 of some offshore document somewhere on AMG website it's also on a different page on the libvert documentation and you essentially have to orbit mass for stuff that's modern style, you know, 1980s style of user interface so now I've created my session and what this did is create a few files that all begin with that VM so there is this god mode thing there is a pack a tick and the pack in the session and all the stuff so that looks good I can probably make some progress from there and put that stuff inside my file so I'm going to insert where there is dh side so dh should remind you of course that it's pdh is the same thing so you insert the copy of the file there and then you insert the copy of the session file in the session section I think that makes sense I mean that's simple that's a good user interface use max, use vi, whatever I mean just text there is some remarkably low entropy on this sequence of A's that I find really surprisingly low I don't know why exactly that's something I like to look into why there are so many A's ok, so now that's edited and I'm going to do it the scv old style way I'm not going to try remote attestation because that's complicated so we'll keep the simple stuff here and so what we do essentially now is that we are going to enter a brave new secure world so to launch a secure VM the first step is to start the VM but you start it in post mode because reasons the reason is you have to check before you actually launch it that's called pre attestation remember I explained that this morning so you have to check that it passes the test before you let it go then you do this dumb launch second full test VM that gives you a measurement of what you are trying to launch it's a helpful tool where you just have to copy paste the output of the previous command but of course with different option names like everything that begins with scv does not begin with scv on the command line other than that it's it's really simple you do that and you try to not type too much oh you have to measure the firmware so you have to tell where the firmware is and with that stuff it looks fairly nuxy to me so far right now I'm going to to get my grade 4 and I get this next answer looks good to me so now I'm allowed to resume my VM I can say hey resume, console and let's go and what happens so putting again into EFI so what should happen there I should see a cv and I should be able to run my application without it's content being visible that's what I expect so I do that I connect, I type tronf which is no longer so much of a secret and I have a cv success I'm really happy memory encryption features active let me ssh into it from another window and run my little test program so let me copy the latest version where I upgraded nothing I don't know why I need to copy it I think because in the meantime I recreated a VM from scratch so I run it it works just as well as before and now I'm going to do my memory dump like before but now I have it in the history so it's easier and I take the encrypted version what do you expect same as I said but let me now that is a pretty let's look with the max something is wrong there I have to wait to see why it's bogus so and the reason for that is that we see the word secret as part of some fight system stuff which is already I'm already scratching my head but I don't actually have my secret stuff with 3F it's not there so something is working I have predicted somewhere something but if I look for stronf ok who can tell me what I did wrong I scratched my head I thought maybe it's something on the console that is going wrong so I'd say if you can't answer if nobody in this room can answer it means I'm not the only one being stupid so you're all as stupid as I am because I had to scratch my head for a while to figure out where did I go wrong what did I do incorrectly no it's much simpler than that so I double check scb is there no we have a problem if you have not seen the what's talk you search google you search google talk what's talk right now ok so it's time to tell the boss dear boss my demo for next week doesn't work which is why I completely changed compared to what I was supposed to talk about dear real wasn't I do cps of my secrets in there and I'm not sure why so that was my secret the first experiment with dictatototo scb is active answer for my boss isn't that the punchline I just suppose a product from that ok so the reason for that is that only memory is encrypted what I did wrong if you if I grep the stuff from my program it's not there that stuff is not visible that part we succeeded with however when you do encrypted memory like this you cannot send that to this directly right you cannot write your network directly so all the IO buffers that are being used by the kernel have to have the famous cbit clear saying it's not encrypted which means that you'd better encrypt your disk right and that's the thing I forgot about KCLI is that by default it uses you know it uses good old card in it and there is no encryption of the disk setup by default so we need tools and we need to build tools to keep that complexity and reduce the risk of misconfiguration so first of all how did I get there you can blame Wayner over there I'm pointing fingers there because I'm also doing a workshop in two days with Wayner and his workshop was about I know it's Q&A but it's the last session of the day so I have a little extra time so I have too many talks going and he did a remarkable series of script for this lab and he was using KCLI all over the place and I said that makes it really shorter compared to what I was planning to present so let me use KCLI and I switched to KCLI and did not realize it wasn't encrypted until the very last step so step one, prepare a list of commands for a live demo step two, prepare another workshop with Wayner step three, notice that this workshop uses KCLI step four, think ha ha great idea, I use that too step five, record a movie with all the plan steps but where you replace carefully all the steps where you were creating encrypted VMs with a fun installation with KCLI totally forget that KCLI does not encrypt disks success so that's how you fail spectacularly in public now there are better ways and I'm going to show them that should be for another talk you can set up VMs in the cloud that was at 10.30 10.30 this morning with with Italy where you can use confidential containers that's the talk we have tomorrow and you can deploy complete clusters which I'm going to show on the next slide with a tool like constellation for instance which is carefully to set up the VMs the right way for you it's just that it would not fit in the 30 minutes that are still alerted to me after the end of the talk and so so essentially you wait and you wait and you wait and after something like 40-50 minutes you get a cluster which is really set up the right way so this is the recommended way to go if you want to really spend your money if you have too much cash on hand and you are ready to pay for all your VMs and all your control plane and everything to be encrypted go with it that's very easy it behaves like your regular your regular cluster getting tired a little bit so then you can connect to a node and check that it's encrypted properly that the disk is encrypted etc and of course we are working on this for Fedora and CoOS so what you see here is the creation of a Fedora CoOS a disk with ignition with a secret injected and that gets encrypted on the fly as the ignition step happens and I have to thank Timothy Ravier for helping me doing that this morning this is relatively new to the talk now it's time for questions if there is any there is no time for questions but so the question is the question is about what is the cost of memory encryption mostly and so in terms of power usage is hard to tell because it's not exactly the same generation of processors so if you do processors with more transistors but they are smaller then maybe it's not consuming that much yeah I don't think I don't think that using encryption adds much in terms of consumption of memory sorry you have energy and it does not add latency in terms of measurements and in terms of latency compared to non encrypted it's the same speed however on the same CPU however what it adds cost is that you cannot do any kind of deduplication you cannot deduplicate your disk because they are encrypted you cannot deduplicate your container images because now you want to download them from inside your virtual machines so you are downloading the same thing over and over again and you cannot do any kind of KSM or anything with encrypted memory anyway don't ask me so your question was when do we get cheap hardware so so to be precise I think it does consume a little bit of power but when you see that you have chips that consume 100W or 130W of TDP I'm sure there is half a watt in there that is encryption but I don't think it's more than that the reason I think it's cheap the reason I think it's cheap is that my phone does it on a battery for a whole day so it keeps encrypting stuff all the time when it's communicating and they accelerated in hardware or the encryption with the network and so on precisely because the hardware could do it relatively cheaply but to be fair you've got the wrong Dinshan you've got the wrong Dinshan my brother floor is the hardware guy in the family I have a brother who does FPGAs and stuff like that he could tell you everything about how much it costs to encrypt stuff I can't sorry and back to your question when does that become mainstream as usual it will probably take a decade so the feedback we got so far is that there are a few markets that are already interested today because they have to deal with regulations constraining the kind of the responsibility in terms of data leaks and that's banks, medical stuff like that but also that because this becomes available of course the manufacturers are not being all they can to make this mandatory and so now it becomes a lot that you have to encrypt your data that way and so maybe there will in some cases you won't have a choice time will tell we'll see how this works no so when I'm seeing it's weak is that it's typically done to be done in real time it's still modern algorithms and the algorithms they use are dependent on a chip and frankly off the top of my head I don't recall which one they use but I think they have AES 128 somewhere some label for the first generation chips and now they are doing more complicated stuff so it's not the stuff that you can break with your computer without paying a lot of energy and by the way I suspect that one reason that triggered these efforts is I don't know if you recall like almost ten years ago there was this big push about memory stores and persistent memory non volatile RAM and this kind of things and everybody at the time thought that we'd have really cheap non volatile memory one decade later that's now and so the problem became for the cheap vendors I need to make sure that if someone plugs one of these non volatile chips they don't get all the data that was in there so that was probably a primary motivation to get started on this encryption stuff initially and in that space it had to be relatively robust anyway I apologize for the use of the world week I meant it's not as strong as you could do if you were really pushing the limits on your crypto but it's pretty good I'm out of time, I knew I knew that I'm sorry