So welcome to the third part of the EasyBuild tutorial. Today we'll cover a couple of more advanced topics. And I got the date wrong, but you know, today's date. So this is part three. Part four is still coming, it's still being planned. I'm working together with Luca from CSCS to get the contents together for this part of the tutorial because that's new, the gray specific part is new. I will plan a date. I guess it will be something around end of April, maybe for the last session. That's fine. We discussed it yesterday already in the software team. Okay. End of April, early May. Yeah, something like that. Whatever is fit. This is just as a reminder. So the tutorial itself is on the website for the demos. It's mostly demos today. There's, I think maybe one exercise, but we'll probably won't have time to wait for people to solve it, you can solve it by yourself. The important point here is make sure you use the centrally installed software stack. Otherwise you'll be installing lots of stuff from scratch. Load the EasyBuild module that's available there as the main way to run EasyBuild. There's one exception in one of the sections today, but I'll cover that in detail. And also make sure EasyBuild is properly configured to use your home directory, but for the build directories use a local temp directory. And this is also something we'll have to change for one of the parts today. This is the rough timetable. Halfway expect that it won't take three hours, but let's see what happens. We can definitely have enough time to cover all of that. So that should be okay. So let's start with using EasyBuild as a library. So for this, the configuration you have to do and put is very basic. Just pick up the central software stack with module use and load EasyBuild and then you're good to go. So this is this first part here. So up until now we've always used EasyBuild straight from the command line with the EB command, but you can also use EasyBuild as a Python library. So you can talk to the EasyBuild framework, use all the functionality that's available there, parse easy config files, run commands, all of that. You can use the EasyBuild framework for it as a library. There's a couple of important details that you have to take into account. The first thing is that you have to set up the EasyBuild configuration. So you have to make EasyBuild read the EasyBuild configuration files, the environment variables, and maybe also command line options depending on how you write the script. Where you'll be leveraging EasyBuild. It's important that you call this setup configuration function first before you do anything else with EasyBuild. There's some functions, probably most of the functions in the EasyBuild framework, expect that EasyBuild is properly configured. And if you don't, you will run into trouble. So I can maybe show you what goes wrong if you don't do this. So if you just start importing functionality from the EasyBuild framework, for example, to read a file and you try calling that function with, we have an example. Let me check, I have a file here, test for example.  Okay, it's a directory, fine. Other example, this Python file here, yeah. Anyway, a lot of stuff will go wrong if you try to directly use functions without properly setting up the configuration first. What you should do in a Python script or if you're running it straight in the Python shell, is setting up the configuration. So for this, you import the set up configuration method or function and call it. So you don't have to give it any arguments, you can, and I'll explain some of those in a bit, because this just sets up the EasyBuild configuration in the background. It will produce some output and you'll get some return value here, which you can use, but you don't have to. But from now on, anything you use from the EasyBuild framework will work as expected and will also take into account your EasyBuild configuration. So if you have EasyBuild configured to use a different modules tool or to use tickle as module syntax rather than the default Lua, this will be active in this session as well. By default, the set up configuration function will pick up the command line arguments that are default in Python. So if I would have started Python 3 with, if it's a script at least with command line arguments and you call set up configuration without doing anything special, it will pick up those arguments as well and take them into account. So that's useful in writing Python scripts and we'll see that in a bit. And if you want to silence it so it doesn't print this line here where it gives you the location of the log file, you can use silent equals true like this. Silent equals true. And then we'll still get a return value of course. So result equals, then it won't print you anything. If you don't do this, it will still print the location of the log file. So that's important. Whenever you want to use EasyBuild as a library, make sure you set up the configuration first. One important thing here as well as when you do this and I'll have to peak here. Yeah, so this function returns two things actually, options and something else, there's something else you can largely forget about, I think let's see. Yeah, that's just sort of an internal thing.  And now it looks a bit weird because I've done it multiple times. But if you just do it once, you can do the import, you do the setup and let's do it silently. And then use options.temdir, you get a temporary directory that is specific to this EasyBuild session. That's useful, you can use this, but you should also clean it up afterwards. If you don't clean it up, it will stay there. And to clean it up like we do here, here in the example, so we can use the remove directory function that EasyBuild gives us. And then we remove options.temdir before we exit the session. And then everything is nice and clean. So this one will not be there anymore now. That's important. This is the sort of minimal script you can run. When using EasyBuild as a library, you do the two imports, you set up the configuration and you make sure you clean up the directory at the end. Of course, this doesn't do anything useful, but this is like the starting point you should use when running EasyBuild as a library. Then what can you do? A whole bunch of things, depending on what you're after, of course, these file operations from the file tools module. So this example does, it downloads a file, it computes a checksum for that file and then it copies the file into the current working directory. So you'll see we're importing a bunch of functions here from the file tools module and then we use those functions one by one. So first to download the file, then to compute the checksum and then to copy the file. Of course, to figure out how to use these functions, you can check the API in the EasyBuild documentation. So this link here points to the file tools module API and for example, you can look for compute checksum here and it will give you the signature and it will describe the parameters that you can give it. So this is well documented. You can also do this straight from the Python shell. So let's import all of these actually like this and with the help, it's also the compute checksum. You can get the help for that function here. So the documentation is there right in your Python environment or in the EasyBuild documentation. So let's see what this one does. Let's just copy it and say example one, it goes. So what we're doing here is, first of all, we set up a configuration. We take one argument for the script. This is a download URL. We figure out the file name from the download URL and download it to a temporary location. So we're using the temporary directory that EasyBuild gives us. Once it's downloaded, we'll compute the checksum on this temporary part, print it and then at the end, we will copy the file to basically the current working directory. So if you try this, it should make it executable and then give it the URL. So there's an example here which downloads one of the files from the tutorial and then we can just run it and give it URL. And that will use EasyBuild to download the file, compute the checksum and then copy it here in the current working directory. So all of this functionality is right there. You can use it. You don't have to figure out how to do this with Python if EasyBuild knows what to do. And since we didn't do anything special here when setting up the configuration, we get this line for the temporary log file. That's useful when something goes wrong. We can open the log file and check what happened and maybe figure out why it went wrong. And it's also useful that we can give command line options to this script. So any configuration option that makes sense for EasyBuild also makes sense here. So for example, EasyBuild has a download timeout option and we can put this longer or shorter to make sure there's enough time to do the download until it times out. So it will happily pick up that option and use it. And if you give it something wrong, it will also complain that the option doesn't exist. So we get all of that for free just by doing the setup configuration. That's also what's done here. Then another example is running shell commands. For this, we have a separate module, the run module with the run command function. And this very short example, we're running the make command. After making short, it's actually there. So we run a which, which looks for the specified command in your part. If it's not there, it will exit early with an error. And if it is there, it runs make with the arguments that you give it on the command line of the script. So let's say, let's call this run make.py like this. Let me first break it. So give a different command name to show you what it does. U plus X. And just to run make, the first thing it will do is make sure that the command is found. So make X, X doesn't exist on the system. And then if we fix that again, and we just do run make without arguments, it will try to run the make command without arguments. And this expects a make file. So this goes wrong, but I can give it options as well. And these will be passed to the make command. So it will run make deseg version and then give me the output of that. So here you need to be a little bit careful when setting up the configuration, we pass an empty list to the arguments to prevent that the setup configuration picks up the arguments from here. Since these are options to the make command, easy build should not try and use them to set up its configuration. That probably won't work. Okay. And that's working as expected as well. And then another second, almost the last example, interacting with modules from a Python script. So basically running module commands like module load or module avail. Example module.py. This is a longer script. But here we use the modules interface or the modules to interface that easy build provides. So it will set up this modules tools, modules tool instance. And again, this will take into account the easy build configuration. So if easy builds configured to use Lmods, this will give you access to the Lmod command. If you're set up to use the environment modules variant, then you will be talking to the environment modules tool. But all of the methods that are provided by easy build will work no matter which tool you're using. So this should be a portable script. So here we'll check what type of tool we're using, which version, how many modules are available, how many easy build modules are available so we can run available without arguments or with the name of software. We'll check how many modules are loaded. And then we will load the default BZ2 module. So that's done here. We'll load and this expects a list of names because you can load multiple modules at a time. And before that we'll check the environment and also after loading the module, we'll check the environment to show what happens. Now the changes to the environment are of course local to this Python script. As soon as the script is done, nothing will be changed in the environment anymore. So let's write this, we can executable and show you what it does. So this one doesn't take any arguments or not any required arguments, anything we pass. So I can actually show that I think. So if we want to run this with the environment modules tool, you can try this and then easy build will try to use that. This will go horribly wrong because it can't find this modules tool. And even I have to tell it use module syntax tickle, of course, when using this, but even then it will still go wrong because it can't find the modules tool. But this just shows that easy build is picking up those additional configuration options. And by default it will run lmods. I will see all this output so it counts how many modules are there. It found one easy build module. There's five modules loaded by default. If you run on the list, you'll see the same thing. Before there's no value set for the EB root BZIP environment variable, but if we load the BZIP module, we see it loaded here. We can check the parts to the loaded module. So this comes from the central software stack that we've provided for this tutorial. And EB root BZIP is set after loading the module. But of course here as module is chose, this was only local to this Python session. It doesn't change anything in your shell environment. So I think this is also very useful when you're writing Python scripts to talk to a modules tool, you can do this via easy build. And then the last example is parsing easy config files. So something very specific to easy builds where here we take an argument, the name of an easy config file and then a list of parameter names that we want to inspect. We set up the configuration again, we parse the easy config file, and then one by one, we print the value of these parameters. So inspect easy config.py like this. I think I have an example in my history maybe, maybe not, okay. So let's do this side by bundle. And let's see dependencies. So here I just give it the name of an easy config file and we'll use easy builds to go and locate the file, parse the file, and then print the value of the dependencies parameter. And here we get, yeah, something that's already preprocessed. So it's not a raw value anymore, but we can give it a list of things as well. It may be a bit easier to look at. Software names, software version tool chain. So that's useful as well. A little bit more about this, there's this function, helper function here, which we define ourselves because the standard API of easy build is a little bit awkward to work with to parse easy config files. It has a function for that, but you have to use it in a specific way. So it expects this additional value here next to the location of an easy config file to parse it. And then for one easy config file, it may actually give you multiple parsed easy configs because there's a way in easy builds to have multiple easy configs in a single file. So that looks a bit weird. That's why we have this wrapper function that cleans that up a bit before we actually call it here below. And of course there's lots of functionality I haven't covered here. Anything that's mentioned in the easy build API, you can use. So most of the useful stuff is in the tools module. So anything in here you can leverage, you can play with the module naming scheme, maybe inspect tool chain things. So there's a whole lot of things you can do. So if you wanna do some coding or if you're missing a feature in easy build, you could of course look into adding it into the easy build framework itself or you could write your own little Python script to do this and you just use whatever is there in the easy build framework that is useful to you. So that covers this part. Any questions before we continue? So you've shown how to use, let's say, yeah the easy build tools for example, sort of as a library function. So yeah, you can do this in SPAC also to get the Python prompt and load the SPAC as a library. And they are, I remember especially in the manual, they show this as a good example for, let's say debugging and trying to understand why something goes wrong when you're trying to install something. You would like to see the exact value of some variable or some compiler or compiler flags and so on. I suppose you can do something similar with easy build or is it difficult if I want to dig into it. You can definitely do it, but you'll have to figure out which parts of the easy build thing you have to use. I don't have an example of that. But parsing the easy config file is already your halfway there. That's also what easy build does as the very first thing. It finds the easy config file, parses it. And then it calls another function. I would have to look it up to create the easy block instance. So it figures out from the easy config file which easy block should be used and has a function for that. And then once you know which easy block should be used, you can create an instance for that. And then you're very close to being able to inspect or even step through the installation yourself. I don't have an example of that ready, but you can definitely do it, yeah. So it's possible somehow to let's say, I want to see the exact compiler flags and compiler build command, for example, run at a certain stage. You can dig that out somehow, okay, good. Yes, let me see if I can quickly get that working. Let's see, the easiest way is actually checking the code itself. It will be a bit too much work, I think to do it on the fly here. But if you're interested in that, I can cook up an example that goes, take the next few steps or create the easy block and then show you how you can set up the build environment so you can start inspecting it. And yeah, if you want to, you can also find that information in the log file. Yeah, there's other places as well, but maybe you want to do it more interactively or step through it and basically look around a bit in various places, yeah, it's a useful use case. You can definitely do it, but yeah. Maybe that was a bad example, because it's true. I think with full logging, you can see that part, but there could be things which are, let's say not logged usually, or you want to see some more details so that it's this kind of. Yeah, or just inspect internal stuff, for example, inspect variables in the easy block or, yeah. Not again, I get it. You can definitely do this, yeah. It's a good use case. I'll try to come up with an example for that and share it with you. Okay, so let's move on to the next part, using easy build hooks. Make sure I have a clean environment here, okay. This is a little bit similar, but it's different. So it's also Python coding, and essentially using the easy build framework, but in a different way. The main use case for hooks is to change the behavior of what easy build does. So if you're doing an installation, EB and some easy config file, and you somehow want to intervene in the middle of that and do something extra or do something different, that's what you can do with hooks. So to use hooks, you'll have to implement a small Python module and tell easy build to use it. So there's a hooks configuration option, which you, for example, can set through easy build hooks. So let's try doing that and make sure I don't have, I do have, let me clean that up. So we can tell easy build to use our hooks Python module and as if it's not there, easy build will probably complain, should complain. Let's do the side-piting like this, yeah. So if you give it something that doesn't exist, it gives up because if you configure easy build to do something, it will definitely try to do that. But if we touch the file, it can be empty, that's okay. And then it will happily continue. Yeah, so you can configure easy build to use your hooks file by setting the corresponding environment variable and then it's happy. So what can we do with hooks? Let's first look how, which hooks are available. There's a long list of hooks that you can implement. You don't have to implement all of them, you can choose only the ones you want or need to use. And it's a very long list, but there's basically three different types. There's a start hook and an end hook. So the hooks are printed in the order that they are executed or considered at least by easy build. So the start hook and end hook are triggered at the very start and the very end of easy build. There's a parse hook, which is triggered every time an easy config file is parsed and you can do something there if you want to. And then we have a whole long list of hooks basically for every step that is performed by easy build, fetch, step, ready, step, source, step, patch, step, prepare, step, and so on. And there's a pre and a post hook for that. So you can, there's a hook that's triggered right before a step is started and right after a step is completed. And you can intervene in each of these locations. For the start and the end hook, important to know here is that the start hook is only triggered once, right after starting easy build and then it's never triggered again. And similar for the end hook, it's only triggered at the very end and also only once per easy build session. If you implement the start hook, so like this, so you can do, you define the start hook function and then you can do whatever here. Hello from the start hook like this. And now let's find an easy example. Let's do the silly bzip one and make sure it actually goes. And here we can already see that the start hook is triggered. So whatever you do in the start hook will be executed as Python code and you're basically intervening with what easy build does by default. What you do there is entirely up to you. You can run commands, you can use the easy build framework like we showed in the previous part to talk to the modules tool. You can check environment variables, check the system you're on, the host name, whatever you want to do is possible. Also very similar for the end hook, of course, you just define end underscore hook as a function and it will be called right before easy build ends the session. One important detail here is that you don't get any arguments to the start hook. At this point, there's really nothing easy build you can give you if you want to inspect the easy build configuration. So that may be interesting to do. Let's see, I think it's this one, build option. So you can inspect configuration options. So for example, debug, you can check whether debug mode is enabled and then let's hope I got this right. So by default, debug mode is not enabled. Let me make this a bit shorter so it doesn't run for long. Okay, so by default, debug mode is not enabled, but if you use dash-desk-debug, this should be set to true. So you can also inspect the easy build configuration and maybe if you see a mismatch between the host and the easy build configuration, you can give up very early. So that's a very site-specific thing that you can do in a hook, for example. So start and end hooks only trigger runs, don't take any arguments. The parse hook, which we can also implement the obvious name is parse hook. This one does take an argument. You get the parsed easy config file here as an argument. So every time an easy config file is parsed, the parse hook will be triggered right after the easy config file is parsed. And at this point, you still get a raw easy config file. So you basically get the contents of the easy config file itself without any additional setup that easy build does at the later stage. So at this point, you can actually intervene and change stuff. So for example, we can do build-opts. We can just inspect the build options for this particular installation. Let me stop it a little bit earlier. Fetch, yeah. So here we see the parse hook being triggered, which gives us the value of the build-opts. That's useful, but we can not only inspect the value, we can also change it at this point. So we can do build-opts equal example, doesn't really matter. This will of course break the installation, but it doesn't matter too much. And let's stop after the build step. So we see it's actually being used. See, why didn't this get used? Build-opts, it's in here, an example. So the example we injected here as build options was passed to the make command. There's other stuff being done as well. That's being done by the easy block that's being used, but our build-opts are being injected here. And then in this example, it breaks the installation. So parse hook is very useful if you want to change certain easy config files without changing the easy config files themselves. So you can do it programmatically. And you can of course do it conditionally. So for example, only for BZP2, you can actually do something and not for something else. So this will still be triggered for BZP, but it will not be triggered for anything else. Okay. So just a quick question. So this step was where you suggested, let's say you have this hypothetical scenario, which I had where I would like to change the tool chain for, or replace the tool chain for all my easy configs, for example, this would be the place to do it, right? Yes, you can do it here. You can change any easy config parameter because you do it very early right after parsing the easy config file. And easy build will only take additional steps like looking for the tool chain or looking for, I can show this as well. So let's change the tool chain to name Peter, whatever, say you have your custom tool chain version 1.0. This will go horribly wrong, of course. Again, well, it doesn't really matter. It will give up when it does the prepare step. Yeah, and here it even says tool chain Peter not found. So I'm using a tool chain that doesn't exist. Ineasible doesn't happen with that. In this case, this is a system tool chains. If I change it to an existing one, this doesn't make much sense, but this should work. Let's do trace, so it shows us what's happening. Preparing, you see here it's loading this tool chain as the one we selected. So this is early enough that you can basically tweak whatever is in the easy config file or add additional stuff to it as you like. And again, without changing easy config files or without changing the easy build framework itself or figuring out where to do that. So the hooks are very useful. Of course, as you can tell, it's broken here for whatever reason, but whatever you do in the hooks can also break the installation. So you have to be a little bit careful and it's probably recommended to do things conditionally. So only for certain software names or for example, in a start hook, you could say, if not, I know what I am doing with the easy build error. So this is how you report an error. This is how you report an error to easy build. The little blog need to import this, of course. But here in the start hook, we're just saying that some environment variable has to be defined and if not, it gives up very early. So this is like expert mode, I guess. So here the start hook will trigger, have a missing import. So here it just says you don't know what you're doing and then you're supposed to know what you can do. And if you set the environment variable, then it will be happy and it will continue. So if you could have your own custom hooks that you don't want to make sure that are not used by accident, for example, that's also possible. And then the pre and post step hooks, that's another example. So here you have to be a little bit more careful. Let's keep this even though it breaks the installation, it doesn't matter too much. Say if we do pre build hook, here we get also one argument, but it's not the same one as for the parse hook. At this point, we get the past, the prepared easy block. So not the easy confi file, but the easy block that actually runs the installation. If you just print this, it will probably show us. Pre build hook, let's keep doing this. So pre build hook here, it's running the, oops, running the build hook and it's showing us that we get an easy block instance. In this case, the BZ2 easy block because there's a specific easy block for BZ2. That's useful. We can also still access the parsed easy confi file through the CFG variable in the easy block. And let's check build up again here like this. So here we get the value for the build options as well from the parsed easy confi file. So in this, for the step hooks, you get both the easy block, which is fully prepared and executing the installation. And you still, of course, get access to the parsed easy confi file as well, which was used to create the easy block. And again, of course here, you have to be careful. This will be triggered every time the build step has started, no matter which software is being installed. And what you're doing here is probably very specific to the software or to specific software. So you wanna make sure you only do this for certain software names, certain software versions or under certain conditions. Some things to take into account here when using hooks. So you have to be a bit careful. This is a very powerful feature. You're basically jumping in between easy build somewhere. You can take additional actions. You can change things on the fly like we do here. You can totally break things as well. Let's see how we can break something. If we do something stupid like we change the name to none, for example, it's very likely that easy build won't be happy with that. Yeah, so here it says, okay, you gave me a none that doesn't make sense and it just crashes and gives up. So you can very easily break easy build through hooks if you're not careful. Or you could write a hook for a very specific use case for a specific software and then when you're installing something else by accident, you broke or you changed that installation as well. So make sure you add conditions and things like this to avoid breaking things and make sure you test things well before you actually use the hooks in production, of course. Another thing to be a little bit careful with is template values. Let's see if we can give an example of that. We can just actually use this one, I think. Yeah, so let's use this example, clean up this junk and just use this one. So what we're doing here is in the pre-source hook, we want to make sure that the software version that is specified in the easy config file is also found in the name of the source file. And if that's not the case, then something's very wrong and we want to give up or in this case, I just print a warning. So doing that check is quite easy. We loop over all the source files in the sources easy config parameter and we check if the version is a part of that. If it is, okay, it's actually different. We were checking for hard coded versions in source file names, okay, that's what's going on. So we don't want to hard code the software version in the file name, we want to make sure that this template is being used. So of course, when doing this check, we have to make sure we're not looking at the template because then the check is just useless. So we have to make sure we get the easy config parameters without templates resolved. So that's why we have to use this wit statement here to make sure that templating is disabled when we inspect values of easy config parameters. Let me show you what goes wrong if you don't do this. So if we don't do the disabled templating, this should work and we can do the BZIP example again. Don't need any of this. So here we already get the warning because when we look at the value of the source file, the version is already in there because this is a fully resolved template. So of course the version is going to be in there. If we do it with disabled templating, let's push this under there and let's also print the value here. So you see what we're checking. Stop after prepare. Oh, whoops, that was not correct. In this, correctly printing the value. So now we're looking at this value, the one without the templates resolved. And this is actually the right place to check for hard-coded versions. If we see this, we're happy. If we see a hard-coded version, we complain. We print the warning and we tell the user, please use this version template instead. So don't hard-code versions in names of source files. So if you wanna be more strict in the easy config files you manage yourself, you can add additional checks through hooks like this, but you have to be a little bit careful with templating. And then here, changing easy config parameters on the fly which I showed for BZIP very quickly. Sometimes it works, sometimes it doesn't. You have to be a little bit careful depending on what type of easy config parameters you change. So if you're changing a string value like built ops, like I was doing before, it's fine. You can just do assignments and that's okay. So sometimes you want to add to an existing value rather like we're doing here in the big Z example. So let me clean this up again and start over. So what we want to do here when we're installing the big Z tool, we want to make sure that the read me file is also being installed. So being copied. Big Z uses the make CP easy block which listens to the files to copy easy config parameter. And apparently for whatever reason the easy config files included with easy builds don't copy the read me file and we want to fix that in a hook. So we don't have to change all the big Z easy config files. So let's see what this does. I think I have this prepared. Yep. I will do a rebuild because I may have this already. So the hook was in pre install hook. So it will only trigger before the install step. It's already going wrong here because it didn't properly configure easy builds. I'm not using a local directory for the build path. So that's why it goes wrong there. The building should work. Yeah. Now it's triggering the pre install hook which injected the read me into the files to copy. And then if we check where are things now in here. Okay. I'm still using the wrong installation directory but that doesn't matter too much. In here, we can see that the read me file is also being copied which it doesn't do by default and to prove that it doesn't do that by default, I'll remove the hooks and do another rebuild and show you that it's not have to stop telling easy build to use the hooks. I do a regular installation. The read me file will not be there. So the hook has successfully triggered. Yeah. Now it's not there when I do the default installation. So this way you can also, you can fix problems in easy copy files by changing stuff. Now what's important here is that we can use our inspect script maybe files to copy. Yeah. So the default value for files to copy is this where it copies two binary files into the then installation folder. This is a list of things to copy and we don't want to override this list. We want to add stuff to it. So we just want to do an append in this list when we do the hook. So that's where we have to be a little bit careful. Let me show you the wrong version. This doesn't work because when we take the value here, we're actually getting a copy of the actual value not a reference to the original value. So when we append something to the list here, the append will work, but actually nothing is happening. We're not actually changing the easy config parameter. If we do this updates methods, so this is an update method for the easy config instance then it does work. So whenever you're changing things like lists or Python dictionary, so mutable values in Python, you have to be a little bit careful. If you want to make sure that you're doing the right thing, just always use this config.updates method and then it will work. Or you can also disable the templating mechanism because that's actually why it goes wrong in the background. If you always do operations with templating disabled, then it will also work as expected. So if you do this the wrong way, nothing will crash or go wrong, but the read me file will not be copied when you try it like this. And that's I guess a bit hard to debug or difficult to figure out why it's wrong, but it will just not work. So that's an important detail to be aware of when using hooks. Any questions on the use of hooks? This is a very powerful feature and you can go a long way with this, but of course you have to be a bit careful. Okay, so there's no questions. I'll continue with submitting installations as slurm jobs. So this is also a very interesting feature. I have something on the slides on this. So here we have to be a little bit more careful when setting up the environment on Putty next to using the central software stack and loading EasyBuild as a module. We have to change things a little bit in the EasyBuild configuration. So we still use our home directory for sources software modules. For the build part, we're actually gonna change it to RAM disk, so to DevShmem, because the default location slash temp user is only very small on worker nodes. And DevShmem has gigabytes of space and that's enough for the examples that we will do. I'll explain later why this is important. And we also have to change the job backend. We have to tell EasyBuild to use slurm as a job backend. By default it uses a different library, GC3py, which is not available on Putty, I think, which will certainly not work because you have to configure it. But this has been the default for a while and since then we've also added support in EasyBuild to talk directly to slurm. Whenever EasyBuild submits an installation as a job, it basically runs an sbatch command. And on Putty specifically, we have to make sure that's done to the right slurm account. So you also have to set this environment variable to make sure that's done correctly. So especially the last one, you will not see in the tutorial itself because that's very specific to Putty. We'll make sure you have this properly set. We'll just copy paste all of these and start with a new session. So I have things properly set up. So this one is very important on Putty to make sure things are working correctly. So what are we doing here? Rather than just installing things on whatever node we are logged into, in this case, the login node, we wanna farm out, we wanna distribute installations to a cluster. This is very useful when you're doing lots of installations. For example, when you have a new cluster or if you want to reinstall all your software after doing an OS update and you have an empty cluster sitting there anyway, you might as well use it to do the installations. How does this work? EasyBuild has a dash dash job option which changes the default behavior of doing the installation locally. Instead, it will create a job script, submit that job script to the job backend, in this case, slurm, and then the actual installation will happen in the job. So important here is to do the configuration first. So whenever you use eb dash dash job, the full EasyBuild configuration will be sent into the job script. So slurm sort of does that by default, at least for the environment. EasyBuild does the same thing, but it will take its whole configuration the way it was configured when eb dash job was used and it will make sure that the job uses that same exact configuration. That means that even EasyBuild configuration files or EasyBuild environment variables that are set when doing the job submission will also go into the job even when these configuration files are not in the job environment, that configuration will still apply. Same for environment variables, you don't have to have these sets in the job environment because EasyBuild will copy the whole configuration. What it basically does is it translates all the configuration options to command line options and it will generate one long eb command to make sure that things are properly configured. So that's important to know whatever configuration you have active at the time you run eb dash dash job, that will also happen in the job script. So we have to make sure that we use slurm as a job backend so make sure that EasyBuild is configured to do so. That's one of the environment variables we're setting here. And then it basically boils down to just running eb dash dash job. So running your regular eb command with the name of an easy config file and adding that dash dash job to it will make it generate the job script and do the job solution. By default, EasyBuild will submit a single core job requesting 24 hours of wall time. Whether that's a good default, I'm not sure, but it sort of makes sense I guess. And of course, you're probably not happy with this. So you can change the number of cores it asks for with the job course configuration option. You can change the wall time it's asking for with job max wall time. And usually you use these on the command line. You can add them in an EasyBuild environment variable or in the configuration file as well. If you wanna change the default, you can certainly do that too. But typically this is the type of command you run when submitting installations as jobs. In the background, EasyBuild will run an sbatch command. So if you need to change things in the way that EasyBuild submits the jobs and there's no corresponding configuration options for that, at least with slur you can still set sbatch environment variables and those will be picked up automatically when sbatch is run. So for example, to specify an account like we're doing here, we can just set the sbatch account environment variable and then when EasyBuild runs sbatch, it will pick that up and it will be happy. Same thing for the partition.  Usually when you do installations via jobs, it's maybe because you have a big installation that takes hours to run and you don't wanna do that on the login note or you have an installation that has lots of dependencies missing and you wanna do all those dependencies in jobs as well. EasyBuild handles this in a smart way if you combine dash dash job and dash dash robot, it will not submit a single job but it will submit a job per installation that needs to be done. And it will also set dependencies between these jobs. So if you're installing something that requires five other things, your main job will wait until the five others are done correctly and only then will it proceed. We just said basic slurn dependencies between these jobs. So those get handled correctly. And then before we go through an example, some attention points here again to take into account. There may be differences between the login notes that you're working on and the cluster worker notes. So sometimes on the login note, things are a little bit different compared to a worker note and you have to be a little bit careful there because again, the easy build configuration that is active when doing the submission will be complete onto the worker note. So maybe the configuration that you have to do on the login note is a bit different from what you have to do on a worker note. And one specific example of that is the build part. So that's why we're setting this one to the shmem rather than slash temp because on the putty worker notes slash temp is very small. It's only a hundred megabytes, I think. And it does give you a separate directory where you can run into, which where you have more space. That's a little bit harder to configure in easy build unless you use hooks, I guess. But an alternative is just using the shmem where we easily have gigabytes of RAM disk space available. So we just use this one rather than slash temp. So, and the main reason we do this is because we need to do it when running installations as jobs on the worker note. So here, for example, as well on the login note, we're running slash temp and we're happy with that. But when we're doing the job submission, we change this to, in this case, slash local disk because we know using slash temp on the worker note is not gonna work. And when we use slash local disk, it will be happy. So you can do it like this in the environment like I'm doing here or only when running the job option, you can change the configuration on the fly and this will be copied into the jobs that are submitted. So that's important. You may run into broken installations there if you're not careful. Another issue or certainly something to take into account is the temporary log files that Easy Build creates and also the build directories. So Easy Build will generate a log file in slash temp where it keeps track of what's going on in the installation and if the installation fails, you can check the log file and figure out what went wrong. When you submit installations as jobs, those log files will be local on the worker notes. As soon as the job completes, usually the temp directory is cleaned up and you will lose the log file. So that's annoying. You can fix this in two ways. There's probably other ways as well, but two ways that are easy to do, you can change the location of the temporary log files with an Easy Build configuration option. So in running EB dash dash job, you can tell it, okay, those temporary log files put them in my home directory because if something fails, I want to be able to check them afterwards. If the installation completes successfully, the log file will be cleaned up automatically. So you don't have to worry about this becoming too big, but if the installation fails, the log file will be kept there. So it won't be cleaned up like it is when in slash temp. So that's useful. Another option is to use log to standard out, which is absolutely horrible if you use it in a regular setting. So log to standard out, just spits out, load, Easy Build. This is pretty horrible if you use it in a regular installation because you'll see the full log file scrolling to your terminal. And this is probably not very useful, but when you're running installations as jobs, this can be useful because then you at least get the log file itself in the job output file. That could be interesting. Of course, those output files may are going to pile up and they will be fairly big, maybe even several megabytes. So you want to be careful with that, that you're not filling up your home directory that way. But at least it's an option as well to get the log file for the full installation and be able to inspect it when it fails. The same applies to build directories, but there it's a little bit more difficult. So if an installation fails and you're using slash temp as a build directory, that will be cleaned up automatically when the job fails and you'll lose all that information. So you won't be able to inspect things again. That's a bit more difficult to handle because if you want to do build directory on a shared file system, you have to be very careful. Like we've seen on Luster, that doesn't really seem to work. On GPFS, it's okay usually. And that probably depends on how the file system is configured as well. So this is a bit more difficult to deal with. One option there is when you see the installation failing, you just redo it either on the login note or you start an interactive job and you redo the installation there. If you really need to inspect the build directory, you can still do it like this, of course. So that's maybe a small downside of running installations as jobs when the installation fails. And then here also a small side note to mention since a couple of versions, easy build create log files when starting an installation to make sure that the same installation is not being run twice on two different systems and then they step on each other's toes and both installations fail. So easy build creates a log file whenever you start the same installation again somewhere else, it will hit the log file, it will stop there and it won't actually start the second installation. When an installation fails, that log file is cleaned up automatically. If you cancel an installation manually with control C, easy build kicks in in time and also cleans up the log file, so that's fine. But when an installation runs in a slurm job and the slurm job gets canceled either manually with sCancel or because it runs out of wall time, that log file is typically not cleaned up. And this is sort of a bug in easy build but it's not that easy to fix because when the job has canceled in slurm, apparently it doesn't send the signal that the job receives down to the job steps. And since easy build is running in one of these job steps it never sees that it's being canceled and it can't clean up the log file as far as I know. That's a bit annoying if you submit an installation as a job and the job gets canceled that log file will stay there. And if you retry the installation, you will hit the log file and it will abort instantly. To work around this, you either clean up the log file so that the log file is in the software logs sub-directory or you tell easy build to just ignore logs which is probably not the best thing to do but at least that's an easy work around to try. Okay, so keep that in mind. You may see log files, especially if you're cancelling jobs or if you have failing jobs. So don't be surprised if you see that popping up. Okay, let's then look at the concrete example. So make sure easy build is properly configured. That should be already done here in this session. So let's just inspect the configuration. We're using that shmem as a build path that's important because slash temple does not work on the worker nodes. We're using slurm as a job backend, so that looks okay. And for everything else, we're using our home directory. And also very important, the easy build is loaded and we do have the central software stack available here. So all these models are visible to easy build. So that looks okay. This is set up. We have easy build loaded. We have the sbatch account one set up correctly to our project here on Putty. So that looks okay. So we're ready to start submitting jobs. Let's try this for this Augustus software package. So this is something that needs a whole bunch of dependencies. Let's check what's still missing here. Probably, yeah, this should match a whole bunch of things. Boost is missing, a couple of bioinformatics tools, sweet sparse, which will take a while to install and so on. So that looks like a lot of work to do on the login node. So let's try doing this in parallel across different worker nodes. What we'll do here is we'll do the installation but we'll use dash as job. We'll tell easy build to use 10 cores and one hour of wall time, that should be enough. So the one hour is for each installation separately. So as the jobs, the course is 10 cores per installation job. We'll do robots to make sure all the dependencies are taken care of as well. And we'll also enable the dash trace so we get some meaningful job output files that at least show which commands were being run in the jobs. Let me run this one first because this will take a while to complete the whole thing. So now easy build will check which dependencies are missing and it should submit 11 jobs, one for the 10 dependencies and then one for the Augustus itself. And you see before it's doing that, it's making sure that all the source files are in place. So if it has any missing sources, it will download these first on the login node and then it will only go ahead and submit the installation job. So it takes into account that the worker nodes may not have access to the internet or at the very least it tries to make sure that the sources are available first before it submits the jobs because there's little point in submitting jobs if the source files are not going to be there. When it's done downloading all these things, it will submit 11 jobs and then we can check with the SLURM command, with the SQL command rather, how these jobs look like. So this is taking a little bit to download those files. Hopefully that will complete. And here, I guess I can already explain the output. So if we, EasyBuild has submitted all those jobs, it will set dependencies between them as needed. So most of the jobs can run, each in there can start straight away because they don't have any required dependencies that are missing. Augustus itself, of course, has to wait until all the others are done. And apparently also BCF tools and suite parts have to wait until one of the dependencies is in place. So now we can check with SQ, what is submitted. And it seems we were a bit greedy with the 10 cores and all of these are still pending. So of course we're at the mercy of the SLURM queue, whether these will start or not, but at least they're nicely waiting in the queue. And we do see the dependencies being set for BCF tools, suite spars and Augustus itself. So eventually these will start running and we'll see the installations pop up, we'll just let that sit there. We won't wait for that. When all the jobs complete and you check the modules that were installed in your home directory, you should see 11 additional modules, one for each of the dependencies and then one for Augustus itself. And then of course you can check that Augustus is there and start using it. So already with 11 installations that maybe we're doing that, at least if the queue has a bit of space. But this is for example, what we do when we're testing EasyBuilds, we run a regression test that where we install as much or reinstall as much software as possible in a sandbox and that involves often 10,000 installations. And basically what we do is we run a single EB command which does this job, does this robot enabled and it submits all those installations into a slurm queue and then after a couple of, after a day or two, we check the result and we inspect what went wrong. Any questions on this job submission feature? I have one. Is there a way to also automatically clean the build path because on some clusters, you may be building on a file system that's not cleaned automatically at the end of the job. Well, when the installation finishes correctly, the build path will be cleaned up automatically. Sure. You also will directly be able to stay there if it might fail. Yeah, so for that, I don't think we have something in EasyBuild itself. We have to check because there's lots of configuration files or configuration options. I don't think we have something that cleans up, that always cleans up the build directory. But this could be a configuration option that we add. First, even if it's easy to say on actually, I don't know whether we will need you, don't know me or not, but I don't know if we may have other problems. That's pretty easy to implement. When EasyBuild sees a failing installation and now it just doesn't touch the build directory, but you could force it to, of course. I guess it might even be possible to do it in a hook. Yeah, so the end hook, for example, you would, yeah, end hook, it will work. You could check how EasyBuild was configured, so which build part it is using and just clean out that whole build part. Yeah, you have to be a little bit careful, I guess, if that build part is a shared directory between different installations on the same node or if it's on a shared file system, then you don't wanna clean up the whole build part, of course, only the ones that you touched. But yeah, if you're smart enough, you can probably do it in the hook, yeah. Unless, of course, the end hook would not be called if a job fails. EasyBuild. No, it is. It's always called. Yeah. Even when the job fails, the end hook is always called. I think on our clusters, we have something that users can define something that is executed at the end of the job in the, I think, job epilogue script or so. That could also be a means. So, for example, EasyBuild could just fail for whatever reason, and then maybe whatever you define in the end hook may not be executed. Yeah, if there's a hard crash somewhere in the middle, then yeah, the end hook will not be triggered. That's true. But if it's just a failing installation, then EasyBuild will still trigger the end hook. There's certainly scenarios where the end hook will not be triggered, that's true. Okay. Any other questions on this? There is a question in the chat. Yeah, so we can use the coffee break and wait until this is completed, but it's actually, it's going, some of the installations are done. Yeah. So I would, I just asked the question on the chat. It is the question about, because you are using this shared memory for a temporary directory. And in fact, we are using some kind of RAM disk, which is just at the start of the job, we could create the variable, let's say MFS. And could we just give it to the Slurm config, to the EasyBuild config for the temporary directory for builds? Or we have to, yeah. You know, I'm talking about the variable, which is not accessible at the login node when you are creating the environment, but only inside the Slurm job. Yeah, yeah, I understand. So right now, the configuration that is done is static. So you do Dev Shmem dollar user, and that's just copied into the job. If you need something more dynamic, like for example, using an environment variable that is set only in the job, for this type of thing, you need to use a start hook, where you tell EasyBuild, look in this environment, right after EasyBuild has started, you check the value of this environment variable, and then you do something with it. One thing that's a little bit difficult to do right now, but it's something we're gonna fix quite soon, I think, is once EasyBuild has started, it's fairly difficult to change the configuration when it's already running. That's something we will fix. But what you can do now in a start hook, if you're, for example, have local FS defined as an environment variable, you could create a sim link to whatever location EasyBuild is using. So you could say slash temp slash user, EasyBuild, please use that as a built part. And then in the start hook, you could create a sim link to the right location, and then EasyBuild will just use that and be happy. So that's a trick you can already do. Yeah, yeah, yeah. And hopefully soon we will allow you to reconfigure EasyBuild while it's already running. And then you can more easily do it in the start hook. Great idea, thanks. So jobs are still going. I'll put it, it's almost done. I'll just let it sit there. Let's look at module naming schemes. So this is an important aspect as well of EasyBuild where you basically define the user interface so how people will be accessing the software that is installed with EasyBuild. The way the modules are named, so how, what module name will be used for an easy config file like this, for example, is determined by the module naming scheme. By default, EasyBuild uses its default module naming scheme, which is very close to how easy config files are named. So software name slash version dash tool chain and then maybe a version suffix at the end. You may or you may not be happy with that and you may want to change it and there's various ways of doing that. Next to the standard EasyBuild module naming scheme, there's a couple of other naming schemes that are included with EasyBuild itself. And you can check what those are with the avail module naming schemes command line option. A particular one we'll be looking at today is the hierarchical module naming scheme and I'll explain what that means in a bit and we'll also look into implementing our own custom module naming scheme. So one that's not included with EasyBuild and how you can do that. So let's first look at the hierarchical module naming scheme, what that means. So that's a very different way of organizing the modules. So the standard module naming scheme, we call a flat module naming scheme because everything is basically flat. You always see all the modules when you do a module avail and the names of the modules are enough to identify a specific installation. So just by the name of the module file, you know which installation it's referring to. This has a couple of downsides, which I'll explain in a bit. With a hierarchical module naming scheme, you organize the module files in a different way. You have different levels of module files. So they form a hierarchy and that has a couple of advantages. It also has some slight disadvantages. The typical module hierarchy has three levels which are also shown in the figure here. There's a core level. So this is anything that was installed with a system tool chain. So anything that doesn't use a compiler controlled by EasyBuilds goes into this top level, the core level. For example, GCC itself, which we installed with EasyBuilds, but is built, is compiled using the system compiler, goes into this core level. Then below the core level, we have the compiler level. So anything that was installed with a compiler only tool chain provided by EasyBuilds goes in here. And then in the MPI level, we have stuff that depends on both a compiler and an MPI library that are both controlled with EasyBuilds. You can define a module hierarchy with as many levels as you want to or totally different levels compared to these. That's entirely up to you. But this hierarchy is also what the standard hierarchical module naming scheme that comes with EasyBuilds implements. So how does this work? When you have a module hierarchy, at first you only see the modules that are on top of the hierarchy. So you see only the core modules. In this case, only a compiler. In this case, we only have a single one. So if you run module avail, you'll only see this GCC module. You won't see anything else. To get access to the other software that is installed, you first have to load one of these core compilers. And this is not an official term, but I call these gateway modules.  Whenever you load a compiler, you also, you make the modules that are built with that compiler available for loading as well.  And then when you pick one of these MPI modules and you load it, for example, you load open MPI, then you make additional software available that was built with both this MPI and this compiler, for example, FFTW or HDL-5. I will see a hands-on example of that at the end of this section, how that works exactly. So this is also described here at length in the text. So a couple of things that are typical for a module hierarchy. So not all the module files are directly available for loading. If you do module avail, you'll see only a limited set of them. And you need to load some of these gateway modules to get access to additional modules before you can load maybe the software that you're after. So why go through this trouble? This has a couple of advantages. So first of all, you have shorter module names because the tool chain with which a module was built is sort of encoded in the location where it is put. So this is actually the full module names that you see here. So open MPI slash version, that's all there is to it. You don't have to encode the tool chain that was used in the module name because that's already implied. You can only actually see this module when the corresponding compiler tool chain is already loaded. So there's no point in mentioning that again in the module name. Same here for HDF5. This is the full module name for HDF5 because just by the way you access it, you know which MPI and which compiler this was built. So you have very short module names, that's nice. You also see less modules at the time. So module avail is a lot less overwhelming. And you see less modules because first of all, you have fewer modules available when you start from scratch. You only see the core modules, but also when loading additional modules and loading gateway modules. In this case, five out of six modules will be available. As soon as you have GCC and open MPI loaded, MPI will not be, but in a real software stack, these branches are a lot bigger and you may have a thousand modules, but at any single point, you may only have 200 modules visible in module avail, for example. So it helps a lot in keeping overview of modules and all the modules that you see in the module avail output are compatible with each other. They are built with the same compiler tool chain, built with the same MPI, so you can use them together and combine these tools that were built with the same tool chain in the same session. Of course, that depends a bit on how the module hierarchy was organized as well. You still could get conflicts depending on Python version, for example, which is not a part of the module hierarchy, but it could be if you wanted. There's also minor disadvantages in using a module hierarchy. So one of the advantages is that the overview of available modules is less overwhelming. The disadvantage is that not everything is directly visible. So if people only use module avail, they will not see everything that's available. So that's a bit annoying, but there's a workaround for that.  So why do they have to care about a compiler? If all what they care about is their bioinformatics tool and maybe they don't even know what a compiler is. So all of these things are explained here in the sections as well. I think I covered all of them. So short module names, you don't see all the modules and that's both good and the bad thing. You can only load compatible modules together at least in terms of compiler and MPI. You don't see all the modules in the hierarchy with module avail, but a tool like LMOD is very much aware of organizing modules in a hierarchy and has a separate sub command for that module spider. So at module spider, you can look for software no matter where it's located in the hierarchy and no matter whether you can load it already or not, LMOD will always tell you whether the software is there somewhere or if it's totally missing. I will see that in the example as well. And then these gateway modules, yeah, depending on how experienced their users are, they may or may not have meaning to them and they may get annoyed if they have to load other stuff before getting access to the software they're interested in. We'll get back to a detailed example of using a module hierarchy at the end. But first, let's look at how to implement a custom module naming scheme. So something that's not included with Easeable itself. How do you write your own? To come up with your own module naming scheme, first of all, you have to do a bit of thinking because it may seem straightforward to do so, but you have to make sure that the module names are unique. So whenever you do an installation you don't get a clash in module names for two different installations. And especially when designing a module hierarchy, you have to be very careful what you do to do this well. So the very basics of it to use your own custom module naming scheme is that you implement a small Python module file which defines a class that arrives from the module naming scheme class with this part of the Easeable API. And in this class you have to at least implement this determined full module name method which takes an easy config file and returns a string that represents the module name. So that's the very minimal thing you have to do. Depending on how you define your module naming scheme you may have to also implement this additional method which does a check between the module name and the software name. So it checks whether the module name that it gives you or that you get and the software name that you get whether there's a match between those. And if you, this has a default implementation but that may not match what you do when your module naming scheme. So you may have to change this as well. And I'll show an example of that. And the argument you get to this method can either be a full easy config instance. So we're fully parsed easy config file and then you have all the easy config parameters or it can be a small Python dictionary that only provides these four easy config parameters. That's because whenever you specify something as a dependency in an easy config file name this is all the information you get. And as long as that's enough to determine the module name then Easeable doesn't have to go and look for easy config files, parse them to determine the module names for all the dependencies. So it's better if you stick to just these four but if you need additional easy config parameters you can tell Easeable that your module name needs more than just these four and then Easeable will make sure to always give you an easy config instance whenever determining a module name. And which easy config parameters you need you can specify via this class constant required keys. That all sounds a bit abstract. There's more details when defining a module hierarchy of course, this determined full module name is not enough. You have to basically design your different levels and control which modules go and which level, also how the levels are named. All of this you have to define yourself when defining a module, a hierarchical module naming scheme. The standard example of that is the one we include with Easeable. This looks very complex because it takes into account lots of different tool chains and combinations of things. And it tries to come up with a proper naming for all of them. So that looks a bit complex but you see things here like the required keys. So when with a hierarchical module naming scheme it needs name, version, suffix and tool chain. It also needs module class because that's how it distinguishes between a compiler and MPI and something else. So this is how it determines the different levels for example. And next to determining the full module name which in a hierarchy looks something like this. So the name of the level and then the name of the module or the name of the level and the name of the module. So the full module name also includes the location in the hierarchy where it's located. And then the short module name is the user-facing thing. So you have to tell Easeable to both how to determine the full module name, how to determine the short module name. This one I'll skip but also here you need to figure out what the name of the directory is where the module file should be located. So all these things you have to specify and that it's certainly a lot more tricky for a module hierarchy than in this for a flat naming scheme. Once you've done all that work and we will show a quick example of how to do that you have to configure Easeable to actually use your naming scheme. And for that you have to do two things. First you have to include your Python module via include module naming schemes. So either as an environment variable or in the configuration file or whatever. And then you also have to tell Easeable that this is the active module naming scheme. So you both have to include the Python module and tell Easeable that this is the one you want to use. So let's try doing that with this example. So this is an implementation of a custom module naming scheme which is relatively short. It looks a bit weird. I would never use this in production but it gives you a good idea. So what we do here is we name, we use software slash version tool chain. So very similar to what Easeable does by default. Here's an example. But we do a couple of things. We do everything lowercase. We don't like dash characters. We replace them with underscores. And we also put the version suffix directly after the version. So not at the end of the module name but directly after the version. So if there's a sub version suffix it would go here before the tool chain. Whoever wrote this likes this better and that's their good right to do it like this. And to make Easeable name modules like this you implement this small Python module which derives from the module naming scheme class. And you define two methods in this case the determined full module name. So this previous is a string that gives you the full module name. So it takes the version. It adds the suffix if it's not empty. It adds the tool chain except for system tool chain and then we don't add any tool chain part. And then it makes sure that there's no dashes only underscores and it avoids double underscores in the module name as well. And we do everything lowercase. Mainly because of this lowercase thing and also replacing dashes we also have to implement this additional method which checks whether this module name is a module for this software. And yeah, that's a very easy check. We just check whether the module name starts with the name of the software where the name of his lowercase and the dashes are replaced with underscores. If that's the case, we return true. If that's not the case, we return false. And that's how Easeable makes sure that the module names are correct for the software it's trying to install. So let's try using this. You check my environment again, this looks okay. So let's do example mns.py. Let me just copy paste this in a Python module. We configure Easeable to use it. So let's use environment variables for that and do example mns.py. So if we now check avail module naming schemes we should already see it popping up. This one, example mns. So now Easeable knows about the naming scheme but it's not using it yet to make it also use it. And we can check that if we do show full config module naming scheme. Yeah, so by default it uses the Easeable module naming scheme if we also define this environment variable, then Easeable will switch to the example module naming scheme that we're defining here. And now to check what things look like we have an example. Yes, we can do a dry run. I'm just not happy with that. I'm not sure if it's not finding this easy config file. But if we do a dry run and here I'm just using the Python 3 side pike bundle. So without this version suffix, the dry run output will show us all the module names what things will look like. So here you can inspect whether the module naming scheme you have in mind is actually fully working and reduces what you expected to produce. And then of course, when you actually do installations it will also use these module names when creating module files. So let's now look at the example of a module hierarchy. So a more complex alternative module naming scheme. I'll do a demo of that here. There's a couple of important things to take into account on Putty. So one thing you have to avoid when playing with different module naming scheme is that you mix modules that were installed with different naming schemes because in some cases a module may be named the same in two different naming schemes but if it accidentally picks up one from a different naming scheme the contents of the modules may actually be different in terms of what they do. So to avoid the trouble here, let me see if I have it here as well. We will start from a fully clean environment. So we will do a module purge to unload all loaded modules and we will fully hide all the modules in the current module part. So we won't actually do the module use to pick up the central software stack at least not at this point. So what we want is an empty module part like this and we don't want to have any modules loaded at all. So that's mainly to protect ourselves from running into weird situations where we're mixing things with different module naming schemes. Now we do have one problem. We can't load EasyBuild anymore as a module. There's ways around that. You can install EasyBuild in a hierarchy but it's a bit tricky to do that at least at first when you don't have EasyBuild yet. So to work around that here we'll just do a pip install dash dash user. So we'll install EasyBuild in our home directory. Make sure the path is updated and we'll tell EasyBuild to use Python three because we're installing it with pip three. So if you just copy paste this on pretty, they should give you, ah, it's not happy when I copy paste it from the slide. Let's do it one by one. Like this, do the installation. I think I have it here as well. Yeah, you copy paste it from here. It should be better. You won't have the problem I have on the slides. So that's important. Make sure you have a clean environment. Make sure you're having EasyBuild installation you can use without having a module installed. So this is the easiest way to do that. So it gets installed into .local. This takes a little while. Once the installation is done we have a working EasyBuild version. We'll also configure EasyBuilds. So the usual install software in our home directory or at least define the prefix to use our home directory. The built path to use slash temp we're playing on the login node. So that should work. We'll use the hardcore module naming scheme in this exercise. And here we do something a little bit special. So usually the software directory is in the prefix. The modules directory is in there as well. We're changing that. We want to use the centrally installed software here. So the existing software installations but we're gonna put the module somewhere else. We're gonna put the modules in our home directory. So what we're doing is we're creating a new view on existing installations. We're not gonna reinstall any software. We're just gonna generate module files for existing installations. Okay, so this PIP three is done. You make sure my environment is properly set up so it finds the EB command that looks okay. I'll do this as well but I'll use the version from the slides here. So this I already did. Module purge, module unused I already did. These other ones here are important. There's one that's different. That's this one because our central software stack is somewhere else. So make sure to use the correct one here. This is the only different one. So what I'll do is I'll copy all of these and then take this different one from the slides. And I hope the formatting is not messed up. Be careful. You don't get any extra spaces. If you do a show config, it should look something like this. The important one here is the location of the software that should point to the existing directory in our project space. We want to use hierarchical module naming scheme and we want to park the modules that Easebook will generate in our home directory but in a separate HMNS sub directory. So we don't have any clashes with existing modules. So be a bit careful with the configuration that's quite important in this exercise. All of this I think I mentioned. Okay. So now what we want to do is we want to set up a module hierarchy for HDF5 which is already installed in here. So the installation is already there. We're not going to install the software itself that would take a bit too long because it has to do with a tool chain, GCC, OpenMPI as well. But we already have all the installations. We'll use this easy config file. We'll use dash dash robot. So we also get modules for all the dependencies and we will use module only which tells Easebook skip the entire installation process, only generate the module files. And Easebook will generate the module file as long as the sanity check passes. So it will still run the sanity check. That means it will try to load the module file and check for files and sometimes run a couple of simple commands. And only at the sanity check passes will it actually go ahead and create the module. So if we do this and remember Easebook has been configured to use a hierarchy, it should generate modules for, all right, it's already there, didn't clean this up. There is again. It will generate modules one by one starting with the tool chain and all the dependencies in there. And at the end, it should have generated 41 modules in a module hierarchy and we can check what that looks like. This will take two or three minutes. So I'll continue explaining things a bit and then I'll show it hands on as well. So after a couple of minutes we'll have 41 modules in our HMNS directory here which are organized in a hierarchy. And then when we start inspecting things we will see, when we just do an LS and the modules all sub directory we will see our three levels. The core level which is the top one, the compiler level which has installations which were done with a compiler tool chain and then the MPI level which has basically everything that was done with a compiler plus MPI tool chain. So this is where HDF5 will be located. To start things off, we have to set up our module path and we start from the top of the hierarchy. So not modules all, but the core directory in there. So remember the module files in here will be core slash GCC slash version but the user facing module names are the short ones which don't include the core. So that means the core sub directory has to be part of the module path. If you do this module use and then you check with module avail you should see output like this where in the core directory we have a bunch of modules. All of these are installed with the system tool chain and the one we're really interested in is our compiler tool chain GCC. Everything else is either a dependency for this one or a build dependencies, something that was needed to just get this GCC installation in place. So the nice thing here, everything is very nice and short. Short module names, software slash version that looks very clean but we only have a limited set of software available. There's no sign of HDF5 here yet. There's no sign of an MPI here either. If we check with module avail HDF5, Elmot will say, sorry, I don't know about this module. It doesn't seem to be there. At least it says it's not available for loading straight away. If you do module spider HDF5, Elmot will tell you, okay, I found HDF5 but it's a bit hidden right now. You have to make sure that these two modules are loaded before HDF5 is available for loading. So Elmot is aware of the module hierarchy. It knows about this GCC gateway module and it knows about the open MPI gateway module and it knows that both of those have to be loaded to be able to access HDF5. So then we can start doing that. We load the GCC module, we check module avail again and we see a bunch of additional modules. So these were here before the core modules. We've opened up another level. It looks like two levels, but that's a detail in how we organize tool chains in EasyBuild. Has a GCC level and a GCC core level. They're basically the same level just split across different directories, but that's a detail. So this is a bunch of things installed with this GCC version and we have an open MPI module available as well. Okay, meanwhile has completed the module files. So let's start setting this up. So in here we have the three levels, core, compiler, and MPI. To start things off, I should do a module use on the core directory. So the top of the hierarchy. And then when using module avail, we see nice and short module names that includes everything on the core level. Our gateway module is this one. So we just load this and that both makes GCC available and it also opens up another level in the hierarchy.  The next one we're interested in is our open MPI gateway module, which we can also load and then check module avail again. So we check module avail after loading GCC. We load the next gateway module open MPI, we check again and then we see our HDF-5 available. So now this one depends on both a compiler and an MPI and specifically on these versions. And now we can load this HDF-5 10.7 and then start using it, for example, this Dump 2. So now HDF-5 is fully active. When we check module list, it's all nice and clean, very short and clean module names. So this is a lot more user-friendly. And in addition, only some of the modules are available. There may be other stuff installed in this hierarchy. In this case, there's not, but there could be and they won't be visible up to this point because anything that was installed with a different GCC version is in theory at least not compatible with what is being loaded here. So if you want to use a different compiler or a different MPI, we have to swap over to another part of the module hierarchy. So for the people who want to, you can run an exercise like this themselves with side-by-bundle and basically go to the same steps like I did for HDF-5. Yeah, the question? Can it, may I still ask you to explain a little bit more to explain a little bit more what is the reason to let's say group these modules into the GCC and GCC core directories. You said that this is only detailed but this is always confusing for me. Yeah, I understand, sure. Let's see, so the reason we do this is because these installations in here, most of these are either build dependencies for something or something where performance doesn't matter too much so they're not very performance sensitive at least in terms of how they are compiled. When we put something here, so with the GCC tool chain, this is where we put stuff that's more sensitive in terms of how it is compiled. Because we're only using GCC here, it's a bit difficult to see but whenever we install something with Intel compilers as well, performance sensitive stuff would go in here and maybe compiled with an Intel compiler as well. So Intel compiler or PGI or something else. While these are always available no matter which compiler is used because GCC core is what we call a sub-tool chain and even when using Intel or any other compiler we always control the GCC that sits underneath as well. We don't want to rely on the GCC of the operating system. So when using an Intel tool chain all of these are considered as dependencies. So you probably don't care how Bison is compiled or how package config is compiled. For UCX here we make the assumption that building UCX with the Intel compiler probably doesn't make a lot of sense. You won't see any big performance speed ups while for OpenMPI itself because especially that uses things like AVX2 instructions we have to be a bit more careful and there it does make sense to compile it with not GCC but something else. So in summary, this is mostly relevant when you look at different compiler tool chains for example, Intel. When using Intel all of these will be considered so you don't have to reinstall these with Intel but as soon as you hit this level of the hierarchy these will be reinstalled with an Intel compiler because that's where performance really starts matter. And that's why we have the split between GCC core which is a compiler that we use underneath and then GCC versus Intel versus other stuff that uses GCC core but builds up from there. Does that help? Yes, thank you. Now it's clear. In the other words, may I think of this split as my target models only go to the GCC and the GCC core only consists of some, let's say, anxiaris that are needed but not interesting for me. Yes? Yeah, so a lot of these are indeed probably not interesting even end user as well, that's true. UCX, that's I guess up for discussion that's of course a very relevant library for MPI but the main thing you care about is the MPI itself and UCX is just a supporting library for MPI. So you could even say that these things don't have to be visible to the end user and there's also ways that you can do that especially in Elmode itself then you can say for example, everything that's in here, I wanna hide and the user never has to see this and there's easy ways of doing that. I guess also here, except for maybe the compiler most of the stuff in here is just supporting installations and it's utility modules that you probably don't wanna expose to an end user. Thank you. I sometimes make the comparison, I don't know if it's a good one but like GCC core together with core is like your operating system on top of which you build the other software in easy build. Yeah, so this is indeed sort of the level where typically you would use a system compiler for this in easy build, we don't want to do that because then we're depends which OS you have, which version you have, we like to control the compiler that we use. So the very first thing you typically do is you install a GCC version and then as much as possible, you use that version that you installed with easy build and as soon as performance starts maturing that's when you climb up to more specific tool chains like this, the GCC one here is actually a combination of this GCC core which is used for installing these together with not this benefit but this one. So the benefit built with this GCC version is loaded when you have both of these when you load the GCC module. That's why you see this one loaded here and this one loaded here and this one is a detail this is actually only used to install this one but then never used again.  It's a separate from what you'll have these but of course here it becomes very visible because you see things split up into different directories at least but these two together form the whole compiler level. Right, so now I understand it's quite clever when you like to maintain a lot of different tool chains, I guess. Yeah, it's a way to avoid that we get an explosion of easy config files. You don't want to reinstall all of these with every possible compiler to make a common ground GCC core and then whenever performance matters you take the next step and you go to more specific compilers. Okay, any questions on using a module hierarchy? I suggest if you're interested in this try to do this exercise. It's very similar to HDF5 and should be quite quick but it will give you a good feeling of the advantages of a module hierarchy and maybe also the disadvantages that you have to pass a compiler and an MPI first before you get to the actual software you're probably interested in. So it's a bit of a trade-off whether you want to use this or not. Uchti, you're actually using your article module naming scheme for everything which is installed through spec but not for the software which is installed through other means. Okay, yeah. Yeah, I definitely don't recommend managing this by hand that's a bit insane. So make sure you do it in an automated way whether it's easy build or spec that doesn't matter too much but don't try to manage this by hand because it's very difficult. There's lots of things you have to take into account to get this right. Okay, so there's no more questions I'll jump to the last bit for today which is the GitHub integration. We have an easy build let me start over. So I have a clean environment again and here let's see if I mentioned it. Yeah, so here it's okay to start again with the central software we have like this and load easy build as a module. That's okay. Or you can keep using the one you installed with pip install to play with the module naming scheme that will work as well. At this point you don't really need any software that's already pre-installed for this last part. So let's first look at how to manually contribute something to easy build.  So how does that work? First of all you need a GitHub account so that's hard to avoid and you also want to register your SSH public key into your account to make sure that you can do a Git push into your GitHub account. That's important. You also want to fork the repository you want to contribute to. So in this case here you go to the central repository and use the fork button on the right. And if you do this, in this case it's already forked and it jumps straight to the fork I have prepared here with a dummy account easy build tutorial accounts. So you want to have something like this where you have a fork where it's just forked from the easy builders central repository. That's important because you will push things in here first before you open the pull requests to the central repository. So that's something you always have to do create a GitHub account add your SSH public key to the clone. From that point on when you have a clone of typically your own so here it's done like this. You clone, you see if I already have this here. Yeah, remove that. This has lots of files so it takes a bit of time to clone it and also clean it up. But I will start with cloning the central easy complex repository. You can do it the other way as well but this is how I like doing it where this is the, let's say the main source for this Git repository. Then you jump into the directory and you add your fork as well. And here I just named this my fork. So it's easy to remember. And if you do this yourself make sure you replace this part with the name of your GitHub account. So don't blindly copy paste this because then it won't work. Well, at least that won't work when you do things manually. So you need to be a bit careful there. So cd into the directory. If we do git remote dash v our origin points to the central easy builders repository and for our own fork in this case I will use the eb tutorial account. Hit remote add my fork. Remote dash v you can check that this fork has been added. So say you now want to contribute something. I think I have an example, easy config file here from a previous tutorial session. So this is the eb tutorial dummy software package. If I want to make a pull request for this if you want to do it manually you start by making sure you're on the develop branch making sure it's an updated version of the branch and create your own branch which is named example. In this case, yeah. So we have created a new branch and we've also switched to that branch. Then we need to make sure that we create a directory where the easy config file belongs. We in this case copy the example to that location and we give it the right name. Tutorial 101 GCC 1020 like this. So with good status we'll show us that we're adding a new file created a new directory with some files in it. Then we want to do get adds of this file. So this is now ready to be committed.  And then you push the example branch to your fork. So if I do this that has been pushed it gives me a handy link to create a pull request. If I go here hit a pull tell me a new branch has been pushed named example. So this is if you're a little bit used to it it's not too complicated but it's a lot of steps you have to take. A lot of details you have to get right start from the right branch put things in the right place give them the right name and so on come up with a good commit message push and that's only half the work because then you have to open the pull request. So you click around on GitHub you make sure you're targeting the right location the central easy builders repository and the develop branch come up with a description make sure things are in the right place check this again and so on. So it's a bit of a manual labor and it's annoying. We actually had many people who are interested in contributing but didn't want to go through all this effort or didn't want to take the time to figure it out. Many people were not familiar with Git. So that was a bit annoying. So what we did is we added features to easy build to make this a lot more simple to basically avoid that you have to do all these manual steps and you can do it automatically. So how does this work? First of all, you have to do a little bit of additional configuration and making sure some requirements are in place. You need two additional Python modules. Git Python, so that's what easy build uses to talk to the Git command and key ring. The key ring one is needed to store your GitHub token safely and in this particular case since we're not on a desktop machine we also need this script file additional package to make sure we can store the token encrypted. If we try this, try to install this it will probably go wrong at least when I did yesterday because the people who developed the cryptography Python module have switched to using Rust as a compiler. And if you don't have a Rust compiler installed then you can't install cryptography anymore. They seem to think that's a smart move. You can dance around that by just installing a slightly older version of cryptography as a quick workaround at least. That's what I do here. And then redo this one. Cryptography is needed for I think for key ring as a dependency. So if it's there, at least for now key ring is still happy with a slightly older version of cryptography. So that has all these additional Python packages installed. We also need to make sure you add a GitHub account a GitHub, sorry, an SSH public key to your GitHub account. I have already done that here for this account. You go to settings and then check SSH keys. And I think this one has a direct link, setting slash keys. You will see I added the public key I have here on Putty into this example GitHub account. This is the key I took from here. ID or SH Putty.PUB and one additional thing I've done and this is mentioned in the slides. I've also sim linked this to the default location because that's the one that Kits will use by default. So my RSA private key here is actually assembling to the Putty wall. So that's the easiest way at least to get things set up here. So that's mentioned here in the slides. Add your public key into your GitHub account and you just grab the contents of this file and for the private key, just make a sim link. So it points to the Putty key in the default location. This may actually be a bad idea for other reasons. So you may want to remove the sim link again but at least for this exercise, it's useful. The fork I've already showed. Okay, the next step is making sure we have a GitHub token. So the GitHub token is used sort of as a password to interact with GitHub automatically. So you don't have to click around. EasyBuild can do it for you. This is in your GitHub account and developer settings, personal access tokens. And again here, there's a direct link here, settings slash tokens. You can generate a new token here. I won't do that now because the token is secret and whoever has the token can access this GitHub account. So this upfront, but you can generate a new token and make sure it has the gist and the repo permissions. When you do that, you can copy paste the token and then give it to EasyBuild. So here, we need to make sure EasyBuild is aware of our, the name of our GitHub account. In this case, I will use EB tutorial. You should change this to your own GitHub account, of course, and then you can do install GitHub token. And in this case, it should say that I have the token already. Oh, maybe it doesn't. Okay, then I will need to create one. So let me do that off screen. So I don't share it with you. So of course, the GitHub token step is you only have to do this once per system that you're trying to use. Seems like I was a little bit too aggressive with cleaning things up again when preparing for the tutorial. So make sure it has the repo permissions and the gist permissions. You generate the token, you copy paste it. And then in here, okay, let me move this back here. So now I have an additional token created here. I copy pasted it and I can give it to EasyBuild by using install GitHub token. I copy pasted it, I hit enter. EasyBuild validates the token, it makes sure it's a correct token and it asks me for the password to protect it. So I'll enter a password here and whenever I want to use this token, I'll have to enter the password again. So you have to be a little bit careful with this, of course, because this gives access to your GitHub account. So that's a bit of setup, creating the token, giving it to EasyBuild by installing it. You should only have to do this once for the next times EasyBuild will pick up on it. So that's a little bit of setup you have to do to make sure that everything is okay. We have this check GitHub option, you can run, it's properly configured. It tells, okay, I will use the EB tutorial GitHub account. I will try to use the token, so you'll need to unlock the token with your passwords. The token looks okay, it found the Git command, it found the Git Python command. Now in the background, it's doing a test whether it can create a branch and push it into your GitHub repository fork that should be working. So the push access was validated, that looks okay. It can create gist files with EasyBuild uses for certain things. And then it says all checks have passed and I can do all these things starting right now. So all these GitHub integration features are now enabled and should be fully working. This one here is a detail, this only speeds things up a bit, it doesn't actually add any additional functionality. So with Check GitHub, you can make sure that everything is in place. And for example, if I have some configuration missing, so if my GitHub account is not specified, then Check GitHub will tell me that it's not there. If the user is not there, it will also not find the token and then some of these things will not be supported. So that's a very useful way of checking that things are working. So make sure everything is green here or is okay here at least before you continue. Once that is working, we can start playing. So what can we do now with the GitHub integration? We can create pull requests using the EB commands straight away. So we don't have to run any Git commands. We don't even have to have a local copy of the easy conflicts repository on our system. EasyBuild will do that automatically in the background. And we won't have to click around on GitHub. EasyBuild will also do that for us. There's a lot of information here. The basics of it is to start a new pull request to use EB new PR and EasyBuild will do all the necessary steps for us. I'll skip most the details here. The important one is PR target account. So to which account should it open the pull request? The default is easy builders. But for the purpose of this tutorial, it's probably better to use a different target account so if you wanna do this exercise or this demo yourself, please open pull requests into the EB tutorial account itself. So basically here, my fork EB tutorial. EasyBuild, easy conflicts. You sit without a dash. So when you do this PR target account equals EB tutorial, if you open a pull request with EasyBuild, it will go in here. So make sure you make that configuration change. Otherwise, you'll end up here in the central repository and your pull request will first of all drown in between all the other ones and we'll get the same pull request from multiple people. So that will look confusing as well. So it's good to avoid that. Make sure you send your pull request here. When you're running EB new PR, you can do a dry run, where EasyBuild will show you what it will do, but without actually opening the pull request, you can double check on yourself. That's very useful. And you can also preview a pull request and this will give you an overview of how your easy conflict file is different from the ones that are already merged. Once you have opened the pull request, maybe you want to make additional changes to it later. You can do EB update PR where you give it both the new easy conflict file and the pull request number. So EasyBuild will go to an existing pull request, check out that branch, add your new easy conflict file to it and push the updates into the pull request again. So also that avoids lots of manual work. And to use a pull request, an existing one, you can use the from PR command line option. So EB from PR and the number will tell EasyBuild go look into the pull request, download all the easy conflict files from there and try installing those locally. That's interesting. If you know there's an open pull request or a merged pull request that has something new that you need, or it's also interesting to test pull request, to test contributions. And I'll show that in a bit. And the similar thing works also for easy blocks. You can also pull in updated easy blocks straight from a pull request, even if they're not a part of your easy build installation. What's very useful when pulling easy conflict files from a pull request is that you can also upload a test report. So you will not only try the actual installation on your system, but you can also send back a test report into the pull request and let others know that this worked for you. And I will show this in the demo. That's a whole mouthful, but let's look at what this does. Again, make very sure you're targeting the EB tutorial account rather than the default central account when doing this. So check with show config, that this is properly set up. I will also do the module use of our software stack here and make sure I'm using the build part and temp and the prefix in home. Because I will be doing a test installation as well. So I wanna make sure I do that in the right location. And now let's look at the example. So if you wanna straight up open the pull request, all you need to do is EB new PR in the name of your easy config file. Here, I'll be a little bit more careful. I want to do a dry run first. So remember, this is an easy config file for EB tutorial version 1.1, which should be fully working. This was tested in one of the previous sessions. So this is the easy config file I want to contribute back. I can do a dry run with new PR. Easy build goes into the background and sets up the branch in my fork of my easy config repository. It will make sure that this file is not only copied into the correct place, but also has the correct file name. It will set up a branch for me. It wants to talk to GitHub. So it asks me for unlocking my GitHub token. And then here it says, this is dry run mode. So this is not actually done yet, but it will be targeting. This looks wrong. This should be EB tutorial, but I think it's just a dry run line to me. What has it done? It will say, it will create a branch for me. It figures out a proper title for the pull request. It has copied this example, easy config file to this location with the correct file name, based on the contents of the file. And without dry run, it will actually go and push this into GitHub. So why did it get the target wrong? Ah, I haven't actually set this up. Okay, yeah, make sure the PR target account is correct. So that's why it's useful to do a dry run. Let's do it again. And now it should tell me that the target is indeed EB tutorial. So it doesn't hurt to double check before you open the pull request. It's so easy that you can easily do something unintended as well. Yeah, this looks better. It will open the pull request to the develop branch of the EB tutorial fork. So let's go ahead and do that without the dry run. The output look almost the same, but we should now see a pull request popping up in here once it completes. So easy build is not only doing the necessary get commands to create the branch, to do the commit, to do the push to GitHub, but it's also doing the equivalent of clicking around in GitHub to open the pull request. And it doesn't have my token. Why? Maybe I mistyped my password. Must have mistyped my password, try again. So it needs the token, of course, to automate the interaction with GitHub. So this step here takes a while because it's downloading a fresh clone from GitHub. It does a fresh clone in the temporary directory. This is the part that you can speed up by telling easy build where you already have this clone available on your system. Then it won't take that long because it's downloading quite a bit of stuff in here. Setting up the branch. Now let's make sure I get my password correct. Yeah, that's better. So now it's actually going ahead and opening the pull request. This all looks the same from the dry run, but now we get a link. And if we just refresh here, we will see the pull request opened. Click on the pull request and check what changed. So notice it has guessed a decent pull request title for us. It has put the file in the right place. It has also come up with a pretty decent Git commit message. So it's telling us we're adding easy config files and these are the ones that are being added. So this looks pretty good. This is the first step, opening the pull request. We can now go ahead and actually test this as well. So if you do from PR and the pull request number, which is number three, easy build will check on GitHub, which easy config files are there and it will try to do the installation. Yeah, so it pulls down this easy config file. It puts it in a temporary directory somewhere and then it tries to do the installation. Of course you need to be a little bit careful when doing this, especially when the pull request is still open. You don't wanna do this blindly, but it seems to work, that's good. And we can even do a rebuild and upload test report. So this will do the installation again. It's my GitHub token again. It will do the installation again and then it will also upload the test report into this pull request. So we should see an additional comment popping up here when that installation is complete. Yeah, there it goes. So this adds a new comment for me. It tells me on what type of system this installation was done, what type of CPU. And it gives me a link to this gist file. And in the gist file, it gives me more details how long it took, what was installed, which easy build version, which easy build configuration. So this is a dump of the full configuration, what type of system was used and so on. This is all very useful information for people who are testing each other's contributions. So this explains all of that. Uploading a test report to Trump PR. It's very easy as well. And we get a comment like this and a gist that matches it. So this also concludes this part. I don't think I have anything additional in here. So any questions on this? We've made it as easy as possible to contribute back to easy builds. And I don't think we can do a lot better here. One question, I guess. So it was not entirely clear to me, but it seems like the concept when you use the quick tool here is that you submit one file at a time. So for easy block, you give it an easy block and that's like one file, one commit. But I think for easy configs, it looked like it took everything that you have changed in your easy config directory, right? But I suppose the best practice here is just to commit one easy config at a time, right? Well, it takes, when I created the pull request, I gave it the name of the easy config file. So it only takes this one, not the full directory, just this particular easy config file. You can give it multiple easy config files and the same EB new PR command. And then it will pack everything together in a single pull request. Two, three, four, five easy config files is okay. It can actually do this as well. If you combine new PR with robots, let's see, can I, if I do missing, yeah, this is a bad example, I guess, because there's only one thing missing. Actually, there's nothing missing anymore because of the test installation. But when you combine it with robots, easy build will resolve the dependencies. It will check which easy config files are not available yet. And the repository that you're sending the pull request to and all the easy config files that are needed, all the missing bits, it will combine them together in a single pull request. So if this one needs two dependencies, and these dependencies are not an easy build yet, it will add the easy config files for these dependencies as well. And you basically get, yeah, everything that is needed to install the software. So you don't want to spread things over multiple pull requests, one easy config file per pull request because that would be way too many pull requests. And also the CI would fail for everything that depends on something else. So typically you want to pack things together a little bit at least. Okay, I see. And we already get 2000 pull requests a year for easy config, so we don't wanna get more if that's not really needed. Anyway, we probably won't be using the common tool chain, so it doesn't make much sense to contribute to the central repository. But I guess with some careful configuration, it may be useful either to have people from associated teams contributing to our easy config repository, or if in case some sites that all have craze would collaborate on setting up a common repository for the craze tool chains to contribute to that one. Yeah, of course. So it just does the pull request to the central repository by default, but you can reconfigure it to. I've noticed there was an environment variable to tell to which, I mean, you have actually used it to contribute to a tutorial instead. Yeah, indeed. So you can tweak which accounted targets for the pull request. You can change the branch if you only have a main or a master branch in your repository. You can even target an entirely different repository, so something that's not called easy build easy configs, but you do need the same structure, the easy build directory, the easy config directory in there. As long as you have that, that will also work. So there's nothing that hard codes you to do the central contribution. You can manage your own repository through this as well, and pull in pull requests with from PR, also from other repositories that works fine. So yeah, indeed for Cray, we do have some tool chains in the central repository, but not up to date ones. And CSCS also manages their own repository. I'm not sure if they use this. I think most people there are familiar enough with Git, that it's not a problem for them. But for people- I'm pretty sure they're not using that. Yeah, but you can. Yeah, there's nothing that- They have one repository which contains everything, not just easy configs, but also the configuration and so on. Yeah, yeah. You can do that too, as long as you use the same easy build easy config structure, you can still use new PR and have other stuff on the side, easy build will not touch the other things, of course. Okay, then if there's no more questions, we can either wrap up here, or if somebody else has questions on anything else related to easy build, I'm happy to take them as well. And if not, we can wrap up here.