You're not any more working with this kind of an interactive, this like immediate interactive workflow. So you need to adapt your workflow so that you can run it and monitor it in this non-interactive way, and usually there's few steps in this process like whenever you submit a job you need to do a few monitoring steps. Firstly, when the job is submitted, you usually need to use the sperm queue to check the position of the job in the queue and when it's expected to start. So sometimes you might need to queue, like for these small jobs that we have here, they're basically immediately done. The queue allocates resources for them, but sometimes for bigger jobs, for longer jobs, the queuing time might be some minutes, some hours depending on the job, and you need to usually check once you're submitted that it's in the queue. Secondly, once the job is allocated the resources and it starts, you need to have some monitoring done usually in the program itself so that you can determine what the program is doing, and this is really important, and it's not something that we can do for you because you need to design your program in a way that you can monitor it even if you're running it non-interactive. Thirdly, once the job is finished, you need to usually monitor the performance and the resource users of the job. So we already had our first stint with the monitoring the job queue state after it has been submitted, so if we look a bit below in the documentation, with the sperm queue you can check what is happening in the queue. So here's an example output of this, and below there's an explanation of all the fields. So basically, when you submit the job and it's in the queue, it gets the job ID and then you get a part, like it's running or queuing in some partition. And then, so for example like what Richard is doing here, he's submitting this sleep script that he wrote, that basically just like sleeps there, and you can see that it's currently running in partition that CSL. Well if we go from the left actually, it gets a job ID, some job ID that is allocated when or set to submit the job, like when you see the submitted batch job, you get a job ID. Then it's submitted to the partitions one or more that you have specified in the partition or your queue automatically sets for you. You get the job has a name, there's some time, like the runtime is zero until it actually starts running, and then you have this start time when the job actually started. If the job hasn't started, this is an estimate of when the job will start. So after a while, this room, like if it can't start it automatically, it will calculate this estimate. But because there's lots of like jobs rolling and stuff like moving and new jobs coming and stuff like that, the estimate might vary, but you will get some sort of idea when the job will start. After this, you will see the state of the job. So in this case the job is already running, so you can see that it's running. Sometimes it's pending, and if it's pending usually, well you get the reason why it's pending. Then the last column is basically where is it running, the list of nodes where it's running and the reason why it's not running if it's not running. So let's say it's queuing, it might say that the priority isn't big enough or something. Do you want me to submit one that uses a lot of resources? Yeah, put something like... So maybe it's best if not... Well I guess people can do this, just remember to cancel it. Yeah. How much memory should I ask for? Put something like... It's better to put something like... Well yeah, 200 gigabytes and let's say four weeks or something. A week is longer than we can, right? Let's say three days. Okay, so this syntax here means three days and then zero hours. Minutes and seconds there, so it's easier to see. So basically three dash and then the time, so it means three days. Okay, should we do it? Yeah. This is quite a big job. Wasn't this interesting, unspecified error? Do you have a... Do you have a... 200... Is there some sort of a... Do you have a like a non-breaking space or something? In the commander in the... Yeah, in the command. No, I mean in the command. Oh, you have partition debug there, in the... In the script. Okay, so I'll comment this out and demonstrate the two things. Yeah, so sometimes like you can see here the output doesn't necessarily... Isn't the most verbose because like the slurm doesn't necessarily like know what you're trying to accomplish and it can't give you necessarily that good one output. Try again. Okay, so it's submitted. Yeah. Q, which I need to make wider. We see it's pending because of priority. Yeah, so basically for these bigger jobs they have like a higher cost and the cost is basically the amount of CPUs or amount of equivalent like memory of one CPU times the amount like runtime. So it's a bit complicated formulas. There's other terms there as well, but basically everybody's getting a priority and bigger jobs have a bigger... Like bigger jobs have a lower priority because they will use more resources. So basically you usually want to set up your jobs to be close to the requirements that you need. So basically don't waste resources because otherwise your priority will not be as high. But yeah, okay, so this is how you monitor jobs when they are waiting there. But sometimes you might want to like watch if the job is done. Is it starting now or is it not? So for that you can use this watch queue, the slurm watch queue. And you can see you get this window where it's like every 15 seconds it's updating the window. And is it done yet? If you feel like you're the kind of person like basically on a road trip that you like asking constantly like is it... Are we there yet? Basically this is the way to do it. So basically if you want something... It's most commonly used this kind of a tool when you have a script that you submit, you get a failure immediately. You do some fix, you submit it again and then you watch the queue and see whether the job fails this time. And then if it like starts running or if it fails. So basically this is something that you usually do when you have this kind of like a development process that you watch. Like does it run correctly this time when I submit it. But usually it's not good idea to like keep this running constantly on the background or anything like that because it's meant for more interactive users. And you can quit this with Ctrl C this watching. Let's cancel the big job. Let's see it's this. Okay and now everything's done. The sleep timed out after five minutes. So should we go on to the next part of this? So we should point out the slurm command is something which someone wrote and we've added to our cluster and since last night also the other clusters in Finland. But it's not a standard part of slurm. But it's basically the other clusters around the other. The FCCR collaboration. The next part is like the tricky part. So this is the kind of thing that we can't give you one good answer. We can give you hints towards the correct solution. So basically how do you monitor your job when the job is running and this is more on you because like you need to write your code in such a way that it produces you output when it's running and you can from that output determine what the code is doing. But we can give you hints of what kind of stuff you usually would want to do. And what is the recommended way or what is the common way is that you have some sort of like a print statement usually in your code that basically it prints the state of what it's doing. And this print statement is usually outputted to the slurm output file where it's read by the user. So you read basically the standard output of the code and then you read it from the slurm output file and your code produces some debug output there. But it's usually a good idea to differentiate between different types of output and in your mind and in your workflow to differentiate between different types of output. Because there's a few of these that you might be well it's good to have an idea in your mind like how do you monitor your job. The first kind of output that you might want to use or probably needs to use is this kind of monitoring output. So this is like simple output that tells you the state of the program. So let's say that it's currently loading data, it's doing analysis, it's doing plots. Like these kind of like print statements at crucial points of the program that you can then determine that. Or in your SBAT script like echo statements there that you know that OK it's now doing this part. So when you look at the output and you see let's say the program crashed at plotting you know that OK crashed at plotting you don't just like you just don't see any output. And you just think that the program failed but you don't know where to find the problem. Other kinds of output might be that if you have like a iteration loop or something like you do like a Monte Carlo simulation and you do iterations. Or if you have a in machine learning you often have this training epochs or something like that. And like you have prints every epoch or you have output every now and then like in physics code you usually have every x time steps print out some values. And usually these tell like what the program is doing or what the state of the program is. So let's say if the solution let's say converges like if you're trying to let's say minimize energy of like a molecule or something. Does the energy go down in time. Like when you when you when you try to minimize the energy do you get lower and lower values of energy for each iteration. Or does it like grow exponentially or something and the program is not working correctly. So this kind of output is something that is human readable and usually like printed out into this kind of output. So it's something that you can then like immediately look and see that OK my program crashed about here like at this point of the program my program had a problem or something like. And usually usually you can have some timing information there as well. But that this monitoring output is different to debugging output and debugging output is usually something that is more verbose and writes the internal state of the system. So let's say it like it prints every time step or something like that and you get like huge watts of code or huge watts of output. And something that you cannot read yourself like basically you have to like grip or like use some find commands to go through empires to see what the program is doing. But let's say you want to print like print what what what is the state of the system like every every time step or every iteration. And it's it's like huge amounts of information. These kinds of info like output is usually like meant for development phase of the program. And it's not usually like people don't usually do this kind of debugging output when they're running like production code. Let's say like production meaning that it's actually like you're trying to solve some problem. Once when you're developing the code you usually use this kind of debugging out to see like if you can't find where in the program you have a bug or something like that. You might increase the word of the city or increase more animal print statements or something like that so that you can like in point where the problem is. But people don't usually leave this kind of output when when the when they're actually running the program. The reason for this is that first of all it slows down the program because the program has to use for much more much more time printing the stuff than actually doing the actual calculations. And the second one is that it makes you hard it makes harder to monitor the output because it's you get too much information.  But if you want to run it in normal mode it doesn't print that much output and you can usually like put a switch or something like that. You know program so that it works in these different modes so either debugging mode or normal mode. Like a dash D or something like that. Yeah people usually and many programs usually have like verbose or debug mode and like different kinds of modes that determine like the verbosity level of how much like output should you get. Because like normally you don't need that that much output is too much output for the like it doesn't help you to. Okay so it's done. Yeah okay. The other kind of output you might want to have is like this checkpoints. So every now and then in your program you might want to save like if you have a really long program that takes hours to run you might want to have these checkpoints so that basically you save the state of the program so that you can continue where you left off if you have any problems or if you run out of time or something. Like you don't want to have like a let's say three day simulation and then you realize that okay it's going to run out of time and it's going to waste all of three days of simulation because like you haven't saved any of the intermediate results and that's usually like what you usually want to have some sort of check. Checkpointing. Usually at critical points like when you switch between different programs like program stages or at certain intervals and finally of course you have the output that you actually need and the bigger the runtime there it's usually good idea to like put something that you don't necessarily think that you need into the output as well so for example like in many cases like let's say.  And then you get the mean value and you look at the results and you think about oh I should have probably gotten like standard deviation as well. And now you haven't calculated the standard deviation and you need to do the calculation again. So it's usually like good idea to think before you run the program of what kind of quantities you want to calculate during the program runtime so that then you can get the best output. So it's good idea to separate all of these different kinds of output in your mind so that you know what you're like trying to accomplish. So let's see so what comes next. Is there any of these we can demonstrate the monitoring outputs or so on. Did one of the serial jobs demonstrate this in the by writing the date every few seconds. Yeah let's let's run this serial example for example that's a good example of this thing. So it's not a good example of an actual like production code but it demonstrates this procedure. So in the previous serial examples the example three we had this like question of creating a slurm script that runs the following program. The following program is basically four loop 30 times then print the date and sleep for 10 seconds. Okay. And I'll paste this into there. Do we need to use default parameters. Yeah well we can we can like put the put like our default so let's put some time and some memory because that's usually what you want to do is set some values. This is five minutes and the trident default is 500 megabytes of memory. Okay. Okay now we submit this. Let's add even more debug at the start because why not save with control X Y enter. Okay should I submit it. Do we want to save the output to a certain file name. Hmm. Yeah maybe. What's the substitution for the name of the job. Hmm. I think it's present. Probably present in. Yeah. No sorry. A present X. X lowercase. Most most yeah most logical. Okay. There's a good question. There's a good question hack in the I thought it's important to run the commands with S run. S run. Let's yeah let's use S runs there. We can we can then demonstrate also the like if we put S run to the date. We can demonstrate then. And not sleep. Okay. Should we submit it. And if we list. It's still in the queue. Hmm. Okay so it's pending. Are there like lots of three dated jobs. You know what you can put the watch watch your watch her now. Like we can demonstrate like the whole she bang. Yeah. So while we're waiting for the jobs to start. So so there's a couple of good questions there. So is there equivalent of ID debugging mode in bad scripts. No like the idea behind the bad script is that it's non interactive. Like that is the that is the thing that it's meant for low running stuff non interactively. And then you debug based on the output that it produces. Of course like in some cases if you're doing active development you'd rather use let's say S interactive. Or S run to run it on the queue without like writing the bad script. But you can also develop using this bad scripts just submit it there and see what the output is. But but yeah. So the basic idea is to like like you can do interactive stuff in the classroom.  Yeah. Should we look at the output now so we see it's running. So control C to exit. Yes. So we can let's say cut the file. Or even use this tail dash. So this is like follow the file. First let's cut. So we see it's been printing. Each thing every about 10 seconds. So actually so tail dot F should continually update and every time a new line comes. It gets printed. And actually it just did. Might might be different like whenever the I don't know what's the polling frequency. Okay. Yeah actually last night when we were testing this it didn't work somehow. But we see now it does. Let's can you run slurm history now like five minutes. Now that it's running. Let's see if we get constant updates there. I'm not completely sure how it works. So the usual control C will exit the tail dot dash F command. So we see it's not in history yet. So let's wait for it to. Or maybe ten was it five minutes or something. Or maybe we can let it run on the background for a while. But yeah so there's another good question that somebody asked that it seems is it important to run like with the S run command and like the S run command when you run it like when you're running on the login note and you're interactively running stuff with the S run it basically it puts stuff into the queue. But when you are in the S batch script and you have an S run it doesn't like do another reservation of resources it will use the already existing resources. So basically you only do one reservation but there you will get like this job steps.  If you have these S run statements in your script you will get like this output that you can then interpret like this job step at a time. And maybe we could cancel the job and see from history to check the output of it. Yeah so as cancel this job maybe I could move this up some Slurm history five minutes. Okay there we see some stuff. Do you want to show the monitor well okay. Yeah well basically Slurm history will produce this kind of output of like here we see like the individual S run date. You don't want to put S run to every command like CD folder or something like you only want to put S run in like for actual like commands that are important for your program. And because then Slurm will like keep track of those programs and it will like monitor those and you will get like these individual step information so you shouldn't have like a job with like I don't know like thousand job steps or something. But yeah so in this output we can see that there's a job ID and then there's job ID there's this underlying like individual S run steps that are done in there. Then we see that there's a name for the job or the command that is being run. We see start for each individual task and then we see the memory requirement for the whole job. So here the memory requirement is like in this case is 500 mn so it means that 500 megabytes per node it can also be megabyte per core but we'll talk about that more in the parallel tutorials. And then after that we see like the maximum memory usage that Slurm has noticed. And this is something that is pulled ever so often so if you have a job that suddenly makes a huge memory requirement it necessarily doesn't capture that but it should give you like a ballpark of how much memory your job is used. So you can use that to inform your decision on how much memory your job is. Then we will have a total CPU time that tells how much CPU time was used for this and this basically like how much calculation time was used. And after that we will have this wall time and these are two different measurements so if you have four CPUs running for let's say 10 seconds you get 40 seconds of CPU time but 10 seconds of actual real time so wall time is like the time of the clock basically the time of the clock that is the wall time basically clock on the wall I think it's like I don't know why is it called wall time but it's basically the time that you experience but CPU time is like that multiplied by the amount of seconds the CPUs were utilized and if your jobs don't utilize the CPU the CPU time can be like zero and if the CPU is fully utilized it's the wall time multiplied by the number of CPUs. Like here we see CPU time was wait, don't ever mind these are actually in different units. Yeah, yeah because the CPU time is measured at higher frequency like higher rate than the actual wall time but we'll look at these when we do more like the exercises of this chapter because there we will monitor like the actual CPU time then after that you will get like the requirements for the job like how many tasks, how many CPUs how many nodes you requested these in the parallel section we'll talk about how you can request multiple nodes multiple tasks, multiple CPUs but for now like they're all one but later on they are something and then you have like exit code so you can see that most of the exit codes there are zero so exit code means that it's finished like it finished fine like zero is the typical exit code that it completed fine but for example the second step at the top is 15 the exit code is 15 because it didn't exit fine because we cancelled the job and you can see on the right side that some of these tasks are completed and some of them are cancelled so basically if you have a job that runs multiple individual tasks you can see from this output already like okay if it failed at certain point you can see that okay this step was cancelled this step was failed and these steps were completed correctly so that you can use this already to monitor like if you have this job that consists of individual like s-run programs or programs that you run with s-run then you can use these to monitor how did your program perform but the real kicker now if you want to show the documentation again of here or? yeah the monitoring documentation so scrolling down so here we had the like you can look at there for the explanation of the slurm history again if you don't want to remember all of them but here so you can also use this SF command to get like more like performance readings basically of your individual individual jobs or your individual job steps so you can use this for both and it will give you like you can see here for this example job that I run previously you get information on how the job finished, how many jobs were utilized what was the CPU efficiency what was the memory efficiency so how much memory was utilized and so forth let's try running these for the job you just run okay so SF and the job ID let's see so the job ID was this so job ID we see it was cancelled one core used second and efficiency 0.44% yeah so it basically didn't use any of the CPU and any of the memory but let's look at the individual job steps so if you put like SF and the job ID and dot there then put yeah and dot and let's say one for example one for example okay that will tell the efficiency of single job steps like you can check various different job steps with this so you can check what is the efficiency for single steps so let's say your program consists of multiple parts you have pre-processing code and then you have let's say analysis code and after that you have some other simulation and you have like let's say you have a iterative procedure one thing then you do another thing and then you do another thing and you do this in a follow up or something you can use this SF to measure the individual parts how effective they were like what was the CPU efficiency and what was the RAM efficiency and you can also use these values to then like to decide what kind of memory requirements you should have and what kind of CPU requirements you should have so we will talk about this more in the parallel section but this you can also use to check whether your code was actually used as in the multiple CPUs you asked for okay so yeah but yeah what's next can you do an example with something a bit less trivial that uses some memory or should we go on if we could like we could go to the exercises and then show the exercise solutions once we once people have tried their hands with those and GPU monitoring I guess we will get to later yeah we'll yeah and this is yeah this is a bit all specific but we'll talk about that more in the CPU section but for now let's focus on the like slurm print statements like what the print statements are slurm history and SF those are like the common tools that you use when you do monitoring so here like we previously run in the interactive like interactive tutorials we run this Pi.Pi that tries to well that uses Monte Carlo methods to calculate the actually maybe we didn't run this do you want to run this once this Monte Carlo so in the HPC examples slurm folder this Pi.Pi you can run it so yeah so I'm cloning this repository again yeah double clone there if you already cloned this you can just use to already existing you can probably your ssh config yeah so I've configured git even though it says HTTPS it will try to use ssh anyway but for you it should just clone it okay so I've got it so instead of copying the files around I'll use a rel to path the run so yeah this is like my usual way of approaching things I have the code first does it run at all Python 3 like this and I can well just try it and I see okay it needs an argument so now I work more let's say 100 iterations and then I run it and we see it ran and then let's try increasing it some so all this time I'm running it on the login node so is this okay so if someone was very pedantic they would say no you're not allowed to run jobs on login nodes la la la but I mean in this case we're making sure that the amount of resources we're using are so small it really can't possibly affect anyone else so it's a useful thing to do here how small of resources can you use before it affects someone well that depends on what your cluster admins say but I think in practice if you're using a few CPUs for a few minutes then that's no problem with our login node it has what 20 cores or something like that so okay so what do I do do I continue further or yeah well you already demonstrated how you can run the program so just as a under test just create a slurmscript that runs this for 10 million iterations it should take something like like a minute to run something like that and you can also test smaller number yeah so I'm gonna copy this copy the command I ran before I opened the file well I should use nano and then I will paste it in here and add in our other framework let's see so I'm using the same output file name with the x in here okay so we want 1 million does 1M work in this pie script I guess not okay so there we go should it be more 10 million put 1 million for this if you want to demo it here but like we could go to sizes after you have submitted it and people can try this themselves so basically to create this serial job run the monitoring examples there that we have okay so exit should I submit it yeah run it once and let's let people try themselves then have an exercise session okay so it's running so basically if you let's go to the exercises now so it's all nice to talk about it but it's good to run to at least the first exercise this is really really handy exercise so basically should we ask for it well let's let people do that we can look at the answers after people have tried themselves so basically let's have an exercise session of like 10 minutes and then should we have a break after that oh yeah I guess it's time like a 5 minute break we can then go through the modules so then we can have another break after that I think in the past we found that breaks less than 10 minutes are almost not breaks okay so in this exercise people should try to do well maybe try both yeah yeah they aren't like they aren't that dissimilar do people know how to use this one with multiple yeah it's underneath there it's not about like just run that like if when you're running the exercise too just run that command and then just focus on the monitoring tools don't focus on the cps but task we'll focus on those later on but for now those are like hints at what's coming in the future but for now let's just focus on on how you can read this output from this sf command and slum district commands yeah and I think this program at least when I tried to make it it should work in both python and python 2 and python 3 so hopefully whatever you have available it'll work okay so good luck not that you'll need it but anyway okay see you in 18 minutes and if you want more time let us know okay bye for now hello and welcome back so let's take a look so in the poll we see I did the exercise maybe should we add neither as an option just so we can see who is watching but couldn't do so Simo what's the plan for the rest of the time we've got a very short amount of time and a whole lot to cover yeah so we are going to like now that we've gotten some familiarity with the serial jobs like now everything is basically like in our heads hopefully in this kind of a way where we can just like do the same thing but we add a few spices there so basically the spices in our case are we're going to do like these aggregate jobs that are basically like do the same thing over and over again and then so this is like for the embaggasingly parallel jobs then we are going to be doing these TPO jobs which is basically like do the stuff but add a TPU in the requirements so we are going to do these parallel jobs which is basically like do the same stuff but get more CPUs involved there are lots of things involved here but basically the commands are pretty much the same yeah like after what we've done so far it's really not that much extra work to the rest so Richard if we want to like quickly go through the exercise let's quickly go through the exercise one of monitoring yeah let's say 1a okay so 1a let's just go through the whole thing it's not that hard so let's create the script so we started it here so we make a slurm yeah so let's put well we did this already we looked at it clear let's look at SF what was the job ID well you can use history slurm history to check it oh you can check it then so basically once you have written this slum script for this you can check put like open the script again let's do the PE step so basically let's add different iterations different amount of iterations so put one with like the 10 million and well from 100 to 10 million and not the last one 100 million won it will take hours okay there we go so basically one question so one question while we're waiting for it to run this 50% what do you make of that we will look at it once the job has finished here is that job like the utilization like the job wasn't long enough that we could get good statistics like it didn't run long enough that we could get 100% utilization for long utilization or utilization for a long time so it run too fast so if we look at now like the exercise 1b so we run this script on this we run this with various different number of iterations so like if you print out the script that you submitted or show the script that you submitted we saw that the s run first one was 10,000 then we went to 100,000 a million and 10 millions so we see in the output there in the history like we see that one of them took are we really running 10 million are we running only million in the last one it's probably not yet finished actually that's probably the reason if you run the history 100 million 10 to the 8 okay well never mind if we now look at with SF the efficiency for the whole job let's look at first if we just put the SF and the whole job ID let me grab the ID okay we see that the utilization was 92% so this is looking better but let's look at the individual jobs because some of them are really small so they don't utilize the CPU almost at all so most likely won't utilize so if we look at the first step well the CPU efficiency is zero basically because it ran so fast that it didn't have a register in the CPU if we look at now the step one well still zero and two we see that it was 7% utilization so it actually did something before it finished up and then the step three that was major majority of the run time that had quite a good efficiency so I guess the moral of the story here is you want your jobs and processes and parts of the job to use to be as long as possible to get the best efficiency yeah and also that you can just do this kind of monitoring I would say that that is the best well that is the thing you want to do let's do the exercise too we will talk about this parallel processing in the future in an hour or so but basically this program has been designed in a way that it can utilize multiple processors so let's just run this program like this so we're running it from the command line so we are basically just using srun to run it interactively basically yeah so and it's 100 million now yeah that's good yeah I'm giving it the relative path to the file so now it's running the 100 million you can see this program has been designed to do two threads that run 50 million iterations and it will take like 20 seconds we can wait but basically we'll talk about this parallelization later on but what you usually when you're going to be running this parallel stuff you need to look at the CPU efficiency that is the most important thing if you are going to be running stuff like this you need to look at the SF and see whether the program actually utilized the CPU it's also for the serial jobs like if you have a serial I mean one CPU job you always want to look at the SF and see what was the efficiency so let's look at like we had this job and we can look at SF okay so we see 2 cores per node we can see there that the CPU utilized was 1 minute and 37 seconds but the wall clock time below there that's 50 seconds so it took 50 seconds to run the job but it utilized over a minute of time so how can this be well the reason is that the CPU time is for 2 CPUs so basically you have 2 CPUs running like you run 2 minutes in a minute basically but 2 minutes of CPU time in 1 minute and you can see there that the CPU efficiency is 97% so this is calibrated so that the 97% means that you get it was 97% of 2 CPUs basically so it's not going over 100% like if you're running if you're running with multiple CPUs it's not going to like the maximum will always be 100% so run the same thing but let's say take one of these run the CPUs but ask one and then threads one or actually yeah just yeah put out threads threads one there and then maybe run less than 100 million yeah maybe 1 million so now we have reserved 2 CPUs and we have we run it with only one CPU so this is quite common in like if your code isn't like if there's a mismatch between the amount of the request CPUs and the amount of the CPUs you run and you can see here that the CPU efficiency is only 50% so basically if you see some sort of like this like 25 like if you request let's say 4 CPUs and you only use one you will often see that CPU efficiency is close to 25% or if you are using request into 2 CPUs you might get like 50% so this using this SF you can often see what is the what the program is doing in the background and we will talk about this more in the parallel computing tutorial but for now like remember that SF can be used for monitoring various kinds of things in your job so should we what do we do next maybe we should continue on the parallel tutorial maybe we should just go there not that we are halfway there already so parallel it is yeah we will do it live we will change it a bit so