Um, Thomas Scripps, okay, let me go from introduce myself, maybe it's the first time here, so I'm Thomas I recently started as a research software engineer at Alto in Richard's group. And yeah, I'm going to talk a bit about Scripps. So back to you. What was your question? Okay. Yeah. Okay. So At what point do like what's the point of Scripps instead of Jupiter? Why would you break out of Jupiter and do something else? Well, there are two things for me. One thing is that scripts can be used outside of the Jupiter environment. So if you have anything calling or that needs to call your scripts, any bigger program, then you need to get out of Jupiter. That's the one thing. And the other thing is scripts are more easily controllable from the outside. Yeah, like, like a common problem, people will say, okay, I have this Jupiter notebook, I need to run it on a GPU, I need to run this 100 times and they want to use our cluster because that's what the point is. But when it's in the Jupiter notebook, that's really hard. And I believe Thomas will walk us through. How to how to money how to modify that. Yeah. The other reason said is Scripps tend to be more flexible. If you have one Jupiter notebook, you have one workflow. If you want to modify things, either either you modify the notebook.  So if you change parameters, or you have to make a copy or you have to make a large parameter matrix in your notebook, from which to select and even the selection can go wrong at some point. And you never really know in what order the cells were executed and so on. So things can go wrong in a lot of ways in a Jupiter notebook, which they can't really in a script because a script is executed from top to bottom and there are no I first execute this cell and that cell. And that wrong order of execution can actually lead to very bad problems. Once you once you have some really nice results and you can't reproduce them because you don't know in which order they were the cells were executed. Yeah. So, yeah, I think that along with running it on a cluster where you have to run it multiple times is I think the two main reasons to go away from Jupiter scripts and go to actual Python strips. And going there is not really difficult. Okay. Should we do the example. Yeah. Well, if you have a trip to lap open there, let me get started. There are two options how you can essentially export or how you can, or there are probably a lot more but we'll mention two of them. So, Thomas, when you start programming like this, do you start in. Do you start in Jupiter or do you start as a script directly personally. I'm not really using Jupiter. I think Jupiter is a good tool to communicate your, your scripts a good tool to present your scripts because you can have very nicely formative documentation along with the script in the form of market markdown cells. So for publications, I think Jupiter scripts are something quite useful. If it's just one workflow. But if you want to run a program, I think they are inconvenient. Yeah. So, okay, yeah, yeah, I'm the same way like I usually will start as scripts and programs directly rather than. Yeah. So in the lesson, where do I start now? Yeah, safe. Just in that place, safe as a Python script. What's my starting script. In that instance, for this, you don't, it doesn't really matter what your starting script is. In the exercise, we will provide a starting script. You can already download that if you want. Okay. Little bit further down. Yeah. And just use that. So what should I demo. Yeah, let's essentially demo the exercise so that we can then give a little bit less time for the people. Yeah. Okay. So I'm downloading this. I think that was observations. Notebook. So let's see. I need to download it and save it to your Jupiter. And then I will move it. Probably from, yeah, from downloads to wherever your Jupiter lab is located. Or if that is too inconvenient, you can also just upload it within Jupiter. It's the alternative. Maybe I should demonstrate doing that. Yeah. So let's say I'm running on a different server, not on my same computer. I can go to new. The upload I can upload. Exactly. And then navigate to. I select it from downloads. And then it's there in there. Yeah. Okay. So I select it. And then you can upload it. Except I've already moved it. So, yeah, here it is. Okay. So you can either right click on that. In the. No, right. I think right click on that also works. No, it doesn't. Okay. Then file. Export notebook as. Yeah. And then executable script. Okay. That would then again give you a download option. So you would have to copy it again. There is an, there is in my opinion, more convenient way. Cause that's the way how it will stay in your Jupiter. Directory. And that is if you go to. Okay. So now I need to save it and do the same thing where I. Yeah. Move it to the right place. This is a bit. Complex isn't it. Yeah. The other option is if you go to file. Okay. Open a new terminal. New terminal. New. New. Terminal. Okay. You can then use the command that's in the, in the, in the lecture. Slides. Or in the lecture course material. That Jupiter and be convert. Converts your script over to a. Python script. Okay. And if I list this, I see. Weather observations that play. Yeah. And if you. Yeah. Okay. And if you go to your explorer and essentially refresh, then it will also show that file. I didn't need to refresh, but. Yeah. Okay. It refreshes itself, but depending on how fast you are. So this is the contents of the file. Okay. So basically what we've just done is we've. Downloaded a notebook. Put it in our Jupiter lab area. And then converted the contents of that notebook into. Well. A non notebook plain text file. Here. Yeah. And. Now it's available as. Just the same thing that we would have programmed ourselves. So we could have copied and pasted this from the notebook into here or something like that. Yeah. The only problem is that if you try to copy from a notebook, you're not really getting multiple cells at the same time. So if you have it over multiple cells, you have to copy each individual cell. Now. I can simply try to run it. I can run it to your terminal that you opened. Okay. That's here. Yep. You could simply run Python or Python three, depending on your. Set up. Python. I guess other observations. So since I'm in my anaconda environment, Python is Python three here. And if I run this it. I don't give any output, but you should get a file. And. Okay. And if I list the directory. You should get a. Weather PNG. Yeah. Okay. Okay. Let's give everyone. Five minutes. To essentially repeat this. So that we are all on the same. Yeah. So. Does the LS command work in windows? Probably not. The. Okay. Yeah, I guess we can wait here. Let's see. Some error with converting the script. Can you paste there? Yeah, it's sort of surprising how difficult it is to take one notebook and convert it to a. Oh, yeah. Windows. In general, if you download any potentially executable code, Windows will say, oh, this is harmful. This is harmful. I'm not downloading that. It makes some sense because it doesn't know what the code does. But in this instance, it's not harmful. Can we. Can we paste the contents of the file in HackMD so people can paste it directly into a. Text file. Um, yeah. Give me. Well, will you do that? I can do that. Okay. So, and should we demonstrate the other option then. So let's say you can't download it. Here's another option. We open the browser. And we make. Actually, no, we can go to file new. I never actually do that, even though it would make sense. Like a file new new text file. And then I'll rename it. To weather. Weather observations to dot pie. Removing the name dot txt. Click rename. And now I go to HackMD and copy all of the text. And then come back here and control V to paste. And there we go. And now. How do you say file, say Python file. Okay. Yeah. So the next thing that is very convenient with Python scripts or Python code instead of Jupiter notebooks is that you can create a function can export functions to a different. To a different. File so that you can can use them from multiple different. Notebooks or not multiple different, different. Notebooks off or scripts. Okay. And that's what we are going to do next. Because we have a rather short notebook so they are we had a rather short notebook and rather short script. But even in this, we do some pre processing. And we do some plotting, which are two things that very often is very similar for a lot of different for a lot of different tasks. By essentially. Exporting the pre processing of the data. You can reuse that for other data for other data sets or very similar data sets. So in our case, this is essentially the. Selection or the conversion of the local time column to a data time format. That is on that is computer better understandable to the computer. And the selection from a start to an end date. Okay. So we're splitting. We're splitting the file into multiple parts. Okay. When would you do this in. In your daily work. Probably in my daily work, I would rarely do that because I would essentially start with having the things split it. But this is something if you have started your. If you started from a. Jupiter notebook. And you do some pre processing things and then think, well, I would like to use a second data set that has very similar data pre processing steps. I would at that point say, okay, I'll take my pre processing steps, export them into a Python script that I can load and then just have that line. But in both in both of the scripts. Because I don't have to duplicate the code. So if I did a mistake in my pre processing, I again only have to fix that mistake once in my pre processing script. But this isn't just about pre processing. It's about any code that you want to be usable from multiple other places. That's for any code that you repeatedly use actually. So if you have steps that are really used very often in very many different. Different workflows. It definitely makes sense to export that code into a specific function. Because if you ever notice any mistake or anything you did there, you only have to modify that in one file, not in however many projects you have where you use that copy code. Okay. So to do this, do we make another Python file? A new file. So I do that from file new again. Yeah. New text file. Okay. Mine actually offers Python file. Interesting. That's, yeah. Different versions. So I rename it to weather functions.ply. Let's see. Okay. I'm rearranging my tabs at what weather observations and weather functions. So maybe I'll just copy from here. This is the pre processing part. Exactly. I copy. And I paste into weather functions. Okay. Yeah. It wraps, but that's okay. And I will file, say Python file. And now weather observations. So I should modify this to. So essentially. You remove the. Weather local time. Okay. And the line below. So this. Yeah. Or the two lines below and replace them by calling the weather functions. Okay. Pre processing. So I replaced that with this. Yeah. But I also need to add this other highlighted line import weather functions. You need to import the weather functions. You need to know where. Weather where to find that. Okay. And I'll put that here. And I will file save. If I don't save, I guess it's. Bad because it won't run. Should I try to run it again? You can. Yep. So Python. Weather observations. Dot pie. Well, no errors. So should we give five minutes to. Allow everyone to. Do that. Maybe five. I'd actually give, give people 10 minutes. Because it's also. You're also asked to add or export the plotting. So export some more days solution for that. So if you don't mention those 10 minutes. That's not an issue at all. Yeah. Okay. Just in the end use that solution, but. Give it try yourself. Okay. Yeah. So this is good. So this should let everyone catch up. We have 10 minutes. And. If you are having difficulties. I'd really recommend ask someone if you're around with someone ask them to take a look and. See together. Okay. See you in. 10 minutes. We are back. So let's see. We have. About half an hour left. What should. We. Do now should we. Can you. Yeah. And I guess we can. Emphasize if something is not working right now for you. Don't worry too much. So what happens in this lesson. Is that. It won't. Won't affect anything else in the course. This is sort of standalone. So. If you'd like, you can stop and take this as a demo. For example. And then. You'll know what you can try to get to later on. Whenever it's time to do this. And then. Well. If you have your own editor not in Jupiter, this will be easier in. Okay. Yeah. So let's go on. So up until now we only reorganized our code a bit. And made it more. Well. Made it better. Better maintainable kind of. But we didn't really. Get into. How to automatically model. Pass in additional arguments, pass in arguments. From other scripts. Or from the command line. And that's what we're going to do now. Why would we parse arguments from the command line. For example, if you're going to run this on a cluster because. It runs for either a long time and you don't want to block your. Your own computer for a long time. Or you have to run it with multiple different, different input arguments. And you just want to not. Yeah. Again, block your computer because you were running like. 500 times with different, with different input arguments. Then running it on a cluster is quite useful. And then to get the input arguments in and not have. You normally use. You normally use input arguments from the command line or from config files. I might simplify that to say you can control. What the script does without changing the. Contents of the script. Yes. And then that lets you do all these other things. So you could have one script and then easily run it on a different input file without editing it. With different arguments and that opens up things like. Running it on. The cluster and. All these things. And it's basically like the universal interface. So. Python has it R has it C has it everything. And. You'll see just in any programming language and anything which becomes non interactive. Yeah. So in Python. For Python. Python itself in its standard libraries has a. An argument party. That's this. Which essentially provides you all the. Input arguments for the Python call. Which is the first argument being your script. So. This is our V zero is your script. And. Any following arguments are the arguments that are passed into the into. Or that are additional command line arguments. And that can interpret you can also take the. The script file that was being called of course. But commonly you don't need it. So. What should we do an example together. Yeah, we can essentially just take the weather functions file and modify it so that it. I'm not the weather function sorry the weather observations file and modified in a way that it. Takes a command line in arguments as. Input. Okay. So. We add import sis here. Yep. And. This is a module that's built into Python. So just like weather functions. And it provides other things within Python. Yeah. And now we add. What set start and end time. No, we actually modified to three lines again. Here. Yeah, exactly. Okay. So we set those. To not use that fixed first of June and first of October. But to use the first input argument. And. Oh, sorry. The second input argument. And the third input argument. Because again being the script name. Zero. So basically these lines are the same, but instead of hard coding. The state here. We can select something else. Take it here. And this is basic. This is exactly what we'll type on the command line. Yep. Okay. And then now the output file name. I guess that's at the bottom. Here. Exactly. And. You can either. Capture that in the beginning or in the end. I personally keep. Pausing arguments in one place and using them later on, but that's personal style. Yeah. Well. Okay. What you're doing here is perfectly fine. Yeah. So here we capture this argument, which is a string always an arc V. We stored an output name and we save it to that output name. Okay. So I will file and save. And we try running. So I go back to the terminal. Are these things, maybe I'll paste these into the hack MD. Exactly as they are. Just to. The second one. The first ones. In. The other script or in the lesson material. Okay. That's there. So going to the command line. So we did Python by their observations. So. The first argument is a date. Oh, one. Oh, three. 2021. And that's just that arc V one. Yep. So the convention in Unix is the arc V zero is always. The name of the thing that's running in this case, weather observations. This is our V one. And then. One. Spring in tapioca and this is our V three, which is the output file name. Not. Pi. Oh. Okay. Should we run. Yeah. Should work. Okay. If you, if you L S you should have a spring in tapioca. PNG. Hopefully. Yes. Spring in tapioca. Okay. So now with this, we can make a script. Like make some other program that will call this. A hundred times with different date ranges and. Areas and all that kind of stuff. Yeah. You could, for example, make individual plots for all different months. Or for a certain four. Yeah. Potentially also every, every single day. Yeah. Depending what's in the data that's useful or not. Yeah. This data only has like four points per day. Yeah. So per day is maybe not that useful. Yeah. And the reason this is so powerful. Is that the command line is basically made for automating stuff. And running other programs. So. Every computer cluster, every workflow flow tool that does stuff many times. The universal interface is the command line. So it's super easy to make a program that will call this program. Many different times. In fact, in a few weeks, although I believe is running a course called Linux shell scripting or Linux shell tutorial. Or we sort of talk about these kinds of issues. Okay. So. Well. My, my, my personal biggest issue with this kind of, just our power powers or a v parsing is. You have no idea. What you're actually putting in.  Okay. This is date, date. Output file. But. There's little to no explanation on this at all. So if you want to know this. You have to really look into the code. Which is pretty inconvenient. Right. Okay. And any small. If you have, if you imagine that you want to have multiple or more arguments. Then this becomes completely unreadable. And it's very prone to you putting in a argument at the wrong place. So there are. Tools that essentially help you to make this a little bit more convenient. For example, art parts, dot, dot, or click. They are all Python packages that. Kind of make command parsing. More robust. And for we'll go a little bit more into details on art parts. Which conveniently, when you set up the, the individual arguments also provides you with an additional help argument. So that. A user that doesn't know the script. Or doesn't know the code. Can essentially just say. Python. Command. Minus. Minus help. And say, okay. How is, how is the syntax for this. Command. I guess these kinds of things here, like somehow we're defining the arguments. And then. With art parts here. Oh, and I think. I missed something. Lessons wrong. Yeah. Yeah. Yeah. Yeah. Slightly misnamed. Let's look at this one. So this is the start argument. And it says. The help is start time. I think. Yeah, they are. They were misnamed. Sorry. Yeah. But in the solution, it's correct. If you scroll a little bit further down, there's also another. Another mistake that I, that I just noticed. Because we are not initializing the arc. The argument parser initially. Yes. It's actually to the solution because that. So is it further? Yeah. Do you want to see the solution for exercise three? Because that at the moment. So. And now for that, because I forgot. Somehow in that statement, I forgot the. To actually initialize the argument parser. And without that, all of the. All of these other commands. I just raise arrows that part is not existent. I'm not sure what I should do. But in the end. Should we give some. Yeah, what should, should we do? Demo here. Or should we give exercise time? We have 15 minutes left. I would go a little bit over the. How this argument parsing works and what kind of things go into the argument parser. Just to, just to explain how. That ad argument works. So essentially the first two arguments. So for example, that. On the third line. So the start time. The first argument is a shorthand. So that you don't have to. Start, but can say. Like this. Then in this instance, this is a. Argument that has a default value. So if you don't provide it, it will take the default value. You give a type for the argument, which is important because you. Because our parse then converts it into a specific format. That's in our case, that's these are all strings, but for example, you can give a give integer or something else in there as well. I guess they don't give a good error message. Like this is not an integer. This is not integer and you don't need to convert it in your code. That's also kind of my opinion quite convenient. And then you get a can provide a help string. What is this and potentially format of something. Yeah. And on the two lines further up. They are fixed arguments. So they are positional arguments. Input output. Because they always need to be there. And they are not optional. So they are not named. You can also use. Text for. For. Fair for non or for required arguments. But you can as well leave them as these are the first two arguments that always have to be there. So what you're saying is that. The start and end is optional so we could do Python weather observations. With just input. Yeah, exactly. Not in your case at the moment because you don't use art parts yet, but yeah. So this would be valid and then start. Equals. Oh one. Oh one two oh one nine. Yeah. So basically you have good defaults and then you can modify. Okay. Yeah. Okay. So, um, yeah, let's. Give everyone another. Should we give. Let's say to 55. Okay. I'll talk a little bit more about. One thing I want to. Yeah. Okay. We will be back in. Let's update hack empty and. See you. Um. Yeah. So I will imagine that. For. Many people. This wasn't enough time. That's okay. This was basically just to get some experience starting. You can go and, um. Explore more. Yourself. Look at the solutions. Um, and that's perfectly fine. Um, yeah. Like making command line interfaces is something that. I think is important. But really, um. Takes. Takes more time than you wanted to take. There's no, unfortunately, since you have to define all arguments and everything, it just takes time. And it's sort of an art. Like, you know, you get better at it over time. And you start copying things you've done and so on. Um, should I go do the example live quickly? Um, Is it basically the solution here? That's a solution. Um, Or should we just discuss. Actually, I think we can we. We will just discuss and I would like to mention one more thing in the end. Yeah. Okay. So what are your comments? So, uh, as you can see in the, well, actually discussion box a little bit further, you can, you can now use multiple. Different input files. You can use a start and dates and so on. And, um, you can actually loop over different input file patterns to say, uh, or different input files to provide data for different, for different, from different sources. And I'll pass gives you this nice help information. So if you give the strip to someone else, they can much more easily. Fit in there. Um, input data or their requests. Um, But if you look at the potential commands here. Partially depending on a partially due to the size of the input data files, they get long. They get really long. So input lines get long. And again, in my opinion, more or less unreadable. So for any tool that actually needs a lot of in a lot of input arguments, I personally, uh, do prefer to use config files. And, um, actually provide information in config files. Uh, there is a section, another section here about config files that. You can read a few. Uh, if you have time later on. And that can give you some useful information on how to, how to build a config file and what, uh, conflict files can do for you. It's just something I definitely wanted to mention, uh, when producing scripts that. Too many input arguments actually, in my opinion, more than five minutes. Um, get even with a, with argument parses. Inconvenient. Okay. What's your, what's your take on that, Richard? How many input arguments do you think are still well sensible? Yeah. Well, I mean, I would usually instead of having a separate configuration file, like I would do that sometimes, but if I'm running at many times, I'd end up having another script which prepares the arguments and then calls the script. So basically a program that calls the program that does what my work and that somehow manages the arguments in a sensible way. Um, yeah. That's true for it. I think it depends on your, on your use case. If it's, if it's for a cluster. Yeah. Then you might not care about the number of input arguments because they are automatically prepared. Yeah. If it's for another user, I think. Mm-hmm. I guess config files is easier. Also the reproducibility of it. So when you have the config file, then that's saved and you basically have it. And then you, what's that called? So you have it. And then you'll always know what you ran in the future. Well, if you run the command line, it might be stored in the history, but then it's sort of lost. And you don't know how you generated it. And you don't necessarily know which, which input line arguments are led to which result.  Mm-hmm. So, um, yeah. Yes, exactly. So we're sort of running out of time here. There's one thing I wanted to highlight from the hackMD. Can you say something about our parse versus Clic. There's a long discussion of it there. So, um, in practice, In practice in my opinion, they're all. Essentially all of them what is have their advantages And in the end, it's up to your personal style or if you if you have some feature that our course doesn't have or that and that click has use click. If you have a feature that our parts hasn't clicked doesn't have use our part. They are not in any way in my opinion critical, which one you choose. Except if you really need something that only one of them has. And personally, I always use our parts just because it's fewer dependencies. So, like, I so click is probably, well, almost definitely a much nicer thing. But many of my programs I try to make run with the fewest dependencies as possible, especially if it has no other dependencies. And then I'll do that. So, but, you know, in the end it's up to you I would give both a try, because they're both nice. Should we head to a break sound sensible. Okay, so we can keep discussing questions about command lines in the HackMD during the break. The break up after the break will be, I believe, 40 minutes about sci-pi and the bigger library ecosystem, and then a break and then 60 minutes about the parallel programming.