Well, that's that time of the week again. It's time for chitchat across the pond. This is episode number 767 for May 13th 2023, and I'm your host Allison Sheridan. This week our guest is Bart Buschatz back with programming by Stealth installment 150. How you doing today, Bart? I'm doing fine. I'm getting awful deja vu though. So there's two types of audiences here. There's the type of audience who downloads and listens to the podcast Absolutely the instant it comes out and then there's the other 98% of the people If this is the first time you've heard programming by stealth 150 then just just ignore the next minute or so of what i'm Going to say, but if you already heard programming by stealth number 150 You will know that for the very first time in 150 episodes We're recording it a second time because we didn't like it. We didn't think it was our best work Is that a good way to describe it Bart? Yeah, and slightly because as some of our most dedicated listeners discovered Because we didn't do a great job the first pass through The challenge was very challenging So challenging Well, it was unsolvable with the work we had done it would have I mean I know I like to say that people at this stage should be able to do some independent research, but some Independent research it took me three hours To find the missing piece of information Oh, and I'm literally paid to be a professional sysadmini person or at least I was until I changed jobs Oh, yeah, you're all rusty now Bart. That's the thing. See that's old. That's all news Do you know the senior sysadmin actually made foot of me for getting out of practice? I wasn't typing fast enough for his liking don't confess about this I'd like to give a shout out to ed howlin because I think he might have been the first one to point it out He was right on top of it and he tuted you on mastodon And I didn't understand the subtlety because I hadn't tried yet and I was like, oh, I'm not even going to try that If ed can't do it so But a few other people in slack have been chatting about it and But the other thing is in the first recording I was really confused a lot And that's usually something a place where where Bart and I know that there's something missing that we've got Some things out of rhythm. Maybe I wasn't on my game. I'm not sure but I believe we're going to do a much better job in this episode. But the other thing is Oh, go ahead Well, it's so the last time I put too much emphasis in the I was afraid of retreading old work from taming the terminal But I shouldn't have been because a that was quite a few years ago and b it was literally a different audience So I didn't lay a foundation. Yeah, apart from that. It was a great idea So I didn't actually lay a foundation and then I proceeded to build a fairly tall skyscraper on top of it and it didn't go well toppled over in a heap as they say um So the other thing we were going to say we did say the first time recorded is this is programming by stealth 150. Yay Yay, so good. We're doing it twice. That's a lot of work Right, right. Oh, and you have nearly completely rewritten the the show notes I think I was 75 through reading when I started coming across something Written the same way it was written last last time we recorded two weeks ago So if you went through the show notes the first time and were confused they are Much much clearer this time. I think I think the introduction is the same. I think the first paragraph states And I think towards the very end it starts to pick up and look similar But uh, there was a lot of it that I looked at and went man, I don't don't think I knew this the first time through but Now everybody who heard it the first time come on back We're going to lay the foundation and uh and really kick into gear this time and understand it for real Indeed So I've actually the people who heard it the first time can go back to see for another few minutes because we did start last time and this time by mentioning that there was a challenge in pbs 149 and There is a solution to that included in the zip file for pbs 150 Basically the challenge was to take the existing breakfast menu script. We have been slowly building up over Few installments at this stage. Actually, I hadn't expected it to be a long lasting one But it's been with us for a while. It's been fun. So we can all resonate. We got pancakes. We got waffles We got sausage. I mean we're having a good time Kind of are yeah, and there are kind of things I like so yeah, okay That's why and so I asked you to add a an optional argument Minus l limit to specify the maximum amount of things they can make up a breakfast And an optional flag minus s to request some snark to go with your breakfast I was sort of thinking the carrot weather approach to being helpful I think that's why we really liked this install this this homework assignment I saw a lot of people solutions and they were pretty funny Yeah To be honest, I'm not going to dwell on the sample solution because it is extremely by the book It is basically a copy and paste of code from installment 49 and I really struggled to find Kind of anything in there that was worthy of a special mention really It's just Yeah, it's just kind of by the book. So it's in there for you to have a look at There's one paragraph in the show notes, but I'm just going to move us on to Our new stuff because we have a lot to talk about today Right, right. I would say I mean I think it was still a really valuable exercise because just having to do it Is the whole point even though it was There was a lot of inventive thinking to work on this one But to go okay, where does the colon go to say throughout the error and where you know I'm going to write my own errors and and how I put the flags and and I like to exercising the muscle That that's exactly what I hope for in the homework Yeah, and the syntax is a bit fiddly because you're using the case statement to go through the different possible flags and options And we haven't done much practice with the case neither in javascript nor here in bash And even if we had done a lot of it in javascript the bash syntax is Shall we say special? That's a good way to put it very polite way to put it Yeah, the double the double semicolons are a particularly unique aspect to the whole concept Right I I did have one question when I was looking at it if I might ask that So in your case you wrote colon l colon s and let me see if I can get this right So the first colon means I'm going to write my own error message the colon After the l means that it's optional arguments Yes, and then you've got an s and that's just a regular flag No optional arguments What if I have another Optional argument would it then be if would it be colon s colon or sorry colon l colon s B colons Think of the colon as being a placeholder for the option So but it's right up with the if there's another one it gets another colon Yes, exactly. So okay, if we wanted to add another one called p we could say s No colon because the s doesn't want anything then p colon. Good. That's what I was hoping. Okay. Thank you Yes, yeah, it's it's very compact syntax. I believe dense is a word for it Lot of meaning in that little bit special Yes, all right now we should start So this is all about what I love calling terminal plumbing Because one of the actual technical terms from the documentation is pipelining So if you have pipelining the other actual technical term is streams So we have pipelines and streams. Well gosh down at that's plumbing to me So I always call it terminal plumbing And it's about taking inputs from one thing and shoving them into outputs from something else and just Rearranging where information flows and we spent a lot of time intending the terminal Learning about that concept from the point of view of a terminal user. So not writing scripts But just using terminal commands that exist And chaining them together in interesting ways and I actually think that the content we have in taming the terminal is really good It's in solvents 15 and 16 So I've this the fact that we're doing this again does not mean we didn't do a good job last time We're just doing it from a completely different point of view. So we need to learn it Frankly in much more detail here because now we want to keep writing Keep mentioning terminal taming the terminal is a podcast that barton I did that's 40 some odd episodes And we've got links to it. He's got links to it in the show notes Indeed and with the help of amazing listener helma. It was turned into a book as well Which an e-book and a witch by the way, I was uh, bart has heard this already But this tickles me so much. I got to tell the audience about it. I was doing a presentation to the cat mac user group it's a user group back e somewhere and when I got on the call someone was asking a question about using the terminal and she said I really want to know more about learning the terminal But I just don't know where to start and of course I had to chime in I wasn't even supposed to be on the call yet But I had to chime in and say what let me tell you about taming the terminal And someone else in the call brought up shared their screen brought up apple books and had taming the terminal in their books I mean that that just proves it right. Yeah, and I have actually been stopped by computer scientists in the university who said Actually, I sent my students to taming the terminal. I didn't realize that was you Wow, that's so cool all right, so What's the difference between the way you explained, uh, the Terminal plumbing in taming the terminal versus what we're going to do today Well, when we're writing scripts, we actually do need where you're using Commands written by someone else the someone else had to know the nitty gritty detail We as the user didn't really care But now we have the other hat on This is programming by stealth So now we actually do need to care because we're writing our scripts that they will play nice With the pipeline so that they will play nice With all of these things so we need to learn that at a much more fundamental level And where we went wrong in take one is that we didn't actually pair up back to the fundamental concepts Which are not bash specific the concepts are actually much bigger And they come from the posix standard so linux unix mac They all have they all implement this posix standard Which is why so much stuff is the same whether you're using bash or zsh or sh or ksh They all have a whole bunch of stuff in common because they're all posix compliant And the posix standard is actually where an awful lot of this is coming from so much so that if you're a c programmer Writing the cat command or writing the grep command you would need to know pretty much everything we're going to talk about today But you just be using c language syntax instead of bash syntax But what you'd be doing would be the same So I think it's important to understand that we're we're going right back to the fundamentals and then we're going to look at how bash handles those fundamentals So ultimately those three related concepts that are Basically everything today is these three ideas streams file descriptors and files And none of them really mean what you think they do but anyway, let's let's go through them one by one so posix compliant operating systems like the mac like linux, etc They represent flows of data into out of and between processes as streams So they're just sequences of information. They go into the stream in one sequence and they come out in the same sequence That's what makes it a stream um, and the os keeps a list of every stream that every process has access to and Processes use the operating system to tell them, you know things And one of the things the operating system does for the processes is it maintains A table of all of the streams the process has access to and the process asks for a specific stream by number, so there's a unique numeric id for each stream And every process gets its own numbering so if I if I have A cat command inside that cat command. There is a stream zero If I have a grep command into that grep command, there is a stream zero, but they're not the same Right, each one has their own stream zero. So if you're running five or six scripts, they'll all have a stream zero now we actually A stream is a two-ended thing Right, there's an in and an out So the number doesn't actually attach to the whole thing The number attaches to an end of a so when we say The stream has a number that's that's me being hand waving The ends of the stream can have number and very often only one of the ends is inside your script In fact, almost always only one of the ends is inside your script because you're taking information from somewhere So the other end of the stream is coming from something you care about and you're interested in your end of the stream So what you're actually numbering is about this like a like a um a wiring diagram where I've got a Circuit that's a little box and it's got a it's got an input and an output and those things are numbered There may be a wire, but the other end of the wire is unknown Because it's not important to that script to that to that circuit Yeah, you're in the sitting room and there's a bunch of wires coming into you and you can connect to them to do cool things But you have no idea what's happening outside the room But some are coming in and some are going out And some are going out precisely exactly that is it exactly input streams and output streams Okay, and so you refer to the stream by the end You have a hold off and the name we give that numeric id for the end of the stream is a file descriptor So file descriptors have nothing to do with files They are the ends of streams Okay, that's nice of them And it's very important to remember that the same stream if I have a stream connecting two different commands The same stream will be known by different numbers Inside each of the different commands because again, they each have their own view of the universe So they just want to drive that point home. Yeah, so the POSIX operating system Gives every single process it starts three streams As standard and they are always the same they always get the same numbers The first stream which is numbered is zero Oh, I was going to try to get you right before you started because I'm going to stop you and make you go back a little bit In your show notes you talk about readable and writable and I don't think you just said that it's that important for us to State emphatically. Um, I think I said it a hand waving me, but you're right. Let's focus in on it So an input is a readable and an output is a writable That makes sense. So you write to the output and you read from the input Precisely. So they're synonymous with each other input read output write Okay, so starting again now we have three standard file descriptors Yes, we get handed those three file descriptors every time we start as a new process whether we're written in C Whether we're bash doesn't matter. We always get three of them from the operating system So the first one numbered zero is standard input, which is usually abbreviated std in That's number zero. It's an input stream and by default That will be connected to the terminal window in which the thing is running So when that terminal window has focus that means the keyboard Now if you move away from that terminal window and go into a word document or something The keyboard is no longer talking to that script that script actually has nothing talking to it It's going to sit there and wait until you focus the terminal window and then start typing again Okay, uh, if it's trying to read standard output then gets numeric id one It's usually called std out And this is a writable file descriptor and it is connected to the terminal window So whatever you put into standard output appears as text in the terminal window. These are by default By default. Yes Okay, unless we change them right which of course or unless something has been done Of course, we will change them, right because yes of by default Okay, and then standard error is id number two So file descriptor two a standard error, which is std er or and that is also connected to the terminal And so by default std out and std er behave identically the same But we can we can change all of them And we can change out an error to different things while we're messing about with things Which is actually useful to be able to plumb error messages one way and healthy output or happy output another way So when we're doing our plumbing we actually can distinguish right So file descriptors are the ends of streams And they've nothing to do with files And nothing to do with files because I said the three things were streams file descriptors and files So streams check file descriptors check files then please tell me files have something to do with files Files like you think of them are a subset of files like the posix universe conceives them So a regular file is a file So thankfully there is at least some sanity there right a regular file your word document your text file your shell script itself They are files That is good Many many many many many more things are also files in posix brain So then the posix universe the concept of a file is used to represent pretty much anything That takes data or gives data. They pretty much all get represented as files so Folders so basically you have what you think of as a file and what I think of as a file is called a regular file in posix beak And everything else is a special file And there are lots of special files the least special of them Is the folder which in posix beat because it's old is still called a directory because it's ancient Um, but in modern parlance we call them folders We also have symbolic links and hard links which you can think of as aliases or shortcuts Basically two file names End up getting you to the same actual piece of data. That's a sim link And the hard link is a simply different version sim links hard links folders slash directories Those are all special files, but they're still those are all special files. Okay. Yeah, but they get more special Oh boy, do they get more special? Yay so If you if your browser makes a connection to a website There is a stream of information connecting the browser's process to the internet We think of it as a tcp connection posix says it's a file The connection processes the connection itself is a file Yes, it is the destination. Oh, oh the connection is a file. Oh, yeah If we have two apps on the same computer that talk to each other Say you you have your web server and your database servers sitting on the same physical server the same operating system They don't need to go through the internet to talk to each other They can talk to each other through something that's a bit like an internet connection, but it doesn't involve the network and that's called a socket That's a file too um Okay, your hard disk is a file slash dev slash hd with usb hd 01 in the olden days, but now they have really weird names Uh, in fact the part your there's a file that represents the raw disk There's a file that represents the partition table. There's a file that represents the partitions inside the partition table There's a file that basically there's files to represent the cpu. There's files to represent your ram Every device how about my e-card? Yeah, definitely. It has a file. It's a file. Okay. It is a file It is represented as a file as a special file in the file system Okay, and it will be under a slash dev. So slash dev slash something is a special file I just went to slash dev. There's all kinds of stuff in there. There's disks and yeah all kinds of stuff. Yeah Yeah, absolutely. So that is how POSIX operating systems represent devices which can take and post information. They're They're in the slash dev folder and there's special files and Some of them are actually really quite interesting to us. So Three of them that just always catch my eye slash dev slash null Is a writable special file that will take any input you like and Ignore it It is a black hole for data if you have error output you want to just Poof into the ether of existence you write it to slash dev slash null and it just vanishes My favorite part of dev null is no this helps us now understand There was an imaginary character that uh leil lapour played on cable tv a hundred years ago and its name his name was dev null That's that how I get the joke Excellent Okay, so this is a place to just everything goes away that goes there. It's just It's not trash. It is a black hole I believe in america you guys have these yolks in a sink that grind up the food and make it disappear Garbage just we don't have them here. Yeah, we don't have them here and I'm really jealous of them That's basically what dev null is. It's it's the ultimate garbage disposal. It's really useful for making information go away If you just need a constant flow of zeros, let's say you're trying to blank out a disk you can read infinitely many zeros from slash dev slash zero It is a readable file descriptor that are a readable special file that will just give you zeros forever and so you can redirect that flow of zeros to something else like say a thumb drive and completely erase it or whatever So you can you can just kind of flow a a fire hose of zeros is dev zero You can also get a fire hose of random numbers now They're actually low entropy random numbers but slash dev slash random will give you some low entropy randomness on demand Which can be kind of fun Yeah, there's also a dev u random which gives you high entropy randomness But the thing is u random will wait until it has enough entropy So you can read from dev random instantly But if you're unfortunate and you read from u random and there's no randomness available because there's no chatter on your network Or whatever you may end up waiting a long time until it's happy that you have enough entropy. Anyway, that's not there no there So You have all of these special files Most of these special files are universal right every single process on your computer shares the same Slash home slash Allison Every single computer at the same slash etc slash hosts file. We all have the same slash dev slash null We all have the same slash dev slash random But the operating system is perfectly allowed to have Locally scoped files that are different for every process. Why not? and so Whatever the heck is connected to standard in Is made available as a special file called slash dev slash std in And when I read dev std in one script, I will get a different input than if I read it from a different script Because it belongs to the process. So every process sees something on dev std in But they're all different okay dev slash std out and slash dev slash std err Making I'm making a confused face. So you're saying yeah So in in slash dev. I can see it. I can see standard error standard in and standard out What process do those have to do with the ones I'm looking at in my directory of slash dev Your terminal or your finder Huh, okay, but it's always stored there like while I'm in the middle. I'm running a script It's it's writing and rewriting that that file in slash dev No, you're just saying what wants to do with the terminal or in there No Every single process Everything on your computer is a process every single process asks the operating system show me the content of this folder And the operating system just has a different answer for everyone. So everyone gets sold. Oh, yeah, there's a dev std in But if you actually try and read from it, it's actually plumbed into something else. So everyone sees A different copy of the universe It's the multiverse It's the multiverse everyone sees that there is a dev std in but if you read a character from it It will be a different character because it will be coming from wherever your std in happens to be plugged into I couldn't love this more. That's that's so obscure. It's awesome Okay, isn't it wonderful? So you have three of them to match our three standard file descriptors So file descriptor zero is connect is also accessible as slash dev slash std in there Effectively the same thing An sd is such dev sdm is dev sdm. So that brings us we have streams And we have file descriptors, which are the ends of streams and we have files Which are basically anything that can take or send data So stream redirection is the posix basically the posix api allows you to Take one end of a stream and connect it to another Or take one end of a stream and connect it to a file Whether that be a special file or a normal file, it doesn't matter right so posix basically says Streams can connect to streams. They plug into each other and streams can connect to files. They plug into each other And therefore you have a plumbing infrastructure okay Now if you're writing in c there's going to be a whole bunch of c functions for doing this If you're writing in a different language, you have different functions So this is the point where we go from generic posix to and this is how bash does it? So now we're changing into we're now entering bash world All right. This is bash's view of The underlying reality that is posix. Okay, that makes sense. Yeah. So now we're bash all the way forward So we have a collection of operators that tell bash how to arrange the plumbing And the plumbing happens before the command executes So if you make use of the pipeline or the less hand sign or the greater than sign that we're going to meet in a few minutes That plumbing that reconnection of all the different streams That is done And then the command executes. So at the moment the command starts std in has already been rewired std out has already been rewired. So you do all of the plumbing and then you say go Huh, okay. So it's not working sequentially through it It all happens So plum then execute okay Okay, you'll have to explain to me when we get into one of these how that's what why that's important to know um Well, it means that you can be guaranteed that the plumbing is connected and you won't spill water everywhere Okay That is the main reason you need to know it basically you can be sure that it's all ready for you when your code runs Everything is connected all the seals have been tightened It's all set and ready to go. It's really the point of making Okay Okay, so stream redirection and bash then I will mention we are doing this in bash 3 because we needed to work on the mac linux We needed to work everywhere. So we are not using the very very very latest bash. There are more operators than we're going to meet Most of them were added after bash 3 The vast majority of the new stuff are little shortcuts to make to do common things with fewer characters of typing I would argue they are more difficult to read because it's not as if the terminalists particularly were both So shrinking it further in my opinion adds more confusion And we have a term for this kind of stuff where you just have some new syntax That gives you no new features. It just gives you A different way of writing it's called syntactic sugar And so we should be going on a diet and we will not be having any syntactic sugar Oh, no sugary treats for us. Okay Well, I'm over here having the bacon and uh and sausage and egg. So I'm I'm good Now I am going to reinforce our key pieces of learning here. So file descriptors Are numeric IDs representing the ends of streams Files can be regular files or special files including devices Okay, so let's meet our first operator the less than sign Which is an operator that takes a file And plums it into a readable file descriptor. So in other words into a stream It's the input end of a stream. So take a file and connect to the input end of a stream Is what the less than sign does Which is a little confusing Because you put the the thing that's going to shove in on the right And then it's a it's a less than symbol. So if you if you read from right to left you can tell that it's flowing that direction Do you have no idea how often I had to rewrite these show notes because it was the wrong way around Anyway, in your show notes, you've got file descriptor less than file and it's all cuddled together Yes, and that is the syntax. So it's file descriptor less than file now file descriptor is optional So you'll very often just see less than file And the reason is because file description is optional with a really good default zero standard input So what you're doing is you're taking a file and connecting it to standard input and that's almost always what you want Okay, which makes sense because we Standard input was our keyboard in the terminal. But now that we're in the shell We're saying we're going to shove a file in instead of having to type Correct standard input would be the obvious place to put it Exactly. So you almost always just see less than some file Okay, but it could be something else, right? You are if you get really advanced in the terminal You can make your own streams and they will get numbered four five six seven eight. Okay. No, no, no, no, no stop No, we're not going there. We are not going there. We're not going there. So there are reasons you might see number less than sign But yeah, just less than sign file So an example would be wc minus l which is word count minus l for lines less than sign slash etc slash hosts That will count the number of lines in slash etc slash hosts Okay, so the the input file Is slash etc slash hosts and we've got a less than so it's going to shove it from the right to the left and then Then do the the command wc minus l Yes, that's it exactly and remember the plumbing happens first So by the time wc minus l runs it says what's my std in and it's already the content of that file waiting for it to go Got it. Got it. That's that's what that's what I said. Okay Our next up is the opposite of less than it is the greater than sign and it comes in two varieties One of them and two of them Uh, but regardless of whether it's one or two It does the same they do the same job You have a writable file descriptor which gets connected to a file So a file descriptor that sends stuff gets connected to a file that will receive the stuff And the only difference between having one or two is that one of them Replaces the content of a regular file And two of them appends to a regular file And two greater than signs makes no sense on a set on a special file that only makes sense on regular files Okay I hate to put monkey wrench in here, but didn't we in a previous installment Use three less than symbols coming from a regular file To shove in like when we put our menu That's a here string and what that was doing was taking the content That was taking a string and shoving it as a standard input, which is not a stream redirection. That's a whole other Okay, so a less than thing is is the uh is the file to an inputs as an input stream Pay no attention to the triple because that's not That's a here string It's not no it's terminal. It's it's a whole other concept. It's a different section in the bash docs It's that's gonna be hard for me to remember because we've got a file here with a less than symbol And it doesn't mean the same thing. Okay, because this is going to bash that a lot Yeah, bash does that a lot by the way, right? Because it tries to make everything be one character And there only are so many of them on my keyboard It is a finite set so but one greater than symbol and two greater than symbols one is to uh Take from the file descriptor into a file and and overwrite it two of them means to append Correct Other than that, they're basically the same. So again, like with the less than sign There is a default file descriptor in this case. It's standard out So by default if you just see greater than sign file, it's standard out to a file Okay, and you'll see that a lot But you'll also see standard error to a file often something called error.log say and That means two arrow File because two is the I'm already saying the number two not two greater than symbols No, so confusing So his example says Some script.sh space the number two greater than error.log So you're saying take the uh standard error from some script.sh and shove it into error.log And then he's got another example with the number two and two greater than symbols Which means append to error.log Exactly. So if you're running something say Automated every hour in a cron job If you want to see what was the output of the last time I ran the script You would use one arrow because that means every time you're not getting confused for what happened yesterday This is a snapshot of what happened the last time it ran the last time around the last time around But if you're running it a hundred times a day and you're only going to check it once a week Well, maybe you actually want to be able to scroll back in time in which case you choose to use two arrows so that your error log You know has a history So depending on the situation one or the other is more likely to be useful Right, so That's two. I know that we've technically done three, but i'm calling it two The third one then is the pipeline. That's its official name And its symbol is the vertical line symbol, which we call the pipe So it's wonderfully named it makes up for all the other silly names in this entire episode The pipeline is the pipe. Yay sense um and the pipeline of the pipe Has always has a command on the left and a command on the right because it is about connecting commands together So the other two were file descriptor and file In either direction This is command to command or right process to process if you're being more generic about it so What it does is it takes file descriptor One from whatever's on the left standard out And connects it to file descriptor zero of whatever's on the right i.e standard in So the output of the command on the left becomes the input Of the command on the right and that is how you flow information between commands on the terminal Or inside your shell scripts Do you want to know a secret mart? The entire time always is taming the terminal. I never understood the pipe I was always kind of like oh if I type exactly what he said it does this and I don't I never used it in anger on my own and Somewhere in programming by stealth. It just started becoming part of my natural thing to type I actually I piped to grep That's the one I really like to use and and I know that was one of the examples, but it never was I don't know it was something I could go look up and make it go But I had to look it up every time to make it go and now I just I used it today to do something just because I knew it I just I just know it finally. So it only took uh, what four years of that and four years Another very common thing to pipe to is word count So wc-l or whatever because I particularly all my work had on it The question is often how many of these are there and is it plausible? I'm expecting it to be about a hundred thousand So I pipe it to wc-l and says 996 or something. So, okay, that's plausible. It's 96,000 close enough I cannot think of a single time. I've needed to do that That's interesting. This this is where real world it kind of comes in because you're interested in I have a hundred thousand students Who need to have accounts do they all did they all get provisioned? Let's do a word count on the list of users Okay. Yeah, that's that's plausible I would open up excel Import the csv and do account a You know, your overest is h into a You're connecting it through the LDAP protocol You're not quite so straightforward sometimes you can't always just hoof it into excel I do a lot of stuff on excel and I've become really good at writing a shell formulae It is programming But it makes bash look Really simple and really clear And not at all confusing Anyway, we're getting distracted. So As an example, we can say grep local Slash etc slash host. So grep will search for the word local in the file etc hosts And then it will pipe So the pipe symbol will then send all of the matching lines in etc hosts to the word count minus l So it will tell us the amount of lines in etc hosts that contain the word local I don't remember what it is off the top of my head, but you can run the command and figure it out Four was four on mine when I ran it Yeah, I think mono's the same earlier So you're piping you're piping the grep command into the wc command You're taking standard out from the grep command and you make making it standard in to the wc command Bingo, that's it. Perfectly Perfectly Now the last thing we need to talk about is file descriptor aliases so I can take A file descriptor which is the end of a stream and I can show it into a file and that's very useful But I told you Streams can be connected to files and streams can be connected to streams None of those three command none of those three operators had a stream on both sides How do I connect one to the other the answer is Bash can turn no not can turn for every file descriptor there exists An alias that is a file It is ampersand followed by the number So if you need one end of standard in as a file descriptor, it's just zero If you need it as a file if you need it to be on the other side So that you can plumb it together into something else you say ampersand Zero now it's a file This is getting a little like Schrodinger's file descriptor. Yeah. Um, I'm trying to think of an analogy. Uh, I've got a a coax cable Coming out and I need to connect it to another coax cable And so I get one of those double-ended coax connectors and stick it in between. That's the ampersand It's a dongle Turn the usb c into usb a it's a dongle the ampersand's a dongle it makes the file descriptor a file Okay, so the makes the hty of vga Okay, now I hope before I lose that analogy you'll tell me why I would not want to do this Okay, so if I need to I can connect standard out to a file And I can connect a file to standard in And I can connect standard out from one command to standard in from another command with the pipe What if I want to take standard error? And merge it into standard out so that both so if I have something I need to grab And I want to grab both standard error And standard out I need to get the two of them into the grip Okay, how do I do that plumbing? I'm missing a tool. We need a y connector We need a y connector and the only way to do that is to be able to refer to The end of a stream as a file because remember it's stream file descriptor to file So I need my file descriptor to be a file so I can Okay, okay Can I use the ampersand on either one of this standard out or standard error? Yes, your convention to only do one or the other and does it mean something different? No, no, so it can only ever be on the file side So you have file descriptor left or right arrow to a file so the ampersand always goes after So it can be ampersand zero ampersand one or ampersand two. It's fine, but it's always after So it was on the right? Okay But I'm going to be merging that with another one. So ampersand one Greater than two, I don't know The question is where the question is where do you want the output to be right? So if I take Because you don't end up with a new file descriptor you end up with one of them containing both pieces of information Okay, so it's either going to be in standard out or standard error when we merge them Which one do you want it to be? Yes? And you usually want it to be in standard out so that you can then pipe it Into your word count minus L or whatever it is you want to do with it So you're normally taking standard error and smudging it into standard out so that you now have one thing Which can be piped as the standard in of the next command because you have two outputs There's no error input But couldn't you merge standard out into standard error and then pipe it into word count? Okay, but that's good to pipe it often done No, no, but you can't pipe because the pipe takes standard out The pipe takes file descriptor warning connects it to file descriptor zero. That is all that is all it does Only standard. Okay. I did not catch that. Yeah So it's always zero. It's always one to zero So if you have error messages and you need to get them shoved into the next command on the pipeline You have to get those error messages into standard out Okay, in the show notes you said for example, the first command's file descriptor one becomes the second command's file descriptor zero But it's not for example. It's always true It's always the first command's file descriptor one becomes the second command's file. It's ie not eg Yeah, but ie means for example No, that's eg ie means therefore means I am backwards again that I messed up your show notes a lot Oh ie means In other words ie means in other words eg means for example Oh, man, and I just explained it to uh to sandy earlier today oopsie Did I really get backwards again? Uh, she can check the diffs. No, no, I did I did that's okay. I can fix him. Okay. Okay. Good So it is always it is always always always one to zero Oh, okay Right. So all of that is a really long way of saying you will see all over the place two arrow ampersand one Take standard error And connected to the file version Of standard out in other words merge standard error into standard out Two arrow ampersand one all over the place. Okay, so two arrow ampersand one so I'm saying take standard error and pipe it into the file Standard out Yeah, because remember the arrows need file descriptor to file. So you need to be able to represent The file descriptor one as a file. So you need a dongle and the dongle is your ampersand Right, right. Okay It's it's weird but that is it has to be a file on the right And the way you turn a file descriptor into a file is with the ampersand. So that is I didn't design the language, but that is that is a syntax. So think of it as a dongle The next thing I should say is that you can have multiple redirects So if you have some command and then you take the forward the regular greater than sign to send standard out to output.log There's nothing to stop you also taking the error stream and sending it somewhere else So you can say some script at sh space greater than sign output.log space two greater than sign error.log Okay, so I'm gonna stop you right here when I read this in the show notes ahead of time I was surprised so Some script.sh when we say greater than output.log that means by default only take standard out And now That has nothing to do with the next command which is two greater than error.log. It's still looking back to that original file It's not taking something from output.log No, so it's those aren't strung together. They just both happen to be talking to some script on sh They are two redirects that apply to Some script that sh the only thing that connects commands together is the pipe everything else is about setting up street, you know It's jiggering the plumbing So we are saying when you run the script over here, we're going to rearrange the plumbing and then run the script So I'd like you to make two changes to the plumbing Then run the script So the first change I'd like you to do is take this what would have been the terminal And shove that into output.log and I'd also like to take you the error output Which also would have been the terminal and shove that one over there. So you can imagine the old switch board But you can absolutely say it in the opposite order two greater than error.log and greater than output.log Correct because the important point is the plumbing gets done and then the command executes That's why I was going to come right back to was was that why that was important was we set up all the plumbing And they could be almost on like on two separate lines. They're two parallel commands They don't have anything to do with each other And they're both taken to some script. Okay, but when we do pipe, that's a left to right thing happening Yeah, pipe joins commands whereas the arrows don't they're just rearranging the plumbing because they're files on the other end That's really there's nothing to stop you having a less than sign and rearranging standard in as well Like there's really no reason you couldn't have a third one on that line just to confuse the heck out of yourself Actually, that makes me feel better Because remember I was confused. I felt like it was writing from the from right to left Uh, but it really isn't it's these are the the things I want to set up and they're all going to act on this Now you had the the lessons I'd cuddled up against some script.sh or the file or whatever we had there before and I remember pointing it out The cuddling is always on the right of the symbol Not on the left So if you scroll back hope the cuddling would have been no you when you explain the less than operator your example says file descriptor cuddled up less than file Okay, so in the example here the file descriptors are empty Apart from the two which is Dog yeah Yeah, yeah, yeah So that's not a cause that's not a file that gets a file descriptor There's an invisible one right I could write some script sh space one arrow output at log two arrow arrow dot log But the one is the default so you generally omit us Yeah, just because it's confusing enough as it is. Let's not throw ones about all the places as well 100% I did not understand this on take one of 150 It's not straightforward Everyone copies and pastes it and everyone gets muscle memory for it very few people understand it Was very very few was it hard to explain it to write it and explain it very Very hard to explain it and I can promise you there are cis admins with 20 years experience You don't understand that they just know that if I put this magic incantation here the right magic happens You should bookmark this one to send them Do you know what that means me? I'll be bookmarked for me Assisted men test If I if I weren't viewing cis admins, I would absolutely ask them to explain for example this next example is perfect. So The pipeline connects processes together whereas the less than and greater than sign rearrange the plumbing So if you need to rearrange the plumbing before you connect the process to the other You do the rearranging before the pipe. So this is where the order does matter Right, so it's very very common to want to take the the error and the standard out of a script or something And send both of them to the next item in the pipeline And that means you have to have the two arrow ampersand one Before the pipe because otherwise when some script executes it hasn't been plumbed in And then the standard error goes out to your terminal the other script then plums in The standard error from some other script Only it's too late now because what you wanted is gone. You already flushed that down the toilet over there All right the terminal, but yeah Okay, and so his code says some some script.sh two greater than ampersand one So that's take standard out. We're going to merge it in with the uh file standard error Sorry to his error standard error. We're going to merge it with the file Standard out standard out and then we're going to pipe that over to uh some other script.sh Okay, so we're merging the two together. Yeah Yeah, and the pipe always takes one to zero. So standard out standard in Therefore when the pipe happens it now has both our errors and they're not errors and they get shoved into some other script Which can okay, so that's that example again where you couldn't say One Greater than ampersand two. Well, you could but the pipe wouldn't have anything to to take Yeah, the pipe would then miss everything It would all miss the pipe. You'd spill it all out of the all the bits and go wrong Okay, I'll probably do that. Okay What a lot of people do because they think The redirect is always after the command They think it must be on the end of the line and they shove it on the very end And that is one of the most common system mistakes And that is why I'd put it on the interview Because that will really catch someone who knows their stuff who's experienced Right I've done a lot of talking Let's do some doing So no new concepts for the next while. We're just going to see them happen So my standard my challenge solution seems like as good a script as I need to play with PBS 149 the challenge solution that sh sitting in the zip file So the first thing I'm going to do is I'm going to take a file and shove it into standard in of my script And the file is also sitting in the zip file. It's called favorite order dot txt And it is a fascinating file that contains three lines of text two three one Now if you run my script without any sort of chicaneery, you will notice that Two is pancakes three is waffles and one is done So if we now pipe that file into our script then instead of reading from the keyboard every time it asks us What would you like for breakfast is reading from that file one line at a time? So we will order pancakes waffles and then we will finish So if you say dot slash pbs 149 dash challenge solution that sh space less than sign favorite order dot txt It will just never ask you because the keyboard has been replaced with the file And it will go two three one Pancakes waffles done Okay, and you'll just see it happening. You just see it on screen right yay So that is the direction that makes perfect sense The next thing we can do is take our output and shove it somewhere else But of course, we're still going to take our input from favorite order But we're now going to send our output to transcript dot txt So dot slash pbs 149 challenge solution less than sign favorite order Greater than sign transcript dot txt. So we're taking standard out And we are shoving it to transcript dot txt And what do you see if you do a cat of transcript dot txt? You'll see it says choose your breakfast as many items as you like Added pancakes your order added waffles your order. You ordered the following two items pancakes waffles Now What you will have noticed is that the output was before before you jump to the To the part about the output because that is the most important thing, but I've been meaning to ask you this when I do that when I Shove the favorite order in but I take standard out and I send it to transcript dot txt It it shows me the menu But then it writes hash question mark hash question mark hash question mark percent Those are the prompts that you were not seeing yourself typing into you because they're coming from the file Okay, okay Interesting. Okay, so I'd have to go back and look at your code to see why it was so you just had a question let's select it's a it's There's a bit of Because we're only seeing part of what's going on with the select Oh, okay. The prompting is having some debris on your screen Okay. Okay. Good. So keep going. So now all but all I saw In in transcript dot txt. I've got the the breakfast what I what I ordered and the process of ordering But I still saw the select list on my screen Yes now Some things you saw on your screen and some things have ended up in the file But nothing has ended up in both places Right the output has been bifurcated Some of it went left and some of it went right and the reason for this is now clear to us since we understand There is standard error and standard out So the greater than symbol on its own without a number before it means take standard out So everything that was on on standard out went to transcript dot txt Which means everything we saw on screen went to standard error And if you read the docs for the select command it says The set of expanded words is printed on standard error output stream Oh, interesting. That's why you still saw it So you might actually have to when you're when you're using this If something unexpected happens go look at the command you're using and where does the output go? Yes, and you can re-plummet Right, right, but you might look at it and go well, where'd that come from? Right, usually it means oh standard error. That's usually how it works, right?  It's like hang on a second. Oh standard error. Okay. That's where you copy paste two arrow ampersand down. Yeah Yeah, two arrow ampersand one and away you go right, so let us now Fix our code so it's at least consistent because you notice that one of the dumbest things is it says select your food Was on standard out therefore it went to the transcript But the list of options to pick from went to standard error So it was on screen, but the actual prompt wasn't So let's update the code So that it actually goes to standard error as well at least will be consistent Okay, so you will see there is another version of my sample solution called pbs 150 a dash menu that sh And what it has done is it has added arrow ampersand two To the end of a bunch of echo statements the ones that I want to go to standard error So no file descriptor means take standard out Send the two ampersand two In other words the file version of standard error So echo will send the standard out. We are saying na na na na over the standard error with you So you want those things to go to the screen I I want to go to standard error, which for the moment we're still connecting to the screen. Yeah, right, right, okay So now when you run You know pbs 150 a Little inside arrow favorite order outside arrow transcript at txt now you get a more sane output The only thing in your transcript is now you ordered the following two items pancakes and waffles And all the rest went to standard error ended up on your screen um Hang on Am I in the right spot? So you're seeing the line that says dot slash pbs 150 a dash menu dot sh Greater than transcript dot txt No before that The one up here one ahead of me. Ah, okay Okay, good because so we're still saying take the input favorite order at txt and then the standard out goes transcript at txt Okay, okay So we're still automating the order We're still having it enter in the three the 231 Yeah, so it's showing me you added pancakes to your order you add a waffles your order But then it just stops magically but if I look at transcript it says you ordered the following two items pancakes waffles Yeah, okay So basically the conversation was on standard error And the results are in standard out and that's a we now have a sane division of these two things, right, right Now if we take out the less than sign favorite order txt and we make it truly interactive again It now works interactively because we can actually have the conversation with it and choose some use Lee and I don't know what else I put in there that was vaguely healthy like a banana or I don't know if there was a banana in there. I actually never think about it. I should have been So you can now interact with this And then when you're finished your order goes to your transcript. Yeah. Yeah. Yeah, okay So think carefully about which of the two output streams you use for what right? There's no right answer You're the script writer. You have a tool and you choose to use the tool. Whoever it makes sense but choose So you can't really control well, can you let's see Yeah, I guess you did so stuff the the um Some of the comments were going to standard out and thus ended up in transcript.txt by echoing those to uh ampersand by redirecting redirecting them to uh ampersand greater than two. Did I say it the right way? No two arrow ampersand one So arrow. Sorry. No. Sorry. Sorry. I just said it exactly backwards. I said it exactly backwards It's greater than ampersand two Yes, so that's greater than without a number that stopped that from going to standard out and said I want to I want to put that out to the screen Which is where uh a standard error is right now Correct, correct. Correct. More correct than me. You got it wrong. I had to read it. Um Okay, so we can take the the standard error or the standard Out of whatever's going on and direct it wherever we want it to go Correct. So really it's about being consistent so that related output goes to the same place And it may go there by default or you may have to Be a bit more proactive about it But either way you as a programmer You have work to do you you need to actually think about it. What am I trying to achieve here? right right So another thing we can of course do is we can pipeline information into our script so we can say echo minus e 2 slash n 3 slash n 1 pipe Dot slash pbs 158 dash menu that sh Okay, so echo minus creating a file. You're just Saying here's where I want you to input Exactly the echo minus e preserves the slash sorry Interprets the slash ends as actual new line characters. That's what the minus e does otherwise it actually prints back slash n Which is not useful. I learned that one on my own when it didn't work in my script I I've had to find that out on my own a few too many times. I think it may finally have sunk in So in this case echo writes the standard out. We then pipe that to become standard in therefore it becomes the actions in our menu And we could redirect that to something else So we could basically send that to wc minus l to count how many things there were in that order Or how many lines there were which is a heading and then two three outputs So anyway, there we have it. Um Now oh, I can't spell oblivion. I'll fix that it's safe So we can also send you didn't actually You've got you did have a lead into the next section. So the answer that came out was four Yes, be sorry, but the output is standard error is coming from our script is confusing things. Yes. Thank you That was my setup Okay. So in our command, we have our echo minus the pipe to our script piped to wc minus l. So standard out Hasn't gone to transcript at txt this time. It's gone to another process word count minus l But all of the like, you know, choose your breakfast all of that glop It still went on to your screen because standard error Is still connected to your terminal Now we're automating the menu choice by piping standard in And we're sending the output to the word count. So all of that glop is 100 useless Right, it's not achieving our goal at all. We're telling it. I would choose this and then count I don't need to see that Okay, so you're talking about the stuff that's showing up on screen the standard out Yeah, it's noise. It is standard error right because standard error Right, right. Okay. So It's very common to want to get rid of all of that glop In these kind of situations and we do that by redirecting standard error, which is to To the magic black hole that is dev null So if you go echo minus e 2 slash n 3 slash n 1 pipe that to our script and then space 2 arrow slash dev slash null pipe wc minus l which means that In the middle step we take all of standard error in that middle bit of the pipeline and we shove it into the black hole and then only standard out Will continue to the word count Good. Now everybody else has probably figured this out But when you're saying arrow, you mean greater than but it's a faster way. I do just say arrow Everyone does and it doesn't make any sense. No, it's fine. It's kind of like an arrow. It's saying shove it that way Where the pointy end goes Exactly Okay Okay, so let's now look at some considerations when we're writing our own scripts So it's kind of obvious, but I'm going to say it explicitly so it's not implicit every Whatever the standard in standard out and standard error were for your script Every command you start within your script inherits those by default Now you can change them But by default if your script has been sent to a file All of the things in your script are going there too unless you change them right If you think about it, we use the echo command inside our script So if the echo command hadn't inherited our redirections, it would never have ended up in transcript at txt So i'm resetting the obvious here, but if your script is redirected Everything in your script is redirected if your script calls another script calls another script It's all inherited Okay, yeah that Guess it seems intuitive Yeah, I again i'm making explicit the implicit Okay, otherwise things would be really weird Like sort of imagine the universe where that wasn't true and you couldn't actually do anything useful The other thing that we kind of sort of know already is that reading a stream is destructive If like think of it like a river if you take a book out of water out of the river It's out of the river So a stream is a sequence of information coming out of some we use a read command to take one line of it Or we can use another command to take it all but When we take it out of the stream, it's gone from the stream now Don't worry If I turn a regular file into a stream, have I deleted my file? No, we haven't so what's actually happened is that a copy of the file has been turned into the stream And so you read the copy and as you read it it gets destroyed So if you need to use information later You need to save it somewhere Because the stream is gone, right? It's it's ephemeral So we actually do have to capture standard in if we want to use it later So we have a couple of ways we can capture standard in so the read command is a way of reading one line at a time from standard in What it's really doing under the hood is it's reading one ifs at a time, right? So the dollar ifs special posix variable is the input field separator And what it's basically saying is keep reading standard in until you meet one of these And then it stops So when you say so that's why changing ifs to the blank string makes read Empty the stream Because I don't care how many new line characters you give me I'm going to keep reading because I don't have an input file separator So if you want to read the entire content of a stream in one go with the read command you set ifs To the empty string and it just keeps reading forever until it until the stream is empty And then it stops because the stream is empty Now I don't like doing that because if I change the value of ifs and then I forget to put it back how I found it I have now made spooky action at a distance So I use dollar ifs I change dollar ifs to something to to solve a problem of not being able to find the new lines and When I showed you that you said oh And you got the spooky action at a distance thing, but if you have discipline Isn't that what it's for? It's it's whole job is to be changed and then put back Yes, but it's a very crude technique that dates back to the very early days of it's It's inherited from shell as opposed to being a good way to do things like in the 70s before we knew better Okay, so it's so your your weak sauce if you have to resort to it. It means you just didn't figure out the right way to do it Is that harsh that's that's a little bit too harsh There are times where like there are times you have to actually genuinely write us every language But you shouldn't do it unless you have to because it's kind of dangerous And the ifs is kind of like that most of the time there is a better way But I am not going to tell you that it's always wrong Because that's that's too strong. I did figure out the right way to do it and didn't have to use it But uh boy, it was offered as the solution all over the the internet told me it was the right way to do it Bart The internet is awfully sticky sometimes Because what happened is someone asked their sissid min friend from the 70s How you do this and their sissid min friend from the 70s said this and it worked And that is now stuck on the internet forever and ever and ever never amen And no one really understands that so they just copy and paste it and it just spreads Okay, that's how it works now I prefer to read all of std in at once using A technique called slurping which I love And we can use the cat command with no arguments because if you open the man page for cat It says by default it reads the standard in and sends it the standard out Now we know that dollar open round bracket some command close round bracket takes whatever the output of that command is and shoves it into a variable It's the value of that command is how I've been describing it to you But it's actually standard out becomes the variables value is really what's going on So if I say some variable equals dollar open bracket cat close bracket I have just slurped all of standard in into the variable Whatever I call it in one go but We don't know what standard in is Right, it's just this is a script that whatever it is a script has standard in And there's always one yeah, if it's if it's going to run it's going to have to get a standard in From the keyboard or from a file or Okay, exactly right So if I need to read my input, I don't care where it came from right? I'm just saying You will send me some input. I will accept that input and then I will do something I can just take all of the input I was given With dollar open round bracket cat close round bracket. Okay, so it is literally cat It's not you could use any other command. It is cat that takes all of standard in It just slurps it. Yes. Okay. Yes. So if you say man cat it says default behavior Take standard in send the standard out So it's default behavior is to do some very simple plumbing Okay, and then standard out is being because we gave it the dollar symbol. It's saying shove that into this variable name Yes. Ah, okay. I read this four times did not understand it until you said it got it. Okay This is what I mean by the terminal can be very um Say a lot with very little characters. Yeah. Yeah, very few. Yeah So we have an example script called pbs 150 b naive shouter that is h Which takes input turns it into all caps and then outputs it again. So it shouts at you So It's just the input becomes equal to dollar open bracket. Sorry open parenz cat close parenz Then it says the output becomes equal to dollar echo the input piped to tor and then some special regular expressions in fact The tor command stands for translate and it takes a string of characters and it maps them to another string of characters So in other words, this is going to take lowercase a and map it to uppercase a Lowercase b and map it to uppercase b It's a translation from lowercase to uppercase Is what it actually is Okay, it's It's a strangely powerful command tor again. It's quite dense And man tor will tell you all about it and you'll get very confused and then you go to google to get a real answer Yeah, that may be how I got where I needed to be and then I just say echo the output So we read from standard in we make it uppercase and we output it the standard out That's all the script does So you'll as I say it's sitting there in the um sitting there in the zip file so We can run it So we can say echo hello world pipe dot slash pbs 150 b naive shouter and it will shout hello world But what happens can I can I describe what happened then so we echoed hello world we we piped it into the naive shouter and uh So the first thing the naive shouter did was it took standard in which was hello world And it slurped the whole thing in and shoved it into a variable called the input And then we ran this funk we ran this command that changes it all up uppercase And then you had echo dollar the output which meant shove it to standard out And that was the uh why it comes out in caps Correct, okay perfectly. All right perfectly described So this worked brilliantly because standard in had some information we slurped The output of our echo command therefore it worked perfectly. We had something to slurp What happens if I run that script or you run that script without a pipe just dot slash name of script Well nothing happens standard in exists It's connected to your keyboard Uh, but we're using the cat command not the read command. So it's going to keep reading Until it meets end of file So you can type away and hit enter and type away and hit enter and type away and hit enter and it's going to keep wanting more And more and more and more and more How do I tell it i'm done What is really waiting for is the end of file character, which is not on your keyboard But it is available to you by the magical combination of control plus d So when you're finally done being frustrated hit control d and it will stop So it not only stops it does something very odd. So I typed in abc in all lowercase Then I hit control d and it wrote capital ad capital b capital c Yeah Why did the where did the d come from? Uh, it must have been your control d unless you maybe you're right, right, but to put it after the capital a Maybe maybe I can't I can't tell I don't remember that happening, but Yeah, well, I tried it twice just in case I you know fumble fingered it and it does So it takes it does the capital a and then it goes. Oh, I got the d And then made it a count even though it was a control d not a regular d. That's that's Spooky action at a distance there Bart Yeah, it is but you really don't basically you shouldn't end up in a situation where your users have to know the type of control d Okay, good. Okay. That's why we call that a naive shouter So we actually need to be a little bit intelligent when we slurp standard in Now my first thought was well, there must be a way to just ask bash Is there something there? Like is standard in empty? Or is there some information in it for me to take out? And that's actually impossible if you'd like your script to work everywhere It sounds like it should be easy Tell me is there something in the stream is the stream empty, you know It's like does this river have water in it should be an answerable question But it is not unless you use bash five when you can use the bizarre incantation read minus t zero Which will then have a success code that tells you whether or not there was something there to read It reads nothing but tells you whether or not it would have succeeded if it had It doesn't work on the Mac It's not cross-platform. So we actually need to take by the way your show notes say never on a Mac There is a reason apple doesn't ship the latest version of bash. It's not that it's behind It's that they have a an r apple disagrees with the license for bash four and above, correct Yes, it is all such a such a philosophical argument But the gpl version three is different the gpl version two And some people have issue with it clearing line is torval's So it's not a particularly niche position Okay, but bash four is gpl three Therefore, it is not on the Mac Okay, I just wanted I just want to make sure people didn't think it was just old and stupid No, it's just philosophically Yeah, oh, it's a nerd fight Okay, so we still have to have a way To know whether it's standard in has content to slurp. Is there a worker? Yeah, and There is the work around is to copy what everyone else does Basically be like everyone else So cat and grep Are under the same restrictions as my script and your script. So how do they handle it? The answer is They don't detect They make you say Take it from standard in So they make you tell it to go slurp And there's a convention explicit Yeah, so it's a two rule algorithm if There is if there are no files specified. So when I just say cat with no arguments It says oh, you must have meant standard in you didn't tell me anything else. So what else could you have possibly meant? I'll go slurp The grep command is kind of the same behavior. If you say grep space one argument That means grep pattern And that should be followed by where to search But if you just say grep pattern Well, you didn't say where so grep goes well, I assume you meant standard in then If you tell me nothing, I'll assume standard in So that's the first part of the algorithm is if you didn't tell me somewhere Don't all commands assume standard in? Well, all commands have a standard in Oh But cat and grep are different No, no, whether you choose to slurp them is your choice, right? Every command has a file descriptor Called zero connected to something Do they choose to read from it? Okay, let me see if I can say that then so cat and grep Uh, do have a file descriptor zero So they have a they have a standard in but whether or not you're you need to tell them whether or not to read from that explicitly Right, because if you read from it when it's empty, it'll wait for you to hit command d guy you control d. Yeah control d Right, right. Okay, right. So the first rule is if you didn't tell me where to look I'm just going to read from standard in because well Why else did you call me? Why are you asking me to search for a pattern? Well, you know, why are you asking me to print something out? Why am I here? Right through an existential question. So it assumes if you didn't tell me anything at all read from standard in The second thing is if you do give me a list of files to go and cat or grep I will check each one if the file name is the symbol minus I'm going to read standard in And basically I'm going to use minus to represent standard in If you give me anything other than a minus I'm going to go read that file. So if you Cat minus is the same as cat Okay, good. That's what I was going to say. Why do you need to say a minus? If cat doesn't require it Okay, so cat can take arbitrarily many files and can catinate them together So you could have header dot txt minus photo dot txt That would take the content of standard in And stick it between header dot txt and photo dot txt That was the top my head going off. No, let me let me let me let me try to do it again Because I might be able to get there So the the minus is standard in but it can be one of a list of things that can be the input And and it doesn't stop it from talking to taking an input file When it runs into one of those it's not like it's got a got a a finish line on it or anything It's it's would it exit or say anything? It okay. Yeah. All right It's just a code word If I it's like if I say the the sparrows are nesting in the winter That means I'm a russian spy if I say minus that means go read standard in I didn't follow the russian spy part, but okay Isn't that a thing in spy novels where you like sit down next to someone in the coffee shop and say some silly innocuous sentence and it means I'm the russian agent who is here to meet you? Have I been reading too much James Bond? Maybe I've been reading too much James Bond Anyway, okay, it's just it's just a it's just a code name for go read standard in Right, this is a lot easier in in human language conversation than it is reading the show notes because it just says I'm not complaining about the show notes It's the only way you could write it, but it says it checks each pass File path to see if it's the string minus comma and if it is comma read from standard It's like wait what mine wait what? But now I understand what you're saying. Okay And the good news is that even though it's a minus dangling on its own get ups does not get confused So it will actually work with get ups. So it's not looking for a flag or anything Yeah, because it's just a minus all by itself and because so many so many things implement this convention that a minus on its own is Will be interpreted as standard in right it doesn't right it doesn't actually mean it It's just we've all agreed that we'll call it minus Right, it's a convention Really interesting. So make sure you cuddle your your minuses when you do want it to be a flag Because well otherwise it's all gonna be two arguments. Yeah. Yeah. Yeah, okay So we have a shouter That is not naive. So pbs 150c shouter and this implements what what I just said in english is here in bash so We've also added an optional minus b flag to prove it works with get ups So the minus b is an optional flag that sounds for bang So if you put a minus b it should show an exclamation mark after it shouts at you So we take the input we shout it and then we shove a minus We shove an exclamation point on the end if you said minus b I just needed something to prove to you that it works with get ups Okay, so our new shouter Will behave like cat So we start off and we do our usual get ups thing and we basically say if you gave me the optional flag b The variable bang becomes equal to the exclamation point So that all of that is just our standard get up stuff So now we come to the bit where we have the algorithm. So what text is going to be shouted Well, I'm going to start by saying text becomes equal to the empty string So I'm going to start by saying I have nothing to shout So then I said the first point of my two point algorithm was if zero files are specified read from standard in so I say if Dollar X dollar octothorpe dollar pound sign dollar whatever other aliases we call this bloody symbol Is equal to zero then Text becomes equal to our slurp command dollar open friends cat close friends. Okay. What does dollar? hash mean So we're going back Four weeks now to when we talked about special variables dollar octothorpe is the number of arguments Okay Thank you It's one of the strange ones that actually makes sense because dollar one is the first argument dollar two is the second argument Dollar hash is how many were there? Okay, okay, good Yeah, so if there are no arguments read standard in so that's a point one of our little algorithm So point two was go through all the other files and if it's a minus read from standard in otherwise read from the file So for path in dollar at so what was dollar at also going back four weeks That's all of them Correct. So we're looping over our arguments So every time we go through the loop path will be the next argument So in our loop the first thing we say is is path equal to minus if it is slurp Text plus equals dollar cat dollar open friends cat close friends else Read the bloody file So text becomes equal a text plus equals dollar cat dollar path closer friends Done got you okay Then we convert to our shouted text becomes equal to our To our stuff again and then we echo out our converted text So now let us prove that my script does what I say it does So if I echo hello world and pipe it to our shouter It will shout hello world in all caps right So that means there are no arguments So it went through and it got as far as if dollar octahsorp equals zero. Yeah, it does. Okay, then I will Read from standard in Which is Uh, there are no are you oh because you didn't put the b on Uh, well, even if I did put the b on remember, that's a flag that we make go away with the shift So what argument are you saying we haven't given it? Well, none we we didn't give an argument What what argument is it looking for? Well, it's looping through all of the arguments to see to see if their files are not but all of the arguments Is none of them so the for loop never happens So the echo hello world is not an argument. That's just pipe What what is the echo hello world? That's one command No echo dollar world is command one Pipe command two is our script So standard out from command one becomes standard into command two They are two entirely separate commands joined together with the pipeline and standard in is not an argument No Okay, it is an input. It is it is it is a stream Right, right everything after the pipe is a whole separate command. So the entire command is dot slash pbs 150c That is the command. Okay. Yeah. Okay. Okay. I somehow was feeling like it. It was an argument, but I see what you mean Okay Yeah, because we're yeah We can do the same thing with the minus b and it still works Because We after we do our get ops dance We have the copy and paste shift echo dollar opt-in minus one That thing we learned about four weeks ago that I told you to just copy and paste that will make the optional arguments disappear This ringing a bell. Yes Okay Said hate to have to do that again because I think I'd probably get it wrong Uh So that's proving point one if you don't tell me where to go shout then standard in So now let's run the script with an argument dot slash pbs 150c shouter space menu dot txt So we now have one argument menu dot txt So our if statement if the number of arguments is zero well that never happens because the number of arguments is one Menu dot txt so now the for loop happens because at dollar ampere dollar at contains one thing menu dot txt So we'll go through the loop once is menu dot txt equal to minus no Therefore we skip over the slurp We do not read standard in what we do instead is we cat The file we were passed and we add that Into our text. So I think a better example would be if you said echo. Hello world pipe The script then menu dot txt because then there is a standard in you haven't given it a standard in but this tells this Doing that proves that it's going to ignore that standard in Okay, but in the real world you would never pipe something to it and then intentionally not use standard in Right, but I'm saying is an artificial example of showing that it's had a standard in but it went because with the with example of Just saying this the script menu dot txt. It had no standard in to look for well, I guess I guess it proves It doesn't wait at the keyboard Right that which is actually the problem to be solved. Yeah, okay, got you Okay, yeah, we safely understood it though. Okay. Yeah, exactly So we have safely piped when we when it made sense to pipe We've safely slurped when it made sense to slurp and we've safely not slurped When it didn't make sense to slurp which is our problem to be solved. No control d shenanigans here. It actually did the right thing and we can also Use the minus To have the menu dot txt get shouted and to have standard in get shouted So if we say echo minus e slash n slash n.  Hello world as standard in so that's sitting For our script as standard in We've set the minus b flag to say add an exclamation point on the end That's also sitting in our script now And then we have two arguments. So let's let's look at our code Is the number of arguments equal to zero? Definitely not. We have right minus and we have menu dot txt So that number happens we come into the for loop dollar at has two values Uh, let me make sure I get this right. So menu dot txt is the first argument And then the second time on the loop will be the minus Right. So the first time through the loop It says is menu dot txt equal to minus. No, it isn't therefore We will read menu dot txt and shove that into our variable Second time through the loop is minus equal to minus. Why? Yes, it is Therefore, I shall read standard in what is in standard in. Oh, it's this new line new line. Hello world So that gets added into our variable after our menu I completely understand this Yay, good because we were so not here last time No, no, no, no. This is sort of like upside down world from the last time Not even close and and again the value of how well you write this and how well you explain it Because I I read the show notes, but I got to this part and I went I don't know what he's talking about But now when you explained it and I'm looking at it the two together chef's guess it's working Excellent. So what actually gets printed out is our menu followed by two new line characters So the first one basically ends up with a new paragraph that then says hello world And then because we said shouting me Oh shout it at you and because of the minus b there's an exclamation point at the end just for a little final finesse So proving that it understood what standard in was with the minus that it wouldn't got the echo It understood the flag minus b It didn't get confused on that and it knew to take the input of the of the text file first Perfect. Yeah, got it and our script Behaves just like cat and behaves just like rep We have written a script that is a native it behaves like the big boys We are capable of replicating the functionality of the core Features of the terminal we can write shell scripts that are first-class citizens of the terminal They behave properly in the pipeline That's really useful because now our scripts can sit there and be used by everyone who's used to using the terminal And they will play nicely with all the other utilities we love And if you write five or six scripts do five or six things you can chain them all together Because they're behaving properly they're behaving like every other terminal commandos And that is very powerful. Yeah, right now this Folks who are listening to episode Take one. This is new stuff here. I need to give you one more piece of very important information Which is a really darn useful So darn useful you will need it to do the challenge only I didn't realize that you would need it to do the challenge so We know that out of the box Whenever the terminal starts a new Command for us a new process a new script It gives us a standard in a standard out and a standard error and unless we've done some Plumbing it's the keyboard and it's the terminal twice And that's great, but what if we need to have our cake and eat it What if we need to take stuff from standard in? And we need to write stuff to standard out And we also need to absolutely positively always get something from the keyboard in the middle So the most common example of this Is you can use the ssh command To run a command on a remote server and you can pipe the command you want to run to ssh And you might want to take the result of that remote command and do something else with it So you want to pipe the output of ssh to something else? But in the middle you need to get a password ssh needs to say to you I need your password. Please type it in Standard out is connected by the way Oh heck and no, you don't yeah very good. Okay, put this on the test too Oh, yeah, that is very high in the test. Anyway, um, so Standard out needs to go to whatever you're trying to do with the result of running the remote command And standard in has to be their remote command But you need to tell them to type and to read what they type How The answer is there's another magic file that I haven't told you about yet. It's in slash dev So it's another one of those files that's different for every process So when the terminal started the ssh command It gave that command a copy of its own private copy of slash dev slash tty And slash dev slash tty is a file Representing the terminal that you can read from and write to Both Both it's a twofer. It's a two and one It's bidirectional. It's a it is a read and a write and you can You can use it anywhere. You can use a file So that means that you can say less than sign slash dev slash tty to read from the tty And you can say greater than slash dev slash tty to write to the terminal So you can echo. Please enter your password space greater than sign slash dev slash tty and it will come on to the screen And not go to standard out Because it's not going to standard out. It's going to dev tty Which is your terminal Okay, so all of your plumbing is still in place. You haven't broken the pipes You've just written somewhere else completely which is connected to your terminal still is that like a Time-based permanent sort of thing like I am now in dev tty And I need to now get back out or is it just for that one thing you're doing Like any other redirection so if you have My script and then less than sign dev tty everything inside your script inherits All of the standard descriptors it was given So inside that command standard out is now dev tty Because it's inherited all the way down Now in this case you're saying echo so the echo command is just so echo less than sign dev tty means It only is the echo command right because the echo command doesn't Have anything inside it. It's just the echo command But it could be my script arrow dev tty and then everything in my script would be using dev tty But the next line of your script is not affected by any plumbing you did on the echo command, right? Yeah, right, right. Just like when we said echo Echo um arrow ampersand two We only wrote the standard error for the one echo command Because that was all we reclaimed. I just wanted to make sure. Yeah. No, it's very important and Yeah, okay So we can use the left the less than sign and the greater than sign with dev tty To get from the keyboard and always the keyboard and to write to the terminal and always the terminal Regardless of any plumbing that's gone on around us So we can have our cake and eat us And this was the missing piece for the challenge. This was the missing piece for the challenge. Okay. All right So let's have a look at the challenge then shall we? I would like you to update your menu script I would like it to default to reading the menu from a file named menu dot txt in the same folder as the script Ie default to doing what it does now That's what we're doing already If you pass the optional argument minus m space minus Then I would like you to read the menu from standard in If You pass the optional argument minus m space name of file I would like you to read the menu from name of file This should sound familiar to you. This is yet another way of saying our convention Only i'm saying by default read from menu dot txt. Otherwise minus m minus means standard in minus m something else means something else Regardless of where the menu came from When you present the options to the user Always present the options in the terminal And always read the answers from the keyboard so You're telling him you're saying that we're going to have it read from a file But then ignore that and always make a build on the menu You know the menu comes from the standard in or the file Your choices are always interactive like the password to ssh is always interactive Oh, oh, okay The menu is what you choose from not what you have chosen So what is path to file dot txt? If you would like to use a custom menu So by default the menu is read from menu dot txt But what if you'd like to use a different menu? Then you can say minus m different menu. Oh, so like this is our specials today Exactly Okay, or you can accept the menu from standard in so you could have like a database query And then pipe the result of the database query to standard out, you know To standard in and then have that become your menu if you're going to get really fancy paths. Okay. No Um, I'm barely hanging on a thread of what what this means. I think I understand. If not, I'll ask Ed He's my go-to now, man. He's got it all going on Now there's a lot going on here. So if you'd like to make your code easier to read You may if you choose to get rid of your minus s for snark flag Okay, but you can keep the menu everything snarky You just don't you don't give him the option is what you're saying Well, to be honest that's optional right if you'd like to simplify your code you can take out those if statements Or you can leave him in or you can be permanently snarky I would like you not to get distracted.  And I actually understand the lesson now. That's even even more important Excellent. Oh, I knew it was going to be a long one Which we both did but hey, I'm I'm glad we've arrived in the right place at the end of it all But I have run out of steam very much. So thank goodness we're here. Okay Well, had you chopped this up into two it absolutely wouldn't have worked. It said to be one big piece Yeah, no, I am I I did look for a place and I was like no There's no place We're doing it. We're doing it. You have final thoughts on this Okay, so we are We are really making hay here. Like we have really done a lot We've a few loose ends to tie up. So Most of our loose ends are actually Revisiting things we've done but explaining them properly So we have seen most of what we need but we haven't given us all the right names Which makes it more confusing than it needs to be so I'm going to do a bunch of cleaning up particularly this concept We have seen what never named called expansions But before we do that I need to teach you something really practical There's a better way to give output than echo. It's printf and it's way more powerful So the first thing I'm going to do is better output with printf And then we're going to circle back We also need to have a little bit of a better look at the scope of things which will make it Safer to mess around with dollar ifs and it's ilk Okay, we learned a little bit more about how to do that in a controlled way And then I definitely want to talk just finish up with a good explanation of what all the different brackets mean And what I mean by these expansions because you have round brackets. You have two round brackets. You have curly brackets A lot of different brackets going on here and actually they do make sense But we've met them higgledy-piggledy bit here and a bit there And so we should pull it together and one of the reasons I'm keen to have the pull it together episode is it will be a quick reference Yeah, oh, I would love that much as for anyone else Instead of okay go over to this lesson and then that lesson and I'm using the pbs index from dorothy and asking dorothy Where did that go and she tells me where to go find things probably be be better to have it all In one one wrapped up place. This has been a great series and I'm glad I'm glad we're not done yet But uh, it's been it's been super super fun I've mentioned a few times here that we have a slack community And uh, you get to that by going to pbs.com. I'm sorry pod fee dot com slash slack Where you can meet people from all of the no silicast at pod feet podcast But there's a programming by stealth channel in there It says pbs and that's where all the really nerdy people hang out and bar Occasionally drops in when he has time and answers a few questions or makes his own little snarky comments as well So, uh, I highly Love heart. Yeah, I often I often love heart things. That's true. There's a little thumbs up He's he's definitely very positive about everything in there But uh, uh, the entire trick to joining is going to pod feet dot com slash slack and at joining There is no there are no hoops to jump through. We never have any spam. We've never had any bots We don't don't tell anybody but it's secretly awesome Yes, exactly and that's because we say it not type it I guess but yeah, it's a it's a no I type it too. I don't know why you get good. Let me deserve time. My merit always says Which is one of the things I love most about working with you because you curate an amazing community It's one of the furthest things that are being involved with the no silicast is all the no silicast ways such a group of rocking people Right. Okay. Well folks lots and lots and lots of happy computing until we reconvene For the penultimate bashing So anyway talk soon and happy computing If you learn as much from barn each week as I do I'd like you to go over to let's dash talk dot i e And press one of the buttons over there to help support him He does 98 of the work here I'm just the stooge that listens to him and asks the dumb questions If you go over to let's dash talk dot i e you can support him on patreon You could donate via paypal or you can use one of his referral links I really hope you'll go over and help him out In the meantime, you can contact me at pod feet or check out all of the shows we do over there over at pod feet dot com Thanks for listening and stay subscribed