Well, it's that time of the week again. It's time for chitchat across the pond. This is episode number 786 for February 3rd 2024, and I'm your host Allison Sheridan this week Our guest is Bart Bouchots with another installment of programming by stealth. Hello Bart Well, hello there Well, I've got a big thing to interrupt the show with before we even get a chance to get started Just a few hours ago Helma van der Linden and I recorded chitchat across the pond 786 I'm sorry 785 the one right before this one and we did that because we are now officially announcing the launch of the beta version of XK past wd net along with the Github work that you can do and the help that we need in order to get this thing started And it's a wonderful episode where Helma explains basically how she kind of Went around behind Bart's back and did most of the work to get it started Which I am very grateful for right right not in a bad way really grateful in an open sourcey way, right? Yes, and even better than that because in the open sourcey way There was no reason for Helma to ever do me the courtesy of checking with me what I would like and talking through the details But Helma chose of her own volition to do that and then to actually like, you know, do do what I said would be nice Which is extremely kind of her and in open source land. There's no reason. She couldn't have forked it and done whatever she wanted Right, right extremely kind it's it's amazing, so it's actually technically working and Shortly there will be a URL people can go to don't know how soon that's gonna be Depending on how wibbly wobbly timey wimey things are Depending on when this episode gets published in Europe so get published sometime in the very near future beta dot xkpass wd.net will host this new code and Because my server is being shut down in three days. Actually my server is being shut down in three days So this is happening soon The may WWE website is going to redirect to the beta But I'm keeping it branded as beta so that people understand that the features they love are not gone they're just Not there yet between phases. Yes, they are their own holiday. They'll be back, but they're on holiday Yeah, so as of right now technically you can press a button and generate some passwords But it's the generic preset. You can't set up your own presets. You can't choose from any other presets There's very little you can do beyond that But you can press a button and get long strong memorable passwords that you can actually type And you can choose how many? Yes, you could choose how many so anyway, we want you to listen the episode It's just chat across upon 785 and Helma and I had a great time chatting and going through the process of how she got there And I tried to do it as a little more generic than I'm sorry a little more Non-programmy we tried to keep it as a high level as we can but it might stretch the Definition of the word light a little bit, but then again we had an astrophysicist on to talk about her Nobel Prizes under the light topic so Yeah, no silly castaways. We're made of strong stuff. We our light is quite strong. I Like it. I like it. We quote that so with that's enough further ado We should we should get stuck in on the work that we're gonna do here today Indeed so we are on our jq journey And I had promised you last time that we would spend this episode Learning about all the different cool functions that jq has for transforming data. So we learned how to build Strings using interpolation how to build arrays using the square bracket syntax I had to build dictionaries using the curly bracket syntax and I we learned how to do some, you know Basically the idea that we could transform the data into different shapes And then we sort of we got a look at some functions like L trimster and or trimster and stuff like that But I said there's loads and loads and loads of functions And then we spent the next installment talking all about the functions And then I realized that I was not doing things in the right order. So I've changed my mind We are going to learn how to use jq as a true programming language today Because I am so many shades of fed up of putting all of my jq code on one big line I can't read I would like to break it into separate files Where we can put things on separate lines and not tie ourselves into knots So this episode is dedicated to saving all of our sanity By using jq as a true scripting language instead of just as one big string. Oh That sounds wonderful because it is a little hard to read All right, and the more we're doing so if I were to introduce you to all the cool functions You're a one long string. We're gonna even worse. So I figured before I make your one long string worse how's about we stop doing it as one long string so that is the plan for today and We also have a challenge solution and I believe that you had a very productive time of the challenge I had so much fun with this one. I spent a lot of time and by a lot of time I an embarrassing amount of time for probably it compared to everybody else, but I spent Probably six or eight hours almost getting to the the conclusion of the first challenge at the primary challenge But I didn't quite get there and so Bart said, you know after I hadn't given up or anything But Bart said hey you want to have a play date where we do a little buddy programming And I said sure that'd be that'd be fun And he got me over the hump of the piece that I didn't get on the main part But I was so excited by that about that that I kept going and I succeeded on the extra credit as well You did and I know indeed you succeed because you did share your solution before you recorded and You did it better than I did your solution is better than the example in the show notes And they're all if they can both be right Bart Absolutely, they are both correct and I always say there's an infinite number of correct answers But yours has the advantage of being shorter Clearer and easier to read and I like pretty code So your code is prettier and I like that Okay, okay I did I did get a little assist from Github co-pilot was part of it where I was asking it some questions And this is again where I want to give AI Thumbs up is that I don't I seem to have trouble with remembering terminology But I can ask it questions without knowing the terminology. Yeah, I could say okay What's that thing in JQ? You know where you got an array and you need to look inside it What is that called again? You're gonna like open it or whatever to go. Oh, I got you Yeah, we're gonna explode the array and it'll do it for it'll show me that syntax You know, that's a silly example But that's the kind of thing I can do and so I was asking it questions about like how do I how do I say I want I want to only get the elements of the or the the dictionaries that have Laureates who have surnames like if it doesn't have a surname I don't want to have it and it came up with a piece of code that I ended up being able to use Excellent and I do like that Microsoft called it co-pilot to make it very clear that this is not a tool to replace the human This is a tool to assist the human and if as long as you use enough words that are going to show up in correct answers The fact that it's really good at language matching means it is going to Figure out the right part of the internet universe that goes end you to and it tells you where it found its answer So even if it's wrong, you're still better off because you know where to go look Yeah, and it actually was wrong, but the peace I needed to learn was something interesting So I got it's like having a cope a copilot who's just about as good as me It's a trainee but but it's better at remembering terminology. Maybe Anyway, maybe a trainee copilot, but yeah anyway, that's cool. And also you you you learn something Independently that we are going to learn either this installment or next I can't remember where it is in the show notes because I reorganized them also I don't remember What's in which show but you're you're ahead of the class on some things as well Which you were able to learn independently and that is the whole point of this right? Because that's what real coders do in the real world. We don't know everything We just know we just have enough experience to be able to look it up and to understand the answer when we find it and to be able to use it and the fact that you independently Successfully used a very useful function is another reason your code is way shorter than mine Because you're doing it better So what was the challenge? So we wanted to transform our Nobel Prizes data set which I have many critiques of I Think it's quite dirty data. Even though it has all the facts in there I don't like how they chose to do it and I would like you to transform it into a new Jason file called Nobel Prize list dot Jason That's basically a simplified version of the data, which is really very focused on Just the list of who won and what they won So it should have just four keys for each prize the year Which prize it was or was it medicine or physics? How many winners there were and just the names of the winners of strings? So it's a much simpler data set than the data set they have which tells you what share of the prize who won and all that kind of stuff So to start building up the solution And the first thing that I figured to do was to filter our list of prizes down to just those that were Actually won because a bunch of the prizes they have entries in the original data set because They decided not to give them to people and then there's an entry in the original data set That says why they decided not to give them to people But I don't want those in the final list So I started by exploding the prizes array and then sending that into a select Where I verified that the laureates key was of type array making use of the type function To determine the type of dot laureates. So where there were laureates It would be array and where there weren't it would be undefined, but obviously only array double equals array So therefore only the prizes actually awarded would make their way through So one thing that bothers me about this is that you have to study The structure of the stuff you don't want in order to figure out like you looked at it and went hey I noticed that there's no array if there's no laureates Like you have to you have to look for what you the bad parts I guess you I guess it's normal though you go through and you try to do a query and you go Hey, what's this garbage? Oh, I don't want that garbage. What's different about it? How can I exclude it? Right because usually you'll stumble across the need to exclude it by trying to do something and ending up with error Is saying cannot index undefined or something like that and you know, why is it undefined here?  Oh, you're weird I don't want you and then you know, you write a rule to filter them out It's that's certainly how I've ended up stumbling onto these oddities in the data set, you know I get an error and then I try to figure out why and So the next thing then is to start assembling what we do want So having removed the ones we don't care about I then pipe it into a new filter where I start to build my dictionary And the easy ones to build are the year where we just basically say year colon And then I take dot year and pipe it to the two number function because I want my years as numbers not strings I would like to either do math on them properly and the You put that way into parentheses and that's a way of that doesn't change it programmatically in this case Right, it's your in this case and you did parentheses dot your pipe to number close parentheses just as a way to make it visually Separate so you could tell what you were doing Yes, and also because I don't know the rules for which doesn't doesn't have higher precedence than bracket always Okay, so you're right that in the case of comma comma would have won But I get very confused sometimes and sometimes my query is break and I just throw brackets around things I could go look up the order of precedence, but I'll just throw brackets around that to be really well It's like when you do a multiplication, right three plus two times five well two times five is going to win But if you put parentheses around it then your brain can see it Yes, that's a perfect example. You haven't Yes, yes, exactly. You've just been clear about what's real Um, the price then is just me renaming the category. So price colon dot category. That's easy And the number of winners is just the laureates array piped to the length function So that gets us three out of four straight away. No problemo The last thing then is we need to build the array of winners Which is a little bit more complicated because a we have to dive into the laureates array this time And then we have to build their names because the laureates array does not contain Their names as one piece of data it contains a first name field and or well, no always Yeah, and not and or and maybe also a surname field There's always a first name But there isn't always a surname So the easiest as a way to get close to the final answer We can say that the laureates array or say the winner's array is going to be formed by wrapping inside square brackets To reassemble an array The explosion of the laureates piped into the string interpolation Backslash roundy bracket dot first name close that one Space backslash roundy bracket dot surname close that one close the string close the array And that will give us A lot of the right answer So if we look at the prizes, we're only human beings won that is in fact the right answer So the 1903 physics prize has three winners. Henri Becquerel, Pierre Curie and Marie Curie and they look perfect But unfortunately in 1904 the peace prize was awarded to the Institute of International Law Null Wait, no Where's that null coming from? Well, that's our friend sir. Sorry surname not being populated And this is where your your result and my results differed So You checked whether or not the surname exists and then use the alternate operator to go and Do a different thing So in your case on one side of your alternate operator You were just doing why I just did their first name followed by surname and on the other side of the alternate operator You were just taking the surname. Sorry just taking the first name Right that was in the extra credit version That wasn't the extra credit version because I did a The other method to get rid of it on the first uh before going into the extra credit Okay Perfect, which gives me an opportunity to say that the way I sort of guessed people would have a go based on what we've already done in the series Is to use the alternate operator to say either stick on the surname Or empty string, which is at least nicer than null Oh, so now I didn't get that. Okay So now we have institute of international law space Because the space between first name and surname is still there, but that's better better So the last thing I did then for my simplest solution was to use the or trim st Or function we learned about for removing the equitation marks in an example in the previous installment to just pull that space off the end And then I had a working solution To get us to international institute of international law Okay, and that's full marks. So so I actually did half of each of those sort of in my solutions In mine, I just took the original solution you had that didn't work and I said trim off space null Yeah, it's actually cleverer That's yeah, I was I think I might have been looking teaching us about that that putting an empty string in there I was like, I wonder why he's telling us about that. I don't know where to use that information I Was looking for excuse to the alternate operator. Oh, okay Well, so I did the our trimster to get rid of space null in the first one, but in the second one And that's where chat gpt came in was I found that there's a select parentheses has And that means so I was able to say select has quote surname And then uh, then I said give it Uh, if so in other words if it has a surname give me first name and surname and then the uh, alternate operator wait alternate Alternate. Yeah, alternate operator and then just give it first name if that's all it had So I sort of did have actually really nice And that is a really nice solution. So my way of getting to full credit So what I said in order to get bonus credit. So this was already full credit My way of forgetting bonus credit was to never add the space then have to go and remove it And the reason your solution works is because of the power of the jq only data type empty So in javascript things are A value or they are null And because the jason language Has a concept of nullness There is n u double l is a keyword in jason and so null is supported in jq So jq needed to have like an empty or version of empty Or a more nothing version of nothing than null So jq has a language invented a new data type called empty, which means genuinely absolutely nothing Like totally absolute absence of anything in jq is empty And so the select function when you look into the documentation, it says that it either returns the thing you gave it or empty So what you are doing with your select was evaluating it down to either the value for the surname or empty And You can actually make empty yourself So there is a function in jq named empty which returns empty no matter what you do with it. It always returns nothingness So with the alternate operator You can instead of saying an empty string or whatever you can just say or empty So slash slash empty Is something else you can do and I love the documentation for the empty function So this is the full documentation from the jq official docs for the empty function Empty returns no results none at all not even no it's useful on occasion. You'll know if you need it smiley Okay, that's awesome. Well, you you referred to it in your in your hint But I couldn't tell I thought empty was a verb I thought we were going to be using it too empty Something and so I looked at that and I couldn't figure out how to use it and I spent so much time Probably of the six hours I spent working on this before I talked to you I probably spent four of it trying to figure out has no or show me the ones that aren't null And what I should have been looking for was the ones that aren't empty That is also true. Yeah. Um, yeah It turns out I don't think you can search for is null. I don't think that's a thing Maybe you could get is a memory of a query like if it returns the query null then Well, and also the type function will tell you if something is null because if you Send something null into type it will give you back the string n u double l which you could then do double equals against I couldn't double equals off that dang null that saved my life. I'll show you the I wrote down every command I wrote that didn't work and uh, my notes are 358 lines long So I try with comets It's a type function though. So you would say type double equals null not null double equals null. Anyway, um I tried my approach not equal to null and that didn't work Select out laureates dot surname type not a pipe to type not equal to null and it didn't work A possibly a bracketing issue potentially maybe the pipe isn't happening in the wrong place possibly um So my approach to never having to take away the space Was to use the fact that if you join an array if it has two elements It will put the space in and if it has one element it won't So my approach was to make an array of the names And to use the alternate operator to replace the surname with empty If there is no surname Which means that you either have a two element array Or a one element array which when you pipe them to join will give you first name space surname or first name Oh Because you also talked about the join thing. I was like, what is he joining? So that was how I ended up fixing it without using anything we hadn't mentioned before in the series So you said that first name that's her name slash slash empty. Oh, you put them right up against each other. No spaces there. Is that on purpose? Uh inconsistency on my part both are valid. I usually do space them Okay, uh, but slash slash empty. Okay dots her name or empty So in other words my array will either have one or two elements. So when I join it with a space It it will only put the space in if there's two elements And it won't join anything if it doesn't find a surname Right because I know the array Okay So it's you know, it is a solution There are in the infinity many of them yours is nicer than mine But there we are but I learned more learning yours. So that's good. That is true And yeah, so and everyone got to learn twice as much because they got both of ours. So there we go Right so moving on to today's topic then we are going to start treating jq as a true brew scripting or programming language And so the first step in doing that is to instead of saying jq space a filter space some files to go read We would like to say jq go fetch that file over there Which contains your filter and then apply it to that file over there So the first step in this process is the minus minus from minus file flag Or it's much shorter and friendlier friend minus f Which says get your filter from a file So jq space minus f space some file name dot jq space noble prices dot jason will apply the content of Something dot jq to noble prices dot jason Okay, so the the the name dot jq that's after the minus f flag Is a file that is just going to contain the query It's going to contain the filter which could be as long as you like because now it's now you're free to have Giant big filters all in that nice file okay, so There's a couple of rules in what goes into that file now that you have the luxury of having a file instead of a single string You can do things like add comments into your jq code Which is very useful and it uses the shell script style of comment So once you get to an octophthorpe symbol or a pound symbol or a hash symbol or That thing with the two lines and the other two lines whatever we're calling it today From there to the end of the line is a comment So if you put them at the start of the line the whole line is gone it's taken at the end of the line It's just up it from the end is gone So just like we've gotten used to in our shell scripting. So that's nice and easy So I like to add comment to the top of my files So you'll notice there is a file in the show notes called pbs 160 a dash one dot jq Which is actually my challenge solution for the homework just sitting in a file So it's still all on one line But it's just exactly the same string as in the sample solution But in the file instead And so when you run jq minus f pbs 160 a dash one dot jq nobleprisers.json It gives you the same answer you would have got if you'd done the giant big Whole thing from the sample solution Okay So the first thing I did to make take us to the next step was to add a comment at the top of that file So there is a dash two version of the file, which is it prettier So the first thing I did was add a comment at the top that says this JQ script refactors the Nobel prizes dataset as published by the Nobel Prize committee into a simpler form The input it would like is json as published by the Nobel committee and the output will be simplified json And so that's the structure I like to use for all of my dot jq files What what does this file expect to be given and what is this file offering you out? So input and output Um, and yeah, there are other things that I put in comments, which we will learn about later The next thing then is code layout Now that we have a separate file Can we Have stuff On more than one line, please so instead of it being this whole thing. Yes. Yeah Can you break it any old place you want? Yes, you can because the way it works is the pipe and the comma separate your filters So the pipe takes the output of one filter is the input of the next filter and the comma says do this filter And also do this filter. So those two are effectively your end of statement So it doesn't care if you put new line characters in it doesn't care if you put lots of extra tabs and spaces in So you can lay this out any which way you like because it will know that one filter ends when you meet a comma or a pipe And so it's perfectly happy This then brings us to the question of okay, so we have infinity possibilities What should we do? I check the documentation to see if there was an official style guide because in javascript There's an official style guide. Um, and which is why you can have stuff like j s lint To or j e lint es lint. Yes lint some amount of letters um Yes lint can apply the rules because someone wrote the rules So I thought maybe there's rules for jq and therefore I can do exactly the right thing and we won't have a big argument about this There are no rules so So I looked at the examples in the documentation and I noticed the pattern followed a bunch of similar languages So there are a bunch of languages out there for querying large data sets the the term of art is data lakes so When you have a giant big amount of data that's unstructured It's often called a data lake and so If you live in open source land the app that does most of the data lake stuff is called splunk Cool name for an app that we go looking for stuff you go spelunking around And splunk has a querying language called spl which uses pipes to separate the different parts of your query If you live in microsoft land the language for exploring your data lakes is called kql The custo querying language and I don't know why it's called kql, but it is And kql also uses pipes and both kql and spl always Sput pipes on their own new line So when you have one filter going into the next filter they put the pipe at the start of the line And then you can see your filters one after the other So I sort of think of like a waterfall or one filter water falls into the next one Well, they just start the next one on a new line with the pipe at the front of it to show you And I'm the next thing and that seems kind of clean because like in in markdown when you're Lining up things in tables. It's a pipe, you know, it's just sort of a clean This is a place to break that that makes sense I've been doing I kind of played around with it just when I was trying to look at my code I see what heck it was and I was doing it after the pipe, but it looks better At the pipe like the pipe starts to the new line I yeah, that seems to work best and that's what the documentation tends to do And like I say, that's what you tend to get in kql and spl So that's what I'm used to so with my work hat on I live in both open source land and microsoft land So I actually speak fluent kql and spl Which apparently makes me quite unique most people pick one. I've just ended up doing both And now I have jq on top of it. So I'm just using the same style I'm using in kql and spl in my jq and thankfully so does the documentation for jq mostly So my three rules that I'm going to follow for the show notes and they are an invitation For others to do the same or feel free to do whatever you like because there are no rules But my three rules which are sort of guidelines Is all non-trivial filters by which I mean Like technically speaking dot year pipe two number is two filters I am not going to put that on two lines because that will not make things clearer That will make things less clear So for anything where it's would where I want to break it up I will break it up with the start of the filter starts a new line If I am starting or ending a large array or a large dictionary I will put the opening square bracket or the opening earlier bracket by itself Go on to a new line and tab in and I will stay tabbed in until the end of that Dictionary or array and then close it back out on a line by itself like a cold block Now that's not exactly what you did in the example You did the pipe and then open squarely bracket for a large dictionary So it's not on its own line, but Conceptually that's part of the pipe, right? Okay, so that's you have then inside the filter you are then doing but if you look at the very very very top So the very first line of the of the example is open square bracket Yeah Which is exactly doing it. So if you take rule one and rule two you end up at the hybrid situation You see on on say pipe space open curly That's sort of joining together rules one and two And then ask a question because there's a We'll go ahead and finish and then I'll ask my question Yeah, and so basically the filter separators I'm going to put at the start of my lines So that pipe space select not Leave the pipe on the line above it, you know dot prizes pipe and then a new line select I've just chosen to put my separators at the start It's right as I said, there are my three guidelines So my question there's a difference between what was in the zip file and I can correct the zip file And what's in the show notes you've got backslashes at the end of the lines in the text Are those I do they're a leftover They are leftovers of something else that the documentation says should work But doesn't actually work in real life. And so I meant to take them out. I could take them out Looks like we're both taking them out Yeah, which would be fine because git will say us that we've done the same and I wouldn't even complain Right. So, uh, they're not in the zip file. So then that's fine Precisely Okay as a little bonus extra So now that I'm putting my k in my, um jq Code in a separate file I'm obviously going to be viewing that file in a code editor Which in my case and I think your case we have both settled on vs code as the editor we love Which is an open source editor by of all people microsoft And they named it after a product. They used to charge two arms two legs and a house vs code used to be stupendously expensive close source poop And now they took the name And applied it to an amazing open source project free microsoft No, we take the name we take the name, you know, I like so it's a completely different product It has nothing to do with the old vs code Sorry, it was used to be called visual studio and this is visual studio code. That's the difference. Anyway, it's an open source project It's gorgeous and it has a plugin architecture So there are jq related vs code plugins. There are two I recommend The first one is a simple syntax highlighter. It is the wonderfully named jq syntax highlighting so You know, I've actually been writing my jq just in cod editor Just as plain text files and it would have been a lot easier doing it I use vs code for all my javascript and html and everything else. I would have done it over there if I'd know this existed very cool So there we go. That's the first one the second one. I have also installed because what it lets you do Is type jq into I don't know how much you make use of vs codes command palette But if you hit shift command p on the mac you get like a terminal window for vs code It's where you can type commands to vs code And if you install the plugin vs code dash jq You can type jq commands into the vs code Terminally thing while you have json files open So if you have a json file open you can just hit that key code type jq and then type a filter And it will show you the output of the filter in the output pane in vs code So you can use your jq knowledge to search json files that you just have open So you could be writing code in any language and you just quickly want to check for something in a json file You can just type some jq straight into vs code and it will know what you mean So like you can search with regular expressions, you can now search with jq if you install this plugin So that's really cool You know i've had trouble figuring out what to do with that Command palette and maybe that'll force me to try using it I do know you can open a plain old regular terminal right at the bottom and that's kind of fun That is very fun. You can have multiple of them and you can split them as well Yeah, I got real confused when it started when I told it to do that. I didn't know that's what I was going to do It's like oh, no there's too many of them You can have a lot so I used to have a bash one and a zsh one whenever I was checking the differences between bash and zsh I'd have them both open at the bottom of vs code And so like check while I was writing those show notes Yeah, one thing that I don't think you mentioned or I missed it is at the bottom of at the end of both of your jq files that you're using as the input here. It says pipe at json Did you explain that? So the I didn't really so the challenge said that it needed to output the results as a one-line json data structure So in the format you would expect it to come from an api so That's what the act so it's the formatting string. So last time we did at csv to format a csv so at json formats as json So it's a jq It's a jq filter That's formatted as json Yes, and now I say formatted as json So You will notice that the noble laureates file when you open it in a plain text editor is all on one line with no spacing That is the most efficient way to send json data across the internet. Oh, yeah, yeah Yeah, so jq by default gives it to your pretty If you pipe it through the at json Formatter you will get it in that Compacted efficient format which is used to store and transmit json. Okay In the real world. Uh-huh. Yeah, that makes sense. That's all that's doing. Yeah, now. I recognize it a bunch of glop I can't read because it's all on the same line There you go. Yes, which is why we use visual code It's sort of like vs code to pretty printer json as well because if you open a json file in vs code You can ask it to format it sensibly for you Oh, that's another reason to play with it. Okay. Yeah So that is another so that is the first thing we want to do With programming in jq is to have our stuff in separate files. That is already powerful The second programmers trick I would like to bring you is debugging so I always tell you to visually Imagine the shape of the data that goes into one filter And then what you would like it to be like when it comes out of the next filter And that will be the input to the filter after and so in your mind you're watching the data transform But if you're if your Assumption about how your filter works Does not match reality Your mental model is getting runger and runger and runger as you go from filter to filter to filter, right? So I have told you in the past to build them up slowly by deleting everything after The point in time in the filter and seeing what it looks like Wouldn't it be nice to be able to put a probe Into that point in the filter and just see what you have Yeah, so you're an engineer. So you remember those electrical probes You could stick on various bits of a circuit board to see what's going on there There exists a jq function Which takes as its input anything Its output is always identically the same as what you gave it as an input But what it does is write whatever you gave it a standard error So it makes no change to the data. It just outputs it to scd error So you can stick that filter anywhere in your chain and it will have no effect Except to show you what the data looks like at that point in your chain So it's a probe you stick into your query And it will appear on standard error as square bracket debug And then some sort of value close the square bracket And if you send debug more than one thing, I think you'll get more than one thing here in the comma So to see what's going on Let's say for example, we we're going back to our We're messing around with that quick that taming the terminal covered this if people don't remember My apologies. That's in my show notes So I call this kind of a thing where you're messing around with standard input and standard error and all that They're called streams on the terminal and you can mess with the streams, which I call terminal plumbing That is my own creation rather than official documentation But using the terminal Exactly taming the terminal 15 and 16 are called plumbing and crossing the streams I may have been watching too much ghostbusters Um And they are episodes 15 and 16 and they talk all about the existence of standard in Standard out and standard error and how you manipulate them. And so that is over on taming the terminal So we should say that the reason it's good that it's written to standard error Is because jq's normal output is on standard out So your debugging isn't going to muck up your data So if you're writing some jq to take one piece of json and turn it into another piece of json It's coming from standard in to standard out But your debug is on standard error Now by default they're both on your terminal, but you could pipe standard out to a file Or to another terminal command And standard error will still go to your screen Or you can pipe standard error to a different file your error file or whatever So you can poke and prod without mucking up your output So that's why it's nice that it's writing it to the other stream. They're not on the same stream That's the important thing you're going to remind us in using this debug tool how to go look at standard error. I hope Well by default, you're just going to see it right because if you do nothing standard error comes to standard comes to the terminal window So by default, I'm just going to see it. Okay, which is nice So we're going to play around with an old jq example Where we are going to look at the command we built in the previous installment to render the details of the noble prize for a friend of the show and noble prize winner and dr. Andrea Gez So the filter basically starts with all of the prizes explodes the laureates array Finds the one with the surname Gez And then it built a new string using string interpolation to basically say first name surname was awarded her prize for motivation But I'm just going to stick a debug into the middle of the stream here So after the select for dot surname double equals Gez. I'm just saying pipe debug pipe So I just literally stuck it into, you know, we would have piped straight to the next filter Which does the string interpolation, but instead I'm just going to interject with a debug and when we do that We see debug id 990 first name Andrea surname Gez motivation for the discovery of supermassive blah blah blah blah share for So that is the content of Andrea's dictionary as it exists just before we do the string interpolation So that tells us all the keys that are available and what their values are Which helps us to do the string interpolation more cleverly if we were to do it again So I think it's actually oh, yeah, yeah, sorry, it's a dictionary, but it's put inside of an array called debug Or right because that's the format debug uses. Yes So that's just the way you've gotten into the right one You're where you think you are You haven't and what the keys are called Ah, ah, yeah. Yeah Can you end it with a vein? Oh, sure. You can put them anywhere Okay, it will show you whatever is currently there. So you keep just keep slide that along. I could have used this last week, Bart I know that's one of the reasons I that's actually our play date was when I decided to rearrange all of my plans It's like no, we need to we need to move this forward. Okay. I like it So because both standard out and standard error go to the terminal by default the default output there shows us the debug followed by Andrea guess was a word at her prize for the discovery of whatever right? So it shows us the result of the jq Filters and the debug all together on the terminal because both standard out and standard error connect to the terminal But if we redirect standard out to a file called say citation dot txt Then what we will see is that the output on screen is only the debug statement and citation dot txt contains Only the output from the jq. So they are on separate streams. So we they don't have to be mixed Which is the key point right, right So that is, you know, just highlighting the the importance of the terminally bit there so There are lots of cool things you can do so On you know, it is great that We can just put debug with no arguments and it will show us the value of dot effectively But we can actually do anything we like we can pass the debug command An argument that is a string and it will print us out that string instead of the value of dot so We can build our own debug messages Which can use string interpolation to include other useful things And one of the most useful things is the function named keys Which when you give it a dictionary Gives you an array of all of the keys that exist in that dictionary So we can say debug open bracket and then the string we have the following keys colon Then backslash round you bracket dot pipe keys close our round you bracket and that will Pipe the current value of whatever it is, you know, wherever we are in the chain to the keys function And stick all of that into our string. So now when you run the command It will say Debug we have the following keys first name id motivation share surname Lot shorter to read It's not filling it up with all of the english and stuff. It's only telling us the keys So you can imagine if your dictionary was really complicated Which had like maybe one of the keys had like an essay in it You don't want it to show you the full dictionary, right? Just show me the keys, please So the keys function is fantastically useful Another one I like to do Yeah, the um the keys as it gives it to us Looks like it's almost in string interpolation format. It's quote its backslash quote first name backslash quote Why does it look like that? Because that's how jq decided to implement it. Okay. What is the backslash? So the backslash is is Uh escape so technically speaking Yeah, so technically speaking its debug and its value is a string And that string contains an array of strings. So it's gone. So this is a valid string We have the following keys first name surname whatever But because it's the string that contains quotes it escapes them to make it a valid string Oh, I see what you're saying. Well, this is back to why did this? Okay. Yeah, good. No, I like it Gotcha. Yeah And actually before I jump on to what I thought I was going to say next I'm reminded that there is another function that the documentation mentions that I think is good to mention is that you Sometimes you take one file and shove it into jq But you can take as many files as you like right because jq says my first argument is my filter Unless you're using minus f in which case it's a file name And then I can have a second argument is my first data file. My third argument is my second data file I can have infinity many data files So if you're working with multiple data files in a single jq command It might be nice to know which file the piece of data you're debugging is in All right, am I looking at a piece of data from file number one or file number 50? Like where is this piece of data that's causing me an error? Where is my problem? Basically, so there is a function called input underscore file name Which will tell you the name of the file jq is currently processing So if you include that in your debug statement, then you know where you are So as an example There are two json files in the zip file ip-bart b.json and ip-podfeed.json We used them a few installments ago. They just contain a little bit of information about the ip addresses for bar to b.ie and podfeed.com And if we give both of those to the jq command, we can see what input underscore file name does So my jq command is debug processing the file backslash input underscore file name close backslash comma ip is backslash dot ip address So basically it's a string interpolation that shows input file name and the value of the ip address key And the input to that jq command is ip-star dot json So the terminal will expand that out to be both of those files Now because I'm Because I'm only interested in seeing the debug statement and I don't want my screen cluttered with the actual answer to the jq I am using terminal plumbing to send the standard out to dev null, which is the computer's black hole So the only output we're going to see is the debug statements, which is just for our convenience here because that's what we're interested in And what you will see on debug is processing file ip-bart b.json ip is 37 139 blah blah blah processing file ip-podfeed.json ip is 104 21 34 So it's successfully telling us which file we're currently working our way through Which could be very useful if you're working with a big folder of data I can barely keep track of where my problems are in one file So if it's multiple files, and if they're formatted, you know once formatted incorrectly or something Yeah, right.  So the you know the input underscore file name is definitely very useful So there are actually quite a few functions that are useful for exploring data structures So I love using the length function in my debug statements because if you give the length of something Then you shove it through a select to filter it down and then you do the length again You would hope that you have a reasonable difference So if I know that there are about 50 valid things in my data set and I start off with 500 and then I have 500 left as I go that select statement didn't work Whereas if I'm down to 50 it's like yay that select statement is reasonable That's what I hoped you know or there should be half of these or whatever So just think when I was doing the the uh is not equal to null Then I if it kept being the same number it's like well, I haven't found it yet That's I would know that it was still not I mean I knew it wasn't working But I didn't know where it was working all the time Yeah, so I do that a lot give me the length of the array do something give me the length of the array again And then see if what I'm hoping to happen has happened Another useful one if you want to just sample a piece of data So you might have a data set with 5 000 records And if you debug the whole data set you're lost you're just in a sea of data So the functions first and last Take an array And give you the first or the last element depending on which function you call So if you have 500 lariates and you just want to see what a lariate looks like just You know lariates pipe first So didn't we learn that we could do it with uh zero and minus one? Or minus or was it minus one? Minus one as an array index you you absolutely can do it with an array index But sometimes it's nicer just to have something really Englishy. Yeah, you just have debug first debug last Yeah, the other one is Filth is limit which takes two arguments, which is a number of answers you want a maximum number of answers and a filter to go and make you some answers and so Limit so you could say limit five and then explode the array and then it will give you the first five elements of the array But it could be limit five do a select And then you could see okay. Well, this select is only giving me things that look this shape Right, you may not want a thousand of them, but you might want 10 Well, I would have liked that a lot in this as I started writing it out to a file because I couldn't see You know, it's just this sea of data coming out of the the right. Exactly. That would have been nice. Yeah yeah keys we've already mentioned and Has Which is the one you found with the help of autopilot so has and then takes us an argument a key name Will return true if the dictionary does have that key or false if the dictionary doesn't have that key Which is very useful in select statements But also in a debug statement because if you're saying well I'm pretty darn sure all of these things should have a citation or whatever And then you just pipe it, you know has whatever you're looking for and if you see true true true false true true It's a whoa My data set has a mess in it. Why is there a false in here? And then you can do a select where whatever equals, you know, it's false or whatever and have a look at it But just being able to see a sea of truths Versus a sea of falses can be very useful. So it has key name. I use a lot as well A structure I will often do for myself as I was saying is to Do a length before and after A select statement now that does mean that I need to turn it into an array Before and after because the length function expects to be handed a whole array So if we say prizes pipe it to debug length that tells us how many prizes there were to start with We haven't exploded them yet, right? It's just prizes to length Then we pipe it into Another filter which has open square bracket as its very first thing Then inside that array we explode So dot and then explode it we pipe the exploded content to select And then we close our array again So what we have now done is we have built a new array which only contains the things that match the select But it's a whole array we haven't we've exploded it but contained the explosion So we started with an array and we've ended with an array We don't have individual pieces. We actually have an array again Which is something we weren't able to do until the previous installment And up until last time when we learned about the square brackets in tax Once we exploded an array there was never any way to put Humpty Dumpty back together again Which meant we went out of our way to not explode things we needed to keep in one piece Well, this is how you reassemble Humpty Dumpty. You just put it all in square brackets So if we do that to debug length explode select recollect debug length What you will see is that the amount of Nobel Prizes that happened before 1950 We go from 670 total Nobel Prizes to 245 that were before 1950 Okay. Yeah. Yeah. Yeah So very useful nice as I say if I'm If I'm working with a big data set I like to use First and last just to see first and last are useful because if you're dealing with edge conditions If the first one is outside of what you wanted, that's actually going to be very obvious, right? So if you want stuff less than or equal to 2000 and the first one is 1999 say oh, that's an age problem Or if you wanted, you know less than but not equal to 2000 and the first the last one is 2000 Well, that's an age problem. So first and last are great for looking for age cases Just to verify for yourself that you really are bounding things correctly Like it should have been a less than or equal to but you used a less than or something Yeah and Also, you can debug So if you have if the filter that you pass contains and and also operator Then the functions the comma. Yeah Yeah, so it'll do it twice basically inside your debug And we can use the limit to give us a bigger sample. So we can say debug limit Five semicolon to separate our arguments and then explode the prices and then we will be debugging just five prices Okay, let me think so What you were talking about the and also operator the comma, but you didn't use a comment in that one Uh, yes, I did debug first comma last So when I in my example of using first and last I actually use them together So I say debug first comma last Okay, sorry show the output that one was long of a scrolled over. I did I couldn't see it. Okay, so Okay, okay got you Yeah, and as I say that gives us the bounds right the first on the last. Okay, that's good So we see those twice um When working with dictionaries, I like to use the keys function So if you want to get the keys for the price of the dictionary we can say Debug first pipe keys Right, so we take our prizes array. We take the first one and then give me the keys of it Oh, I like that one a lot as well that I could have used that a whole lot sooner Right that's this whole installment. I was thinking gosh, we should we could have used this a whole lot sooner Yeah, but but I wouldn't have known to care Right, right. I wouldn't be as excited about this because I wouldn't have been going I gotta open it up again. Let me look up You know open the whole file and look through it and read it and this is much better Yeah, and as you've already discovered we can use has to tell us whether or not something has a key But a very powerful thing So we've already met the all function which will return true if all of its arguments evaluate true So you can use all In conjunction with has to make sure that every dictionary Has a laureate's array Interesting. Yeah, so instead of getting a whole list of true false true false true false, it'll by the be all True, right? Hopefully or if it's all false. They go, there's dirty data here At least one of these state of things in my data set is not like the others I got what's the word? Sing it. Sing it Bert. One of these things is not like the other one So At this stage we're doing pretty well for ourselves So we now have our jq filter as multiple lines nicely laid out so the more complicated it gets we don't get lost We have the ability to add a little probe into the various points in our ever more complicated jq filter Right because they're growing here and they're not going to get smaller when we learn more So we can lay them out so we don't confuse ourselves and we can probe them with debug If you put it in a file Wouldn't it be great to be able to give it some arguments into the file? It's like if i'm searching for Andrea Gez's Nobel Prize How different is the logic to find Marie Curie's Nobel Prize? It's identical logic Just somewhere in my query Is going to be the string ghez instead of cu or ie or whatever it's with curie So wouldn't it be great to be able to pass in some parameters Into our jq filters that are now nicely separated out as separate files. And the nice thing is you can So this is going to be our first Encounter with a much bigger topic. We will come back to in a few installments, which is variables in jq So this is a use of variables in jq There are many other uses of variables in jq But jq is an interesting language because the jq documentation makes it very clear that Unlike in other programming languages where the first thing you learn is variables And the most fundamental thing is variables in jq variables are considered an advanced feature and discouraged unless you're doing something Suitably advanced and i'm going to quote you from the documentation because I have discovered something while writing these show notes the documentation for specific functions is quite I would say not beginner friendly is me being polite But the documentation explaining the philosophy of jq is actually very good So this is what the documentation tells you about jq's approach to variables Variables are an absolute necessity in most programming languages But they're relegated to an advanced feature in jq In most languages variables are the only means of passing around data If you calculate a value and you want to use it more than once you'll need to store it in a variable In jq all filters have as have an input and an output So manual plumbing is not necessary to pass a value from one part of a program to the next Many expressions for instance a plus b Pass their input to two distinct sub expressions here a and b are both past the same input So variables aren't usually necessary to use a value twice For instance Calculating the average of an array of numbers requires a few variables in most languages At least one to hold the array Perhaps one for each element of the loop counter In jq it's simply add slash length Now add is a function that takes as its input an array and adds all the elements Length is a function that takes as its input an array and tells you its length If you add all the elements and divide it by the length you get an average So that is the full jq filter for averaging an array add slash length Wow No variables right absolutely no need for any sort of variables or loops And that is that is jq the jq documentation on loops also says Don't use these most of the time you won't need it Do you know who wrote this this was the team that finally after a year and a half of arguing together And there were two camps You know sally wanted variables and and joe didn't and they went back and forth back and forth and finally Joe was like okay fine, but we're going to make them feel stupid if they break down and use them Make them feel lesser if like you're you're you're not you're not understanding the spiritual Philosophy if you break down and use it that means there's something you didn't understand It's a sign of weakness What's funny you say that because the closing sentence is definitely that sentiment So there's generally a cleaner way to solve most problems in jq than defining variables In other words, if you're defining a variable double check that you're not doing things the hard way because you might well be doing things the hard way Still Sometimes they do make things easier Fine so You can build variables in the body of your filter But that's what the documentation has just told us Is usually not necessary. We're not going to do that yet There is a way to do it. There's a whole operator for it. It's called the as operator We will meet it in our very very last jq installment I have saved it to the end in keeping with the spirit of the documentation My last installment will be called advanced jq and we will cover variables in advanced jq But that's good because we won't we won't grow to lean on them Precisely, and I don't want people to because there's usually a better way Add slash length right every time you're thinking I need a variable think add slash length thing. Well, maybe I don't But a really useful place for them is to take arguments from the jq terminal command To pass a value to a jq filter in the jq language And there are flags on the terminal That allow you to specify variable names and values that you can use in the filter Hmm So I'm going to give you a practical example because that sounds like a word salad Um, what I do also need to tell you is that inside jq. So when you're in the jq language So inside your dot jq text file All variables their names are prefixed with the dollar symbol. That's how jq knows it's a variable Function names are just bare names Variables are dollar something So it's always dollar. So it's not x. It's dollar x And now I'm going to confuse you Because in the terminal The dollar symbol has meaning So it would be very awkward if the jq terminal command Made you put the dollar in because then the terminal would think the dollar was meant for it Not and they would just break everything So the jq terminal command does not use the dollar But when the variable appears inside your jq file, it will have its dollar And it's a case of Don't break things in the terminal to fix things in jq Do the right thing in both places Okay, it does it's it's confusing but sensible So there are actually four ways of passing arguments into The jq so from the jq command into a dot jq file and we're going to intentionally ignore two of them So There are flags called minus rgs and minus json rgs and they will produce Positional arguments they're called and they're really awkward to get to because they don't go into your code as like dollar one or something They go into your code as dollar rgs in all caps dot positional open square bracket their index So the first positional arg appears in your code as dollar rgs dot positionals open square bracket zero The second one appears as dollar rgs dot positional square bracket one that is horrible So we are going to use named arguments for our own sanity So we're going to do them using minus minus arg and minus minus arg json and they're going to let us make a named variable These guys are a little weird We're used to terminal to terminal commands having options that have minus minus something space one value This is minus minus something space One value space another value The first value is the name the second value is the value So to pass an argument named x you say minus minus arg space x space the value for x Okay, we're we're getting real abstract here and what we're talking about. I know we're I am I am seconds away from pulling this in But I got to tell you I got to tell you it and then I got to show you it I'm a little closer. Okay. Keep going.  So The two arguments are minus minus arg space namespace A string value, which means it will appear in your jq code as a string So of type string The other thing you can do is you can pass it in a data structure as json So minus minus arg json space name space Needs to be followed by some json So if you want to pass an array you would use minus minus arg json Because that way you can specify an array or a dictionary or whatever you'd like So that's the difference in minus minus arg and minus minus arg json So What if we wanted to pass a variable named dessert? With the value waffles Because that's the kind of thing we do here so We can say jq now i'm using minus n to say don't wait for any input, right?  I like waffles So we didn't give jq a file And we didn't give it all we gave it was arguments. There's nothing for it to be querying Right, I said no input Yeah, I know I know you said that but what is it if you if you're using jq you you're not querying a json file You wouldn't query doing a debug to show you nothing. I'm just doing a debug to show you the variable working I tried to simplify it and I think i've confused you trying to make the example simple Well, good. I just didn't know you could use jq with no input But we've done that repeatedly had when we were demoing the math or did I reorganize that into next week? I may have reorganized it into next week I don't remember the order of my own show notes anymore. Do you know about the post operator? Uh, don't think so. I don't think but it's possible. Remember now we're working with my memory. So that's not a good test Barton Okay, so anyway, so I we're able to just not pass it jq We're not telling it to query anything And we haven't written a filter We have The first argument is the filter debug open round bracket quote. That's the filter. So debug is a filter. Okay Well debug is a function and this filter consists of a call to the function debug if we're going to be really persnickety about it Okay. All right So this shows this demo we're able to use dollar dessert inside the jq But you use dessert when you're talking to terminal without a dollar There we go. They are the two takeaways We have made a variable named dessert with the value waffles On the terminal side no dollar on the jq side. It has a dollar. All right So If we tried to so minus minus arg always makes a string so if I say minus minus arg n 42 and then I say Dollar n is and then backs us in is it greater than 100? And then I do a string interpolation dollar n greater than 100 The output will be true 42 is greater than 100 What Because it's f it's it's the string 42 Precisely it's the numbers come after letters in alphabetical order. Is that right? No four comes after one in alphabetical order So the string back comes after the string a a a a the string 42 Comes after the string one zero zero zero. Okay, if you're comparing it to numbers, okay If you alphabetically compare numbers 42 is earlier in the alphabet than 100 In the same way that a a a a Well, because 42 is after 100 so greater than yes, it is you just said it was before for it before 100 alphabetically All right, okay. I'm sorry. I am dyslexic. I made a mess of that anyway, or I heard a string comparison Okay, I know you're trying to say that before but it still hurts my head Right alphabetically, it's the wrong way around Yeah, because you shouldn't compare numbers alphabetically. That's why I'm so cranky that the Nobel Prize people put the years in as Strings as the numbers. Well, we learned to number exactly So how do we get numbers in as arguments and the answer is we tell Jq that we should treat four two as a piece of jason Not as a string by using minus minus arg jason Then 42 will go in as 42 Which means is 42 greater than 100 becomes false as it bloody well should In your show notes, you say alphabetical alphabetically quote 42 unquote sorts after quote 100 unquote should that second that hundred be without quotes around it Well, no because it's doing a string comparison. So there are both strings. That's what's actually happening there and Quote 42 unquote sorts after quote one zero zero unquote. Yes 42 as a string sorts after 100 as a string Even though numerically it's before No, okay I don't see why see is after a 42 is after 100 Okay I have to think about why I can't see it. Okay. I thought I understood that one before In your dictionary does the three letter word Bad come before or after the one letter word Do you're saying sorry the two letter word do bad would be before do 100 is before 42 No, but bad is before due because it starts with a b not a d not because of the number 100 is before 42 100 is before 42 Because it starts with a one even though it's got three digits. That doesn't matter. It starts with a one Okay, okay there. I saw it Not that that was that that wasn't interesting to anybody but me. Okay. All right. So now when we uh, if you give it r json n 42 You say our mason allows it to be a number Correct. So the the variable named n is now coming in as the jason 4 2 Which is the number 42 because in jason syntax 4 2 is a number not a string So then when we do a comparison it is a numeric comparison now one very last thing What if you want to have a variable that you don't always want to have So what if you want to be able to pass a variable sometimes? And you'll see why in the worked example If you just try to do it And you can't basically you try as you might you can't say if dollar my variable name is null It will just break if you if you ever ever ever use a variable name and it doesn't exist It's happening at compile time It explodes your script So you can't have an optional variable name using The variable's full name. It's really annoying. I tried every which way from sunday. It doesn't work The only way to have optional variable names is to use the variable's long name So when I make a variable named x it appears as dollar x which is nice and short It also appears as dollar args dot named x So dollar args dot named is a dictionary No dot x Okay, you didn't say the dot that's what it's looking for dollar args dot name dot x Exactly. So dollar arg names is a dictionary where all of your named arguments go And so dollar args is the variable which always exists So then the question becomes does or does it not have a particular key which the has function will tell us So an optional variable is going to be dollar args dot named Dot name of key. So you can say has dollar args dot named and then your name of your key So that does allow you to have optional variables with just long syntax But it works and you can use the alternate operator to get a default value So if you say dollar args dot named dot sum args slash slash sum default That will always evaluate you either the argument the user passed or your default And so then you can use optional variables inside your jq. So I've now been very very abstract The whole thing I've been getting you to is searching for laureates by name So let's write a jq script that we can search not just for friends of the show and re doctor and re guess But anyone's noble price So we already have our code For searching for a particular person So as a starting point we have that code which is in the file pbs 160 b dash zero which search for marry curie in this case by saying take the prizes explode them Select where any of the laureates have either a first name or a surname that contains curie So we basically say first name surname stick them into one string and then check if it contains curie Does that make sense? Yes So that code allows us to search for curie We can run that script So we can say jq minus f pbs 160 b zero noble prices dot jason And that will tell us that marry curie won three prizes 1935 and 1911 for physics and 1903 for chemistry So let us replace the string curie With our variable which i'm going to name dollar search So pbs 160 b dash one is identical to the other file But the contains instead of saying contains the string curie now says contains search dollar dollar search So to use this we would now say jq minus f pbs 160 b dash one dot jq minus arg search minus minus curie Thank you that is very important So that will work otherwise Minus minus arg space search space curie space noble prices dot jason So that will now run our script Geez, that's gonna drive me crazy I know I did warn you So that will find exactly the same answers Yay Now what happens if we try to use our script with the search curie with a lowercase c Yet no results boo This gives me the opportunity to highlight a very important programming trick and to teach you a new function The function ascii downcase will convert a string to lowercase If you ascii downcase both sides of a contains operator Then it will do a case in sensitive search Both sides of it you have to pipe right Ascii downcase pipe to your contains pipe to ascii downcase So ascii downcase pipe to contains and inside the contains dollar search pipe ascii downcase Oh, so you're you're you're Ascii downcasing the input and you're ascii downcasing what you're searching Bing bing bing. Okay much better said than I did. Okay Yeah, so if you do if you convert everything to lowercase and then do your contains It will find it regardless of case Which you can prove to yourself by running the dash 2 version of the file and searching for curie with a lowercase c And it works Okay, so we have successfully made a script that takes an argument named search So we can search for any noble laureate once we have figured out how to search for one noble laureate Which is an example of why you want our why you want variables So your challenge now So Yes, selly was right when joe didn't want to have variables Exactly. So when you break your code out, you do want variables So your optional challenge should you choose to accept it is to take my example of searching for noble laureates by name And make it a bit cleverer by Optionally having a minimum year and a maximum year And if they don't provide a minimum year then it should be all prizes from the start Until perhaps a maximum year And if they don't provide a maximum year, it should be all prizes from either the minimum year or the beginning So you should be able to find all of the curies before 1911 Or all of the curies between 1901 and 1912 All right, so basically A minimum and or a maximum year Both optional, but you always have to give it a search string Okay I think I understand that And actually I've just realized I didn't make it. Okay. So actually for full credit You don't have to make them optional for full credit You just have to make it always work with search min and max and for bonus credit make min and max optional. Sorry I forgot to say that Oh wait, so the isn't the first And just don't do min and max that would make them optional No, no, sorry. I I misspoke. So the first the actual challenge for full marks is just to make it work where there are always three arguments So don't worry about making them optional. Just solve the problem of there will always be a search string And a minimum year and a maximum year and make it work that way And then if you can make it more flexible And make the last two optional So for full credit, you don't have to do the the difficult part of making them optional. I misspoke there. Okay Got you Boy, this was a lot This was but On the one hand, it's only three things But it's three things that enable us to write arbitrarily complex jq without killing ourselves without driving ourselves insane So we can now put our jq in a separate file Lay it out. However we like Stick in little probes to tell us what's going on where And pass in arguments There are the three things we've learned today, but they allow us to script with jq Okay, okay Um, did did you explain what the syntax should be for more than one variable? So minus minus arg space name a variable space value can be used as often as you like You just do it again. Just minus minus do it again. Just minus five. Okay Yeah, it's weird Sorry, I know I skipped saying it because I was going to say it's weird for two reasons and I told you one reason And then I carried on so the first thing that makes it weird is that it's minus minus value value Normally it's minus minus one thing But in this case, it's minus minus one thing minus and then another thing And also normally you have one minus f or one minus whatever But here you can have as many minus minus args as you like which is weird. So it okay. It is weird Yeah, that is a very strange way to do it Okay I think I understood all of this got a little stuck in the alphabetical order numbers, but I think I pulled it off Which is not technically jq related That is that the lexical sorting holds true in all programming languages and confuses all programmers forever It is a fun way to make first year programmers and see us 100s heads explode So I think I finally got it. It's because the four comes after the one Didn't have anything to do with how many of them it was. Okay Yeah, it just looks so wrong. Your brain is just like no How how how but it's because you're doing is because you're treating them as letters Not as as numbers great great lexically All right Good is going to do it for now And then next time we're going to get to learn about all the cool functions Which is going to make our jq filters even more powerful where we get to do really fun stuff like De-duplicating arrays and sorting arrays and lots of fun stuff that jq can do But that is all coming up next time and until then happy computing If you learn as much from bard each week as I do I'd like you to go over to lex-talk.ie and press one of the buttons over there to help support him He does 98 of the work here I'm just the stooge that listens to him and asks the dumb questions If you go over to lex-talk.ie you can support him on patreon You can donate via paypal or you can use one of his referral links I really hope you'll go over and help him out In the meantime, you can contact me at pod feed or check out all of the shows we do over there over at pod feed dot com Thanks for listening and stay subscribed


Alrighty, I wanted to post a quick update regarding the David and Dana rig and that's regarding the textures because one of the things that happened when I posted this I got some very helpful comments mentioning that they're actually three textures and also you dim sets. I don't even know what you dim sets are.  And even though I thought I was clever by reading the read me text, which has all kinds of information regarding the rig and updates, you can see this all the way there and disclaimers. I did not go through the FAQ because clearly I am not the smartest cookie. So in here you have an extra view what the partial and full rig is, which in this case it's for different clothing and simulations. I thought this was for a lower res version if you don't want to face because the David rig didn't have any facial features. So this is added information and here we have extra added information regarding the textures.  So when you check out the Dana skin underscore Matt in here, you click on the color input and you browse for the Dana diffuse one one PNG, which I did, which gives you this. But on the UV tiling mode, you click UDIM Mari. Again, I need to check what that means. I have no idea.  They are in high quality. You can change that if you want. And you also have Dana shoes that you want to load. So it looks like this. That absolutely missed. I thought that Maya was clear enough that when I load one texture that it would load the whole thing. But then again, this doesn't mean that I shouldn't double check and look at the actual image on Gumroad to see the actual look of it. So what I'm going to do tonight or tomorrow is read through all of this. Because since the textures are different, you have specific textures. Apparently there's a cheek blush that is set up on here that you don't want on arms or whatever you will have the skin visible. But here you go. I just wanted to re-record this and follow up on that. Just in case you follow my severely ignorant step walkthrough of how to apply the textures. And I will be more careful going through all the folders. Now the full is the same for David, but it shows the same setup for her. So double check his texture setup when you load that rig. And that's it for me. Just a quick update on this ring.


Now if you look in that picture on there and you might have to look close depending on the connection and you know How good this video ends up being but you'll notice one two three four rocks that are painted red I want you to think about that for a second. This is a real picture by the way. This is not made up This isn't Photoshop And I will tell you that this is taken in the eastern provinces in Afghanistan So you may know what those rocks are saying Because they're it's a rule The red rocks are a rule for mines It's like don't walk here. That's kind of what the rule is not interestingly enough You can't really see it, but there's actually writing on those rocks So the rule is actually written verbally here as well And it's written in Dari and it's written in posh to and no one writes in English for whatever reason but probably because there's Not many native English speakers living around those parts But what they do is when when you have a minefield or when you have a field that hasn't been de-mined All right, and you got to remember that Afghanistan was so heavily mixed with the heavy It's the heaviest Mind country in the world and and the de-mining there is still gonna take like another five or ten years And it's been going on for ten years. Okay, so Everywhere you walk especially when you're out in the sticks around the boonies It's there's guaranteed to be mines unless it's been de-mined So what they do is that the the de-mining folks will they'll come through and they'll clear the sides of roads Right, so they'll they'll clear the first 50 meters on either side of a road And that's kind of the low-hanging fruit and it keeps a lot of people safe It's not many folks are gonna be out walking through the countryside But then they put these red rocks for areas where there are likely mines where they've found mines nearby But they have not de-mined this area. So it's basically a rule saying don't go in here And if you go in here, you may die, right? No, some people go well, no, it's a discriminative stimulus and it's really not and the reason it's not is because of it was an SD And I guess it could be in this case but Then it would signal that you're gonna get reinforced for something and it is kind of that because there's a negative Reinforcement of avoiding the field and avoiding the death and all that stuff But that all implies that it's gonna have to happen multiple times and the probabilities and death is like less than one shot deal Okay, so they really are more of a rule per se So I've always talked about how rules need to be verbal Here's one case where they actually don't where that signal is telling you something else But in order for that to work in order for those red rocks to work I had to tell you this story Because if you're just cruising down a street in Afghanistan, you know, you're heading over to the local province or something like that Or actually like where this picture is head not to not Nangahar, yeah, Nangahar University. So heading out east To Jalalabad and this is right alongside one of the major highways You wouldn't know what those are you'd need somebody to explain it to you and once they explain it to you like, okay I can't hit not going anywhere near the red rocks well, this is More it's frustratingly common to see these red rocks over there. In fact, there's even some out by the golf course There's another rule that exists over there. That's just kind of funny is that if you see What is it You see sheep droppings in the rocks, right? So if you've got these red rocks, but on the other side, there's sheep droppings like in the background here There's a bunch of sheep droppings. You're okay You can ignore the red rocks because the sheep would have blown up any of the mines. Anyway, that's kind of one of those crazy things I'm gonna take a pause here for a moment Okay, so let's just keep cruising forward now that we know all about sheep poop and red rocks in Afghanistan, right? I Don't like what your book talks about in one sense in another sense. It's nice. It's great, but your book use the term Rule governed behavior. Okay, that implies that the rule actually has some Power over you is this governing power that's going to just like punish you and tax you and all these different things right so in other words, you know joking aside, it's saying that The rule has the strength and the rule is what is controlling your behavior, right? In reality, it's not Rule governed behavior is better described as rule following behavior And the reason that I say that and the reason that the handful of us not well more than a handful of us a majority of None of us behavior analysts now usually use the term rule following behavior Is because we know that rule following is a behavior that by itself can be reinforced or punished You can get reinforced for following the rules you can actually get punished for following the rules Imagine that if you if you follow the rules and for whatever reason the rules didn't work for you And you still ran into a problem or something like that Then guess what rule following behavior is going to decrease so rule following Is what we would all tend to use now? I'll bounce back and forth in the slides and I know there's a couple of slides that say rule governed behavior I slipped back into that because that's the way I was originally trained and because that is Well know what we're talking about when we say rule governed behavior. It's because we're setting up a scenario that we'll talk about in a little bit, but Just keep in mind that I prefer the term rule following behavior And it gets you to focus on the idea that is in this point Which is that the behavior itself of following rules is usually well developed as you grow older, right? But it's something that you learn throughout life Ultimately what happens with following the rules is it gives you access to more reinforcers And it allows you to avoid punishers if you follow the rules on the road All right, you're likely to avoid punishment when you avoid punishment Then you are getting negatively reinforced, right? You're avoiding some stimulus That's probably aversive like a ticket or an accident or something like that When you listen to what somebody else is telling you In terms of somebody trying to teach you a skill and say here's a rule for you and Follow that rule and things will work Okay, and that tends to be the case like apprentices have to follow a lot of rules and they follow the rules because they they don't want to have to Learn the hard way so to speak so by following the rules you get in contact with those reinforcers much quicker And we'll talk about some of those clear rules for the consequences are always best Don't hand plane against the grain You guys are going what the hell is he talking about now not point simple You know hand planes and the idea is you're trying to smooth out a piece of wood Well, you have to read the edge of the wood in the direction that the grains going so assuming you're gonna go with the grain always We can never really go cross grain. There's no reason to do that. So you're going with the grain No, there's still another direction that the grain is going the grain doesn't always It isn't always parallel with the surface of the wood. It's usually going down Into the wood or up against the wood so the idea is that You need to follow the grain in a way that won't cause it to tear away From the wood underneath that you want to slice it off, but you want to tear it out So when we talk about it, this is almost impossible to do without looking at a piece of wood, but Ultimately what we do is we say, you know plane with the grain of the wood read the wood And those are rules that we use To help people come in contact with the reinforcers of having a smooth finish and if you're doing hand-planning So clear rules for the consequences are always is always the best the rules here for For the red rocks pretty straightforward don't go in here Otherwise you will die or lose the limb or something like that. That's pretty clear You know, it's not like saying ooh look at the pretty red rocks. There might be something behind them end-to-end You know, that's a bad rule. That's not a good thing that you want to tell people First off, it's just mean second off. It's just not clear Unclear ones do work. They just typically don't work as well and they have a tendency to be Easily the rule-following behavior from unclear rules tends to be punished rather quickly or Extinguished rather quickly because you don't always come in contact with the reinforcers And there's the whole threat or command thing if you're gonna threaten somebody you're gonna threaten somebody and they Don't do what you're telling them to do You actually have to follow through with threat because if you don't follow through with that threat Then it makes those rules completely ineffective. So if I say, you know, I want you to study this material or else You know, it's like well, what the hell's the or else? And what am I gonna do to you?  Now I've at least followed through on my threat. I'm not gonna do that You know, so I'm not gonna give you those types of threats So because if I gave you the threat and didn't follow through then you now know that my rules are just pointless the same thing goes with commands


Hello, and welcome to this analyst brief where we're discussing how Dell is attempting to elevate your multi-cloud experience with Dell Apex Navigator and initiate what Dell calls multi-cloud by design. Dave Vellante and Rob Strecci and I will unpack what the latest Dell announcements mean to customers. First, let's take a reminder look at cloud usage, some data from ETR, the movement between public and private cloud continues. But looking at the data, we're seeing the market become much more balanced. This chart from enterprise technology research shows the percent of usage of public clouds relative to private clouds, specifically within the S&P 500, which represents a very large portion of IT spend and is generally considered a bellwether. The data shows about 43% usage of public cloud today growing to 55% by January, 2026. And note that the expected 2026 figure is down from 10 months ago, implying that the market is reaching, let's call it a more balanced state, maybe not quite an equilibrium, but many more workloads are where they belong is the point there. And steep momentum to migrate off-prem has generally stabilized. And the cloud growth is coming from new workloads on existing cloud platforms versus a lift and shift approach. And Rob, what do you make of this data? Yeah, I think that it's really clear that that equilibrium is being balanced and people are not just going to cloud first, but they're going to cloud when right or doing it right. And I think a big piece of that is, hey, we want to make sure that we put new workloads where they deserve to be. We always talk about AI and you bring it to AI to the data or you're bringing data to the AI. And I think people are trying to make a balance of that now as well. All right, now I recently talked to Shannon Champion of Dell Technologies and discussed the Apex cloud platform family and the universal storage layer. Then Rob spoke with Maggie Kapoor and Allison Langen from Dell about the SaaS control plane that is Apex Navigator for multi-cloud. Dell recently announced the expansion of Apex storage for public cloud with the availability of Apex Navigator. Rob, what are the salient points here, the customer requirements that people should know about? Yeah, I think what Dell is really doing a good job is really working backwards from that customer and understanding that TCO is really king right now and understanding cloud is an operating model, not just a place. So what they're looking at is how do you bring that with centralized management and seamless integration, seamless data mobility, also being able to bring data from on-prem up into the cloud and vice versa and really trying to bring those things to, I guess you could say to fruition through what they were talking about with the Dell Apex cloud for public storage for public cloud and this Dell Navigator for multi-cloud which really tries to bring that all together. So how do you think specifically this addresses the customer requirements and do you think it changes customer requirements? In other words, will it force them to kind of rethink what's capable with multi-cloud?  I mean, people are not using just one cloud, they never have been, but they're also looking for a platform and operating system that brings them across all of those clouds, both on-prem, in-colow and out in those public clouds. And I think a big piece of it is really how do you do that securely? So they're looking at it from a security perspective, they're looking at it from an automation perspective, they're really looking at it from a management perspective as well. And then kind of on the fourth is you've got to be able to manage and monitor everything across all of those different domains, which is huge. And kind of the fifth piece, and really I think we've talked about it because and they talk about it a lot is the data mobility aspect of it. Customers are really struggling with when do you get to that next level? And I think really people are looking at how do they extend their on-premises to the cloud and to other locations and making sure that they're using the same stack. And really that move is, how do I bring these critical applications that have some unique performance characteristics? Because a lot of things were either cloud native by design but then we had the lift and shift and some things moved back to be reformatted not repatriated but necessarily but brought back to be reformulated. I think that's been a big piece of it. And then integrating with those cloud services is a big piece of it as well. When you start to look at, hey, I'm bringing my data up to the cloud because they have some unique cloud services I want to use. I really think that that's a key for where Dell is building this platform, that unified storage layer with this SaaS delivered Dell Navigator to help with customers managing what they have existing. And I think what was interesting when we talked to them was that also it's about using it smartly from a cost perspective. And how do you really lower those costs? And a big piece of that was, how do you use Dell credits that you have with a contract with them or how do you use your AWS credits? How do you work with partners and do that? And that's a big piece of what they're trying to bring to service. I also wonder, Rob, how generative AI is going to affect this whole multi-cloud play because if you think about the early days or even recent days of cloud prior to, you know, Gen AI, it was really, okay, I got AWS to get the best cloud infrastructure. I'm going to keep stuff on-prem, my crown jewels on-prem. I'm going to use Azure for my collaboration. You know, I might like to use BigQuery for some of my data. And they were generally sort of stovepipe environments. Now when you think about what we've done with the power law of Gen AI, which basically says there's going to be a lot of specific domain-specific AI, much of that could be on-prem. There's also a lot of different optionality in clouds, Azure obviously with open AI, AWS with bedrock, you've got Google with Gemini, you've got Lama 2 from Meta. I don't know if you saw, Stanford just came out with the Gen AI or LLM transparency index, which showed a very wide disparity in the transparency around each of the LLM. So I would think the customers are actually going to be looking to do more workload movement, cloud mobility, where each of the clouds is not so much a stovepipe, but they've got capabilities that they want to operate on coherent datasets. What do you think about that? Yeah, 100%. And I think that also adds to the TCO aspect of it, which Dell went and did a really big story, did an early study with some early adopters of Navigator and the Unified platform and saw that 87% reduction in costs over native cloud storage, for instance. And I think when you start to look at that, what does Gen AI do? It consumes tons and tons of data. And I think that's why people are looking at, how do I put the right data in the right place with the right AI to really get leverage? So if I'm doing my financial data, I want to keep it secure. I want it to be able to be mobile between places that it's supposed to be, have that zero trust security around it, and really then use that for maybe doing my 10Ks and building those types of, like you said, specific applications or SLMs, as we've been kind of joking about, as segmented or small language models. And I think that's key. And I think, like you said, the transparency is key because it is the crown jewels of an organization. It is their data. And in fact, it's the intellectual property of that company. All right, Rob, thank you. And thank you for watching this analyst brief. We'll see you next time on theCUBE.