Well, it's that time of the week again. It's time for chitchat across the pond in this episode number 790 for March 30th 2024, and I'm your host Allison Sheridan This week is our guest is as it seems to always be true lately because I'm not sure I do any other chitchat across the ponds anymore It is Bart Buchatz with another installment of programming by stealth. How are you doing today, Bart? I Am good. Does that mean that the average weight of chitchat across the pond has gotten heavier because it's all heavy now I think so. I mean, I'm not saying I'm not going to but I've just I don't know It's a lot of work to do two shows. You know that, right? Oh, right. Yeah. Yeah And I have a couple of lights actually lined up But I don't want to stop over doing now because I'm having too much fun But I have a few lights in the bag for you at some stage. So we will talk about less Less deep stuff. Okay, let me might even talk about a vacuum cleaner that makes water My vacuum cleaner is called a submarine Okay, there's if that isn't a tease, I don't know what is All right, let's kick in we're still playing in JQ. We got more fun to do We do indeed so we are picking up for part two of what I had thought would be a very quick I was even afraid it was gonna be too short for a whole episode was our look at lookup tables And I guess I'll start this episode by saying that the people listening along will have heard me use the contraction lookup all of last time two weeks ago and it confused you and Jilith can't agree with me that it was confusing Yeah, and I was afraid of a different confusion because I thought it might mix things up between records which represent rows and tables And so I was afraid of using the word table in two places Or I thought I would have been confusing and ironically that made it confusing. So Yeah, to me lookup is an adjective.  So perfect Yes, exactly. So the people who read back will notice the show notes have changed and the show notes for this installment are now lookup Tably, so there we go. That's a little PSA out of the way So where we had we spent a lot of last time being very Philosophical we talked about the whys and the house and that there are different types of dictionary and also forth And we learned about records representing rows and tables and lookup tables Being indexes into data so if you need to search the same data very often if you have an index or a lookup table you can jump straight to the answer and that's really efficient and If you're looking up a hundred thousand records or something the efficiency really stacks up quickly and the other efficiency is to your code Instead of these big long lines with select statements and all this kind of stuff You can just go name of lookup table dot the value you want and hey presto you have your answer and I think you you talked about that last time and This time I've cheated and read ahead in the show notes this time I think your examples like three quarters of the way through this really bring that home that efficiency and how much Easier it is to search if you have these indexes or lookup tables Yes, exactly. So we learned about them in a very broad way last time and then we looked into building them So make it we have data and we want to turn it into a lookup table And we did that using an intermediate format called entries. This is something jq made up To make their functions simple that basically what meet us halfway You make your data be this shape and then you can use this function I haven't worried about lots of arguments and things and so we just arrange our stuff into entries Hand it over to the jq from entries function and it produces our lookup table and that's what we stopped and What we now need to do is the opposite we need to start with a lookup take it apart and Maybe put it back together or maybe reassemble it completely Because a lookup table is extremely efficient at going straight to a specific answer So if you have a lookup table that maps say the name of a food item in your main you to its price You can find the price of a hot dog Instantly effectively because you go basically name of lookup dot hot dog and hate press so you have your answer But if you need to find all food items that start with a D Well now the lookup table is actually in your way Because you can't actually easily do a select on it So now you actually need to pull it apart to search it. It's not difficult to do But you do need to pull it apart to filter it down. So if you have a lookup table with 50 keys and You want half of those keys to go away You actually have to pull it apart pull Show it through a select while it is not a lookup table and then the pieces that are left You put them back together and then you have a smaller lookup table right And of course the other thing is you may want to completely rearrange the shape of the lookup table if you are handed a lookup table of employees by say employee number and You are constantly trying to look through server logs and you always have user names. Well, if it's indexed by User ID that's useless to you you need it by username So what you actually need to do is be able to take someone else's lookup explode it jiggle it about and put it back together until they lookup table that you want instead of the one you were given and This brings us back to why this series exists at all Because I ended up fighting a lot with JQ Because Troy Hunt decided that when you download the list of breaches for your domain It should be indexed by user who was breached back back up back up Troy Hunt who is Explain what Troy Hunt is when he has have I been pwned which we will revisit in great detail in about two or three paragraphs I think okay, so start your sentence over again Just wanted to give context of what you were talking about when you said Troy Hunt Yes, so basically you can download a list of all of the compromise all of the Accounts on a domain you own that you've proved you own a caught up in data breaches and Troy decided that he should index it by the user because I guess he assumed the most common search would be is Bob Compromised or where is Bob compromised because at this at this stage everyone's been in a data breach So the question isn't a yes. No question. The question is where? which passwords does Bob have to change and so the index is by user and Once you've done an initial onboarding what matters is who was caught up in the breach that was added last week So you'll get an email saying, you know, but 10 of your addresses were caught up in this newly discovered breach So what I need to do is the opposite I need to index by the breach and get the list of people not by the person to get the list of breaches and I knew JQ was the answer and I knew just enough JQ to Think I knew what I was doing Spend three hours swearing and eventually get there and decide that I needed to learn this and then I started learning it and because What I'm learning I want to share We ended up doing this series and then I ended up learning way more JQ than I ever thought and it turns out to be spectacularly useful in my work life and I am now doing Jiu-Jitsu on Jason, which is really hard to say and that I would never have imagined Which is really cool and you know a wise person has often said that the best way to learn is to teach So that's some dog food in this one here folks So basically that's what we're doing today. We are doing the Pull them apart and reassemble them and then we're going to move on to my favorite thing about lookup tables And it sounds cool data enrichment. Who wouldn't want some data enrichment? So this is I give you a real-world problem So you're looking through some log files and it tells you that someone with the user ID Bart B was logged into the VPN at night and you're starting to say Bart B Is that is that that Brad fella over or that you know that Bart Brogan fella or is that that you know Bart? So I don't know. I'm terrible at imagining things. Anyway, someone else But you probably have data that maps user names to actual people So what if while processing one data set you could reference a completely separate lookup table It's not a lookup table you built from the data you're processing as a lookup table That's just adjacent that just exists next to it somewhere and pull that data in to Enrich what you have and then say well actually Bart Bouchox was a naughty boy logging into the VPN after hours out there He should he's not supposed to be working late. It was three in the morning for God's sake I know you hate it when I say you could do that in Excel But I'm gonna say you could do that in Excel That's one of the things that a V lookup or an H lookup is for as you can say okay Look in this table over here take the third column over you know find the match in this column Take the thing in the third column and then take it and then stuff it over into this other spreadsheet It's not anything like what you're doing, but it does the same kind of thing and so it's it's super useful It's a different implementation of the same concept, right? And if you're in database land you're thinking oh Bart you're talking about joins and if you're a database person Yes, I am talking about joins because that again is the same concept just implemented differently because Excel Excel uses the word lookup because it's philosophically the same thing But in Excel, it's not a dictionary in Excel It's a table because in Excel the universe is a table everything in Excel land is tables, right? It's tables all the way down in Excel land And you'll be happy to know I'm actually starting to like Excel and actually starting getting good at writing formulas Even though I think the syntax is still horrible, but I am getting better at it I may love it yet. You sent me something the other day that said that you'd made fire. I Had I've made a I've made a genuine pivot table all by myself to solve an actual problem I had and I was smart enough to know that a pivot table could solve it and how To use a pivot table to actually solve it and it was a very real-world problem And I got some nice pat on the back for my boss doing excellent that answered my question perfectly Great. Yeah, we'll convert you yet Bart There we are now I set you a challenge at the end of the previous installment since we were building our lookup tables all over the place So we have been working very heavily with our list of Nobel Prizes and So I wanted you as a challenge to write a lookup table to be able to just get what were the Nobel Prizes for? 2017 or what were the Nobel Prizes for 2022 so basically a lookup table of Nobel Prizes by a year And I was I was slightly naughty of me to do this way because there is more than one Nobel Prize in a year So that means that you had to do the hardest type of lookup table Which is one where you map to an array of possible values Not a one-to-one mapping like with you know the name of food to a price right a hot dog has a price Whereas 2024 has actually 2024 has zero Nobel Prize winners But it will have five of them when we get around to it. What is it for? Yeah, anyway An amount greater than one So that meant you had to use the group by function to Split your data into appropriate these size little chunks and then build your lookup table against those chunks of your data so I Most of the sample solution in my case is comments It is a very very heavily commented and not that much actual jQuery So we start by Exploding our prizes and then sending the pieces to the group by function and telling it to group by year So that will build actually we don't explode the prizes Sorry group by once in a race we just send the prizes straight to group by and group by takes one array and makes it an array of arrays and every child Array is grouped by whatever you told it to so we now have an Array of arrays of Nobel Prize records and each one Each child array is all of the ones for 1900 the next child array 1901 the next child array 1902 So we now have our groups of prizes which we then Explode and recapture all at once in that structure We were so used to seeing now open square bracket immediately followed by dots to square brackets to explode it apart So we basically explode it but contain the pieces and while it is exploded We then need to build these Entries as jq calls them which are dictionaries with a key named key and a key named value They will become a look up table. And so the key we want is the year now our child array has Many prizes but all of those prizes have the same year. So you could take the year from any one of them But every array has a first element Right, because if there were no elements group, I would not have made the child array So they all have at least one. So just I just take it from dot zero dot year It's as good a one as I need to take it from and you didn't do dot string because you knew year already was a String yeah, because it's dirty data as we keep calling it But well, I threw it in as to string just to prove that I remembered that the key the value for the key Key had yeah, I know right had to be in quotes. So I put it in there on purpose Excellent, that's a that's a good way to learn that is I approve entirely defensive coding And then the value Has to be the entire array of all of the prize records for that year, which is just sitting in dot So my value key has the value dot At that stage it's already we have created the pieces of our look up So we then send them to the from entries function which builds our lookup that is that So the bonus challenge actually know before we do the bonus challenge is is where I get to my examples So now that we have this file existing if we would like to find Actually, sorry, the first thing is we can take our file and we can save our lookup table to a file So I have said jq minus f and the name of my challenge solution Nobel prices that Jason and then the redirection arrow so the greater than sign Nobel prizes dash by year Dot Jason and so now our lookup table is saved safely in Nobel prizes dash by year dot Jason So if we wish To find out a particular prize we can just query that file Now if you're lazy, not if you're lazy If you don't need to keep your lookup around forever And if you don't mind wasting a few CPU cycles because you're an m3 Mac or whatever You don't have to save the lookup to a file to use it You can pipe the output of the jq command that builds the lookup to another jq command that uses the lookup So if you immediately pipe the output of building our lookup to jq dot 1980 it will tell you the prizes for 1980 the small subtlety being you have to quote 1980 otherwise jq thinks you're asking it for point one nine eight No, of course it would Which it will tell you is zero point one nine eight great. Thanks, jq. That was not what I meant at all Let me guess you found that out the hard way Yes, I did what what what what you know, and I you know, I took out the second pipe like no my data is fine Why is oh, I see the confusion. So yes, that is the answer there But the key point here is that instead of having to do a select So instead of saying dot prizes explode it pipe it to select dot year double equals 1980 I just said dot 1980 And I got straight to my answer So even that very straightforward lookup already makes our queries way more human-friendly a lot easier for humans to read Yeah, what I said a bonus challenge Because the more complex your data is the more complex your lookup tables may need to be and in the case of Nobel prizes The most logical lookup table is By year by category. I would like to know who won the physics prize in 1970 Not just all the prizes in 1970 or all the physics prizes with the physics prize in 1970 So what we'd really like is a two-level lookup where we could say dot 1970 dot physics And have it just jump straight to the answer. So that was the bonus challenge and the key concept to a multi-level lookup table is that it is a lookup table where the Thing you look up is another lookup table Where the thing you look up is the final answer. So it's a lookup table of lookup tables of records So like a two-dimensional raise an array of arrays. It's a lookup table of lookup tables to our final piece of data I Can't I'm coming to you're looking at my face. No one knows why he's pausing there It's because I've got this look on my face What confused me at this point is I don't know the shape of the data that you're talking about like I don't know what it looks like in my head to I would I think I'd understand it better If I knew what it looked like when you said that, you know But are there squirrely brackets inside of square brackets or vice versa or what? Squirrely squirrely square it's a squirrely squirrely square Okay, so at the very top level you're gonna have an open squirrely bracket and it's gonna say 1900 colon okay, and then a set of that being a square bracket and then your Nobel prizes It will be another curly okay, it says physics colon and then there's your square bracket containing the price Wait, is this then there's no square brackets, sorry then you're straight into the answer because there is no you're now I know what you're now a one One mapping sorry miss my question you just described what the answer would look like I meant what does the data set that you're creating this from look like after your solution? To the original problem. What is the data look like? That then you say it's a level look up Wait, which one you asked me to describe the answer to the first at the after you do the first challenge You've got a data set that looks like something it has a structure. Ah, what is that structure level is it? So the top level is a dictionary so it's a curly okay, and then the first key will be 1900 Colon mm-hmm open square bracket. Yeah a curly for the first prize in 1900 comma a curly for the second prize in 1900 comma a curly for the but everything under 1900 is a is a square bracket Yeah, and then you've finished the 1900 square bracket, and then you have 1901 colon new square bracket good Okay, one prizes. Okay, so now when you start talking about a two level look up Are you saying that you have to do two levels inside that or the solution you're creating is two levels? So we're starting with the same input data, right? It's just our normal noble prizes But the output this time is going to be a curly bracket 1900 colon a curly bracket physics colon Straight into the data. Okay. Okay. I thought you were going from the end of the first solution into that would be The input to the second solution, but you're not No, no, no, no that would actually be more difficult because then you'd have to break it apart again That's why I was done and then reassemble it. Aha. Yes. No, no So basically we're back to our original list of physics are sorry not physics all the prizes. Oh, I might like the physics one best But they're all important Sure so We again start very similarly and there's a real symmetry here if you look at the solution because it is our First solution with a more detailed inside So we again start by grouping our prizes by year because the top level lookup is still by year So we still want to get all of our prizes broken into a year So we start the same and then we do our thing where we explode them inside square braces So explode and catch or catch and exploded everyone. You want to phrase that? but now instead of building our Simple entries. We're now going to build a whole other lookup table. So effectively we're starting our solution again Only inside here Okay, so we already know that we need the key to be the year so we can still steal that So we say key colon dot zero dot year. So so far. We're still the same comma and Now things become different. So the value is now We open a roundie bracket to catch everything together and in here we we build a whole other lookup table So we catch and explode and then we build a whole new set of entries where the key is the category and The value is the full record. So key colon dot category value colon dot and Then we build those into a lookup table and All of that is still the value of our first lookup tables. And then we finish that And then we're left with our top level lookup table and that goes to from entries to build the outer Lookup table. So we have a lookup table inside a lookup table Okay, and the lookup tables themselves are arrays, correct No, a lookup table is a special type of dictionary So we call them lookup type dictionaries a few times in the last installment to try get that as you're into your head But your code says build the entries for the second lookup table. Oh It's going to be an array, but when it goes through from entries, it will no longer be an array But in you have to build it as an array. That's right. That's right. Okay. That's what's being confused by Following the indents on this one was tricky And this is why I'm a big fan of writing him in separate dot jq Jq files and and indenting imagine if this was still all on one line like we started doing a jq at the start Yeah, you know what though? I actually I worked on my homework like this But I find it easier to read and understand when it's in a in a string just in one long command That to me is easier to read Wow Because of the indent things I don't know Maybe it's because you indent by four that it gets really wide and I'm going back and forth I like indent by two because then I can I can look up and down and see the lines easier Technically speaking I do indent by two, but I think the I think Printer we're using yeah I think the pretty printer we're using is because I indent by two but they're tabs and then I guess they get turned into four by our theme Actually, I'm not looking at it in the theme. I'm looking at it in you're not as a plain text file and they're four Is it a viewer? Yeah, okay? Well at some point in time something is translating up to four. Yeah, okay Yeah, because I work on them as two. Yeah, interesting. I mean, I know you take great care to line them up But I sat there going okay, where's the end of that squirrely bracket? Well, cool, so it's still not all that complicated Well, right exactly. It's just repetitive because it's nested one inside the other But now when we go to use our lookup table we can say Give me dot 1980 dot physics and we jump straight to the physics prize for 1980 Whereas if we were doing that the old way We'd have to start with our prizes array explode it send it to select dot year equals 2023 send it to another select dot category equals peace or physics or piece or whatever and then we'd get our answer out Whereas now we just say dot 20 23 dot peace or dot 1980 dot physics or dot 1970 dot chemistry, whatever we'd like nice and that's why I love lookup tables That's why I adore lookup tables So, yes Right, so now let us start going the other way So I'm gonna take a moment now to tell everyone about this amazing service called have I been pwned which was started by an Australian security researcher sort of on a whim because he thought it will be useful and he didn't think it'd ever be very popular and Now it's his job And in fact, it's his job and he's now it's now owned by one password No, it's in collaboration with one password and owned by Yeah, it's a partnership with one password and there's all sorts of really cool stuff. Basically, it's become a giant big deal It is a massive database of all of the known data leaks That are public that are publicly known and it's searchable and indexed And so if you are the owner of an email address You can prove that ownership to the website and it will tell you every Data leak that your email address has been caught up in or if you own an entire domain You can prove ownership of the domain and then you can download a Report of all of the onage all of the all of the breaches that everyone on your domain is involved in in two formats You can have it in I think it's csv is the other one But I don't care about any of the other ones that I'm pretty sure it's csv But you can have an adjacent so you can get a nice Jason download Which will tell you all the breaches that you were caught up in and if you're a small You know if you're a hobbyist and the lowest tier of have I been pwned for a whole domain is free So the link to the signup page is in the show notes And if you you know if you have a small domain for your family or something or a little project You do you're almost certainly in the free tier And I would actually advise signing up because it means you then get email notifications when anyone in your domain is caught up and something nasty and you can Take a little bit of remediation Hmm, okay I mean it's useful to sign up on your own email address if nothing else But if you own a domain, it's also useful to sign up for the whole domain And if you send up for the whole domain then you do get the Jason downloads, which means you can play along If you sign up you end up with a file When you click the download button you have a file that has a Structure as follow so at the top level it contains a dictionary with two keys One called breaches, which is the only one I care about and one called pastes Which is basically anytime your email address is mentioned in a pasted bin post What what is pasted bin? It's a imagine a clipboard in the cloud. That's public It's a place where you can just put text and it just exists at a URL and it's what's its relationship to breaches and There are times when attackers will put things on pasted bin Okay, as a way of publishing for free. It's a basically like a like tweeting it So then it's public right if someone tweets it, it's public. So if someone sticks on a pasted bin, it's public No, do people like to use pasted bin, okay It's a thing. I don't care. So basically we have some what I'm calling Semi fictitious data There was a real export used to build this example data But the actual names and stuff are all completely fictitious and I just completely emptied the pastes key So it's just empty. It just goes an empty dictionary the bit we were interested in is the breaches dictionary and That is indexed by the name It's basically the email address without the domain and I think it's done like that so that if you lose the file You haven't actually done yourself another data breach Because you know that it's you know a bar to be at something but you can't tell from the file Where the bar to be was who would be caught up in whatever breach so I think it's a really good way of keeping the file safe is to not have the full email addresses just a bit before the At and it's your domain. So you know the bit after the ad you don't need Troy Hunt to tell you what comes after the ad It's your domain So I do kind of like the idea that it's just a bit before the at similar you get so the these are these are a lookup This is a lookup table with the index is on the name Which is the bit before the at sign and the value is an array with the names of all the data breaches That you're caught up in so in the case of our fictitious J. O'Sullivan who does not exist They were caught up in just one breach called online or spam, but e-green was only in one dropbox But whoever the fictitious person MW Kelly was there was it caught up in five breaches two different LinkedIn ones For that there were three breaches of LinkedIn in total by the way So she whoever that is they haven't quite collected the full set They've been a LinkedIn user a bit longer. They could have been in three anyway, but you said they're fictitious These aren't real people in this There are real people whose data matches this but they don't have the names in that file. Okay So basically I had a download with about a hundred thousand records and I asked jq to pick me five random records Okay, which is why we have a very realistic spread of actual breaches and then I completely change the names great So yes, there is someone on planet Earth with those breaches But you have no idea who and neither do I anymore because I totally forget The other thing that is useful to note is that you can get a Details of any breach at a standard URL So if you're going well, what does it mean to be caught up in online or spam bot? If you go to have ibeinpwn.com forward slash pwned websites Octo sort of pound sign hash symbol whatever we're calling it the name of the breach It will give you the details for that breach So then you know that that was pretty nasty one which included passwords. I always hate the ones that include passwords So If we where am I going here? Right So we now have a lookup table. It is indexed by The account name we're going to call it the bit before the at symbol and it gives us a list of our breaches But if we would like to search it based on the breach we can't at the moment, right? So we need to disassemble that lookup tape and the opposite to from entries is Two entries wait a minute wait a minute last week you taught us how to build our lookup tables now You're telling us you're teaching us how to take them apart That's it exactly. Yes, it is a mirror image Okay, so to build a lookup table. We made an array of entries So when you ask jq to rip apart a lookup table jq will give you an array of entries So if we say to jq dot breaches pipe to two entries we will get back an array of Dictionaries each of which has a key named key and a value with a value So in this case key j o solvent value the array with just online or spam bot key e green value dropbox key Mw kelly value dropbox KOMU LinkedIn LinkedIn yada yada yada No, you can see that the structure we get back is awfully familiar looking Because it's actually the structure we were putting in right what we started with before in in previous lessons Exactly so now we have an array containing records. I mean they're Oddly named. I wouldn't have called it key. I would have given it a more sensible name But hey, it we have it. It's got a shape now that we know so if we want to query that we now can and So we can also Save it back to a normal Array we can save this back to a normal array of records so that we now have the data in a normal shape Right, so if we take our dot breaches and we pipe it to entries and Then we do our usual trick of exploding and catching the pieces Well, we can just use the standard jq for building a new dictionary to build ourselves a new dictionary where we give it the key account name with the value dot key and The key breach names with the value dot value and now what we get is a sane Structure that we could save to a file and now we have the data in a structure that makes sense for searching in any which way we like Now the code I have in the file is quite long So I've redone the code with all the comments taken out Dot breaches to entries Exploded and catch Open curly account name colon dot key breach name colon dot value. It's actually really short code It just looks very long when I fill it up with comments It's also giggle making that it's a count name colon dot key. It's like, okay. We just had that key and that value now We're just gonna turn on say okay pretend just read everything we learned last week read it backwards Well, yes to be perfectly honest because what we're ending up with now as the output here is very equivalent to our inputs last time Which is a an array of record type dictionaries, right? They all have an account name. They all have breach names So we now have a nicely structured regular piece of data that effectively represents an Excel table with the columns account name and breach names So great how we can query that in all the normal ways we can use our select To find everyone caught up in a particular breach So if we wanted to know who all was caught up in online or dot spam, but our online or spam bot we could say Take our list of records explode them pipe that to select and Then ask select to look at all the breach names And if any of those breach names match dot online or spam bot pass that through so If you we're going back in time here a little bit So I'm gonna remind you what the any function does because you haven't seen it in a few weeks You haven't seen in a few installments and we only do one every two weeks. So we haven't seen it in quite some time So the any function if you give it two arguments the first argument is a way to make values So we are saying dot breach names explode it So the any function is going to explode the breach names and then what is it going to do with the pieces? It is going to apply the test dot double equals online or spam bot And so the any function is going to end up receiving a whole bunch of booleans true false true false true false if any of These booleans are true the any function returns true So the select function gets true And what a select you when you give it true this passes it on through Passes everything that it got Yes, there we go. Thank you. Yes, that is exactly the piece. I was hoping you would say So in other words when any of the breaches are online or spam bot the entire record gets passed through otherwise Nothing gets passed through so This is going to filter down our list of records to just the people who are caught up in the online or spam bot breach and then we pipe that through to a simple filter that says dot account name to just pull out their name and So the end result we get is that J. O. Sullivan and a Hawkins are our two users caught up in online or spam bot So we have now queried our data from have I been pwned Having transformed it into a list of records. We started with lookup tables sent it back through using two entries turned it back to a Regular a just an array and then now we can use the tools we learned before you taught us how to do lookup tables Right because in this case, we're not we we don't want to find all the breaches for a person We want to find all the people caught up in a breach. So our lookup table was no use to us It was a hindrance. So we disassembled it and turned it into normal data again and then queered at the old-fashioned way But of course if you say to yourself goodness me This is the kind of query. I'm going to be doing very often Well now the logic tells you that you should build a new lookup table Only it should be the other kind of lookup table. Oh, no, are you gonna do that? We are but I'm actually going to take a little pause here just before we do that So what we did here was a two-step process, right? We took the lookup table We transformed it saved it to a whole new file and then we use that whole new file to do our work and That's great for something you're going to use over and over and over and over again It's like we do the work to build the file once we save the file and then we use that file Okay, great, but what if you just need to quickly Change it you just need to quickly query a lookup Do you really want to save it? Well, you don't have to right you can just pipeline these things together So if we want to query our lookup table without saving it first We can just say dot breaches Pipe that to two entries. So now it has become our array with key whatever value whatever Well, we can just select on that. We don't have to save it. We can just select So we select any it's now called dot value Okay, fine. We'll call it dot value Equals online spam bot and then the answer we want is it called the dot key now Okay, we call it a key and we get exactly the same answer as we got before without ever building a whole separate file And when I'd save me it, we just took our lookup table broke it apart queried it and then took our answer. Okay And Again, we can filter these things using You know Any criteria we like really? right, right, that's where so as To sort of to take this idea and make it generic The process doesn't really matter what we're trying to answer right whether we're trying to answer the question of you know All the breaches that begin with a B Or all of the users who are cut open at least three breaches Like whatever it is you want to do it doesn't really matter what it is The shape of the answer is going to be the same. It's going to be convert the lookup to a list of entries filter the list of entries and Then maybe or maybe not convert that back to a lookup table if you're what you're trying to do is shrink your lookup table permanently or Output the answer or whatever. Oh, I didn't think about that example where maybe you just want to make the lookup table simpler So that you don't have all this other garbage in your way to fuss around and make sure it's not selected or whatever Exactly, I mean you just may actually want the smaller lookup because you got the hand of this giant big set of data So you could filter it and then reassemble it and then you have a smaller lookup So effectively like you would filter an array using a select You end up filtering the lookup by breaking it apart filtering it and putting it back together right, right, so Let us do that for real by taking our big lookup table and making it a lookup table of all of the people caught up in the Dropbox breach. That was a nice one That was 68 million email addresses and panceries in 2012 which in 2012 was a very big deal Nowadays we're used to these giant big breaches, but in 2012 we really we really lost our stuff over that one You and I talked all about it on security bits. I'm sure if we were recording that early. Are we I think so probably were yeah Yeah, we go we go way back Bart. We do. That's that's a hole. That's a whole dozen years ago. Anyway, so The filter we want our dictionary isn't quite at the top level so we start off with dot breaches and We don't want our answer to be different in shape.  We're gonna update dot breaches We're gonna basically so we have a dot breaches and we want to end with a dot breach So we're gonna use our update assignment operator from two or three installments ago So just redefine the value of dot breaches to our smaller lookup that only contains a Dropbox people So we're gonna start by saying dot breaches pipe equals in other words update this in place Calculate a new answer and stick it right where you found it here in dot breaches and To collect together all of our work. I'm just using a round bracket just to collect it all together So we're opening around bracket and we close around bracket all the way at the very end of our file here All right, so everything we're doing is going to go into dot breaches. It's just going to be a new value for dot breaches So we start off by converting it to our list of entries. We say two entries Then we do our usual explode and catch the pieces trick and Then we take those pieces we've exploded and we shove them into our select any dot value dot equals Dropbox Okay, so if any of the Values in our entries contains Dropbox it passes through and so now we have a smaller array of entries So we just send that back to from entries and there we have a smaller lookup table Just for anybody who is following along It's that double equals Dropbox just so we don't get any corrections earlier or later. It is correct in the show notes Yes, and so what we do that what we see is that we now get an output that is the same shape as the input Right, it's a top-level dictionary with the key paste that is still empty and the key breaches But now it only lists two people e green and mw kelly and e green was caught up on Dropbox and mw kelly and Dropbox and four others. Okay. That's cool. You know We have filtered down our doohickey So This is this is a good illustrative example, but we have hard-coded Dropbox into this But would the logic be any different whatsoever if we wanted to filter down to a Facebook breach or if we wanted to filter Down to the infamous yahoo breach or whatever else I mean the shape of the entire thing would be exactly the same except the word Dropbox would be something else each time So many many installments ago We learned about variable names that we could use the minus args Command line argument to pass in a value that would become a variable. Oh, yeah So let's rewrite let's rewrite this so that we use a variable named dollar breach And we just shove that into our double equals And now we have a completely generic filter that will filter us down to just The relevant entries for any breach of our choosing So let's now Do that by saying jq minus f the name of the file where we save this nice new generic filter minus minus arg And then minus minus arg if we remember is one of those weird Terminally things where it's one flag that takes two values The first value is the variable name The second value is the variable value So minus minus arg breach linked in means dollar breach will have the value linked in And then we say the file to use which is our dubby data And the output will be the breaches dictionary containing only mw kelly because mw kelly was the only Person code up in the linked in breach, which is what we were testing for Makes sense. I'd forgotten all about that. I'm glad you brought that one back up and I do want to thank you for Reiterating the things that we haven't talked about in a while because you know how easily I lose things fall out of my brain Well, I sort of I do it intentionally because I figure you're you know The reason I love working with you is because you are representing the every person Or maybe the lowest common denominator All right, but I know you will want this little help which means that anyone reading along is going to want it too So Yes, you you put it into my brain, but I'm really happy you do because I think everyone benefits Well, you're welcome At this stage we have Disassembled and turned a dictionary into a list of records we have Rarely disassembled shrunk down and then reassembled our lookups So the last sort of big piece of this that I promised you was to reindex a lookup table to basically Change it from indexed by one thing to being indexed by something completely different In the case of the have I been pwned data The most logical way to reindex The data would be instead of it being Breaches by user Have it be users by breach which is exactly what I wanted when I started learning jq all those months ago before I went to my big christmas break and We can do it And the result of doing it will be the very useful Um Lookup table where you have breaches being collection one colon p trainer dropbox colon the array egreen mw kelly Kayo moo being m kelly linkedin being m kelly, etc and If you want to see how that's done in jq it is in the zip file. It's called hib p dash transform to by account name dot jq I'm not going to show you now because This is not a one-to-one mapping There is not a one-to-one mapping between the user and the breach Therefore the only way to reindex it is to use a variable And we have not learned how to declare variables. That's a next installment So if anyone would like a sneak peek of next installment Then That's it. That's the sneak peek You may be very pleasantly surprised how easy variables are But that is a bonus extra So you can see what we want to do, but I can't show you how to do it with the have I've been pwned data So we're going to invent a new data set where we're going to transform the data set from one key to another So this is the fictitious staff list for all of those wonderful workers who make up the pbs creators It's basically me allison and helmer So it is a lookup table of employee records. I say employee that implies payment. There is no payment I want to raise Staff actually I call it staff. Yes, you may have a hundred percent raise a thousand percent range It's from zero to zero So it's indexed by Username so allison s which then is a record with name username email and website to bar to be name username email and website and helma vdl With a name a username and email on no website because I couldn't remember what helma's personal website was And I wasn't sure if she'd want us to tell people even if I could remember So uh Helma has a null for that one. Sorry helma. Um So what we would like to do is to transform that Into a lookup table not by username But by email address because that is a very sensible alternative way to index that same data And that gives us an excuse to learn how to do this on a simpler dataset Because there's a one-to-one mapping from usernames to records And a one-to-one mapping from email addresses to records, which is why we don't need a variable Because they're one-to-one mappings variables are not needed Variables are only needed when you have a lookup table where it goes to an array of records Then you need a variable as we will discover Next time so it doesn't really matter what you're transforming right the algorithm is always the same Like the set of steps is always the same regardless of what lookup table you're transforming into another lookup table You basically disassemble the original lookup table into an array of entries You then change the key and the value inside of that array of entries to make a new shape And then you put those new entries together to form a new lookup table So if we go to entries we change the entries and then we go back to a lookup table Okay, so in this case we're going to make one by email address So we start by saying two entries So our staff data was a top level dictionary. So we just we can start straight away We just say two entries and now we have an array of entries So we explode and catch As we're so used to And what we want to do now is we want to have the key become the email address So instead of the key being the username we want the key to be the email address So we do an up we do an assignment But it's the plain old equal sign, which is plain assignment And there's a very important reason to use plain assignment and not update assignment So and the difference in the two which again, we're going back a few weeks So that's why I'm sort of stressing this point. So when you use pipe equals To the right of the pipe equals the value for dot is the current value of The thing you're updating So the value for dot would be the value of key, which is a username But we need to be able to reach back to get the email address So we want the value of dot to be the entire thing Not the current value of key And so that's the difference in pipe equals and equals Okay, so the value of dot is the entire current entry So the email address is at dot value dot email Okay, so that makes a dot dot key becomes equal to dot value dot email And we don't touch anything else in here and everything else stays the same or what? It does because the value is the full record Which is still what we want the value to be in the new lookup table, right? Right, we just want the index to change. So the only thing that needs to change is the key Okay, so the result of this will be will be a dictionary with Uh, just the username Or sorry the email address allison at pbs.demo and underneath that will be name username email website for allison For allison pbs.demo. Okay. It's just going take this We Except so literally the only thing we're changing is the index right because we're just reindexing. Yeah, just reassigning the thing We have to change Yeah, exactly So we we pull it apart we give the key a new value and we put it straight back together by palping it to from entries So two entries change the key from entries. It's very elegant. Yeah, dang it I was I didn't want to interrupt you the word time is going to say well, that's very elegant Literally, right and again when I pull it apart and take all the comments out It's just three very simple lines of code. Right.  It's it's in the show notes if people want it So that then brings us to enrichment Of data. I can't I can't make you money. I can't make your data pretty here So let's go back to our have I've been pwned data set, right? So we have our breaches and we know that jl Sullivan was caught up in online or spam, but But really what does that mean? There must be a source With info with details try hunt must have captured all the metadata about all of these breaches course he did And not only did he capture it, but he makes it available in jason format So the url have I been pwned.com forward slash api forward slash v3 forward slash breaches Will download a giant big glup of jason, which if you pipe it through jq for its pretty printing abilities We'll show you that it is a dictionary of records indexed by the name of the breach You took the key out in this case. So we can't we can't see it in in the text there, but I trust you Actually what it actually downloads is a flat list and I transform it into a lookup My humblest apologies. So I transform it into a lookup as I download it So the curl command is a terminal command for fetching a url Right, so we're saying curl and then we give it the url And the url is j returns jason so we can pipe that straight to jq So we say curl the url pipe to jq And in jq we explode the list that have I been pwned gives us And then we pipe it into the jq syntax for building an entry We say key is dot name The value is dot in other words we The key for a lookup becomes the name from the record and the value is the entire record And then we show that to from entries And so we end up now with a lookup table that indexes data breaches by their name And for each data breach treyhunt gives us a lot of information So I have a sample one for online or spam bot in the show notes But it has a name it has a title which is basically the name in pretty English So online or space spam bot Some of the foreign breaches the title field is actually in like chinese characters and stuff So the title is genuinely the human name of the site It has the domain name if there is one there isn't always a domain name The count is how many people record up in it So in this case one two three one two three seven hundred and Eleven million four hundred and seventy seven thousand six hundred and twenty two Wow, the date the breach really happened which was 2017 And the date the breach was added to have I been pwned which was the same day it happened. That's interesting It's not always the same exact time down to the second that seems improbable Oh, no, that's added in modified. Okay Yes, that means that treyhunt hasn't changed it since he added it Then there's a description, which is a nice piece of html that describes it pretty nicely And then the one that I find most useful is data classes It's an array of the types of data that were caught up in the breach In other words email addresses and passwords for this breach And so whatever was caught up in the breach would be listed under data classes And that's particularly useful to enrich your data So let me back you up a little bit here When we so we did the curl on the breaches file from the internets piped it through jq We catch and explode our array Uh, and then you said uh squirrely bracket key colon dot name comma value colon dot I expected to see A series of dictionaries where there was a key of name And there isn't this is just globs. No, no, this is No, no the whole file is absolutely what you've described, but the whole file is massive This is one of the values. This is one of the blobs. This is just one record Okay, I see what the problem is the name that name is actually really weird stuff like 000 webhost colon And that is one of the names. Yeah one two rf colon So those I I couldn't see it because they were so weird looking one l two six 17 media. Okay. I was expecting name like Online or spam bot is the name in the thing that you found. So there must be one down there that does make sense Oh, yeah, scroll down a long way and you buy the weird ones to start with digits because they're in Alphabetic order got you so if you scroll halfway, then you'll find dropbox and linked in all the ones you're expecting Okay, well that makes me happy. I did understand Yes, perfect So we have this nice extra data About the stuff in the data file we care about So can we marry the two together while I'm querying my list of who has been pooned? Can I bring in this information about what each data breach actually means? And the answer is of course we can because otherwise I wouldn't have asked the question, would I? Okay So we are going to learn about another flag for the jq command. This is the minus minus slurp file flag Oh, you love you a good I do love me a good slurp as I have a big cup of coffee here in front of me as we record So the minus minus arg let us bring in a variable name with a value The minus minus slurp file is very similar. It will bring in an entire json file as a variable So it appears in the jq script as a variable with the name we give it But the value of that variable is the full content of a json file So we are going to have a variable named breach details And we're going to say pull it in from our breaches file Okay, the file we downloaded from curl the whole file Will be available to us as a variable named breach details Oh file Okay Now there's a very small subtlety here that I don't I wanted to say that without confusing things and now I'm going to very very very slightly Confuse things Slurp file doesn't insist on there being one piece of json data in the file The file could contain multiple dictionaries or multiple strings. It could share multiple pieces of json So slurp file always arrives as an array because it's like well, they might have given me more than one So we are only giving it one thing in our json file. So it will be Breach details zero So we're just always going to be using breach details zero Okay, that's the only subtlety is it is an array that can kind of make sense Like if you don't know what you're getting you better slap it all in an array. So you know, you've got a thing Precisely and then you can use length to figure out what you got right within your script But yes, absolutely. It's a very safe way to do it is to say, okay fine. I'll just make an array So now Let us do something very real world Let us imagine that we would like to send a mail merge to all the people caught up in the Whatever in in there I think I do it for the online or spam bot breach because that was the first breach to come up in my sample data Um, so we want to send a mail merge to all of our staff caught up in that mess And we want to enrich that mail merge with the data from our nice new file. We got from have I've been pwned Not that this is a made-up example for you or anything Definitely not something I may do regularly for reels. Definitely not Um, and of course a mail merge we like it in csv format So that it can actually be plopped into word and then spot out in a setting on the in the normal way So we're going to we're going to re-remember that we can use at csv to output some csv from our jq filters Oh, that gives us another excuse to bring back some old knowledge So we are going to start by finding all the relevant Breaches or so all the relevant people and then we're going to enrich the data for them So we start off with our dot breaches and we pipe it to two entries. So we now have our breaches in a four, you know in As our entries We explode and catch as we always do We select any dot value is equal to online or spam bot. So this is exactly the query we've done a million times already So now we are left with only the entries for the people caught up in online or spam bot And so now let us build a little piece a little dictionary for our mail merge So the two is going to be the account name at the domain so Account name is sitting in dot key So we're going to build a string which contains so backslash open round bracket is how we pull something into a string So we're saying build me a string with the value of dot key followed by the symbol at followed by pbs.demo That's our pretend email domain. Right. There is no pbs.demo You know that could be whatever right Then we're going to say the breach title. Okay, great. Where do we find the title for the breach? Well, the title for the breach is going to be in breach details zero Dot online or spam bot because it's a lookup table. So we just go straight to online or spam bot dot title the date Breach details zero dot online or spam bot dot breach days the html description You get the idea here, right? We pull in breach zero dot online or spam bot dot description And what was breached? Well, we'll shove that through a join and then that's our breach data field So now we have little records That have been enriched. Let me ask a question here. So You've said um, you gave us the two the breach title breach date You had a comma between each one of those but then you say join it with commas Okay, so If you so we're building up a we're building up a dictionary here We have an open curly bracket two colon So the key two has the value that comma the key breach title has the value that comma So that's just building our dictionary But inside the string for breach to data Like a csv file needs a string.  Okay. So the commas are not The commas are not the csv part The commas between those uh key value pairs. That's the that's the dictionary Yes, exactly. So we're we're still in plain old jq land here We're just building a dictionary with A two a breach title a breach date a breach description and a breached data key Okay, it's still it doesn't sit right in my brain that the join is inside of that dictionary the creation of the I guess I would have thought it would be outside the after okay, but the Okay, but we are only doing that to make the value to go into the breached data key All right, the breach data keys value is a string That is joined together because I'm sorry. I'm sorry. I thought this whole I missed a Parenthesy I thought this entire everything you did the to the breach title breach date. I thought that was all in the join It's just the breach data piece Uh, no, why did it have to have a join? Does it have because it has a bunch of information in it? Because it it is an array. So if you look at the example data classes is an array of strings And we need to turn that into one string Because the csv file expects a single value. Okay. I'd forgotten what data classes was it that didn't have that memorized Okay, because that's where it was what did they lose? And so you want to you want those with commas? I am now with you I'm glad I asked the question because it was a completely different question than I thought but okay good Perfect. So we have now put together our exploded pieces So what exists at this point in our script? Is an array of dictionaries and we now want to output those in csv So the first thing we'd like to output is the titles So I just wrapped it in brackets to hold it all together But basically we make an array with the string to Breach title breach date basically it's an array of strings that are our headers And then we pipe that array of strings to at csv because at csv insists on getting an array as input And it will output one line of csv So that will give us our first line of our csv, which is just the titles Then we need to output the actual data. So this is a this is an example We haven't done in ages of the and also operator We want to put out the headers and also the data so comma And now again captured all together inside round brackets to keep it together We explode our data so that we actually have lots of entries now We pipe all of that to the jq syntax to build a new array and the array is going to have the values dot to Dot breach title. So in other words the values from our look. Okay, we're now being pulled into this array And then that array is being piped to at csv So now we're going to get output a csv one line for everything in that array for every dictionary we have So it's going to give you the the title row first and then all of the other rows after that. Well, that's cool Exactly. That's really slick Exactly. So if we do that on the terminal, we say jq Minus or because we need raw output for the csv. We do not want csv in json That would make no sense. So every time you use csv. It's always minus or for raw So then we say minus f for pbs 164 dash b dot jq is the file that contains all that stuff. We just talked about Minus minus slurp file breach details Then the have I been pwned details And then our data file of of breached people is the input And then we redirect all of that to mail merge dot csv And I didn't save mail merge at csv because it's in csv format But if you run that yourself you can open it in excel and it is A perfect csv file ready to go into outlaw can be mail merged with your word document with your fields in it Okay, I'm going to do it Good because I want to drink of coffee. I was rather hoping you'd uh, all right I'm opening it It's opening in excel and I'm hoping that I didn't my email address wasn't in this uh in this breach Why does excel takes so long to open in this uh In uh because it's massive sonoma. No, but it's it's it's always been mad massive Okay, no good. Joe Sullivan and a hawkins they got they got stuck with email and passwords. Wasn't me Yay so The only way to make this better So we hard coded again, didn't we we said dot online or spam bot dot title Well, it doesn't take a genius to figure out that we do again what we did previously We had an extra variable. I'm just going to call up breach name And so everywhere in our code where we had the actual string, uh, what is it one liner one line? Online or spam bot not one liner online or we replace that with dollar breach name and the only very mild complication is that dollar breach detail zero dot Well, you can't put the dollar sign there without wrapping it in square brackets because that is the syntax for looking up Based on the value of a variable So we just say I need you to go into the key With the value of the variable breach type breach name which may be online or spam But it may be dropbox. It may be linked in or whatever That's just saying the value of this variable is where you got to go and then pull me the title So other than that, that's the only thing that's changed in this entire script And so now We have a completely generic searcher that will build us a mail merge for any breach Can't imagine why I would have developed such a thing So the only difference on the command line is that we now add a minus minus arg breach name dropbox minus minus slurp file All the rest is now the same as before so we just have the minus minus arg So that we pass in what breaches that you're looking for In this case, we looked for drop off. Yeah, I said it before but I'm going to say it again. That's really slick I like that Isn't it isn't it? Um, and especially once we have a csv file, we really can do anything when they're right So we've now we've taken some jason data we've All other piece of jason lookup data And we've used it to build a csv file that we could open in excel and do a pivot table on or do a mail merge on or do Whatever we want. This is data manipulation, right? This is bread and butter. This is real data processing here And this is why I adore jq. This is this is as bread and butter It is working with jason as a data structure as a source of data So I have a challenge for you. Yay. I would like you to tweak my tweaked version here and make it just a little bit cleverer So instead of finding the breaches that is exactly Dropbox with a capital D I would like you to treat the the variable as a search string So any breach that contains The search string and I would like you to be case insensitive So if you ask it for a mail merge for everyone caught up in LinkedIn You should end up Finding the people caught up in three breaches because there were three breaches that linked in 2012 2021 and 2023 and If you'd like some bonus credit I would like you to Filter down to only the breaches where passwords were exposed Oh Breach didn't if the breach didn't leak the passwords. It's not worth doing. All right, right Let me ask you on the first part of it. Are you making us do regular expressions? Uh, yes There's a short answer, but remember jq has regular expression support. Yeah. Yeah Okay And you don't have to do it with a regular expression. There is there is a jq function called contains that may be useful. Yeah. Okay That sounds fun I I don't want to disparage the Nobel Prizes, but we've done a lot with it I like having a new data set to something else I haven't quite decided what we're going to be Meeting next week. I think actually next week I think we'll say what the have open phone data set next week because variables fit that data set nicely actually So probably anyway I had thought A month ago That lookup tables were a simple topic that we were doing a day But actually there was a lot of meat here But I think it was really good like this is really good stuff So I learned a lot writing the notes and I'm hoping people learned a lot looking into it And I had thought that the next thing we will be talking about would be Altering arrays and dictionaries without exploding them because you can edit them in place Which is very useful right because how many times a day that is where I say explode and catch Well, there's a better way than exploding catch you can just edit them without exploding them But that's going to be the installment after next because I've been trying to find the right time to do variables Because the jq documentation is very clear that variables are not your variables should never be the first thing you reach for with jq They are needed For advanced usage So until you're doing advanced things if you're doing variables, you're making it more difficult than it needs to be But we haven't been at this a while Yeah, but even in this example that you just gave where it's like, well, I don't want to have to type it out every time Yeah, I don't want to hurt you so but I say sorry When I say use variable, I mean declare a variable we have we have not to this day declare the variable We're sure you did dollar breach details Right, but I did I imported a variable from the command line I didn't inside my jq file make a variable come into being right the variable was Brought in from outside inside the jq command line the variable came from the terminal on the terminal. I went minus minus arc No, you wrote select any dot value equals equals dollar breach name So you invented that that was where you declared it No, it isn't it was declared on the terminal when I said jq minus minus arc If I had not done the minus minus arc that would have been an error, right? I created the value on the terminal when I said minus minus arc or minus minus slip file That they were created on the command line With those two flags Okay, I'm gonna let him off on a technicality What's an important technicality because you do not know how to make and you could not make a variable called dollar pancake Anywhere but on the command line at the moment you couldn't make one come into being inside the jq file Well, okay. Hang on. Hang on. I'm gonna still see it looks like there is one on the command line You created or you used breach details was a To represent this json file But in this in the um jq you said equals equals dollar breach name Right, but look to the left of minus minus slip file is minus minus arc Where I made breach name I said breach name becomes dropbox. Slip file becomes breach details No, um I'm definitely We are looking at different lines because the line right above where you use dollar breach name does not have breach name in it So I must be okay, but that's that jq. Okay, so that jq file doesn't run until the term until the command line Down at the bottom, right? We don't actually run that file until down at the bottom. So that The variable is written in the file, but it doesn't have a value Okay, there's run the jq now. I see there's another command line Command after you describe the file. I was reading the one above it. Okay okay We will I understand the technicality and we will allow it Yes, and it's that second kind the technicality kind is the kind of variable you only need for very advanced things Like trying to reorder a lookup table Where the value is an array of values. It is impossible to reindex that without a variable As we will discover next time and as I discovered yesterday and spent two hours Being quite cranky and then going. Oh, I'm missing something. What am I missing? Oh, yeah variables And then I learned all of that variables and now now I understand and now I know why they why they are what they are anyway I'm starting to ramble, which means it's getting late and I should wrap up. So Next time we're doing variables and then we are finally going to stop exploding our tables So no more exploding no more exploding arrays in four weeks time Until then we keep exploding them and collecting the pieces Okay, that sounds good. This is this is really interesting bard. I liked it Excellent. Well until next time happy computing If you learn as much from bard each week as I do I'd like you to go over to let's dash talk dot ie And press one of the buttons over there to help support him He does 98% of the work here I'm just the stooge that listens to him and asks the dumb questions If you go over to let's dash talk dot ie you can support him on patreon You can donate via paypal or you can use one of his referral links I really hope you'll go over and help him out in the meantime You can contact me at pod feet or check out all of the shows we do over there over at pod feet dot com Thanks for listening and stay subscribed


I love Tristan, do you know what I mean? I resent dishonest f***s. It's just all processed sh***. If they're all f***ing chemical, we're not beyond that. What are you the f*** bag of beans then, dude? That's some matrix sh***. They rescue those animals from... How does that feel? How's it going, brother? What's up, I'm the boss of Tristan. Alright, cool man. Interesting conversation. Do you agree with him? He doesn't care about animal cruelty if you eat him. Don't care about chickens. I like what him and Andrew stand for. In other regards, but in fact that he doesn't care about animals, he just puts a blind spot for farmed animals because he wants to eat them. I think that's pretty f***ed up. I've always wanted to have a debate with you, but the trouble is I care about animals, but I eat meat, so I don't.  You care about animals, you've got to align your... what you believe with what you do. It's such a tricky conversation because obviously my partner cares about animals as well, because it's called a burger. That's what it is, you're disconnected.  And that's just why I'm vegan. But besides feeling morally good with yourself about not eating meat, if I walk away from you right now and I'm like, I'm never eating meat again, it's not changing anything like that. Of course it has. So when you think of a sort of blind demand system, if you buy whatever the product is, that's the demand for that product. So you've simply cut off your demand for the animals that will be torturing slaughtered for you. So it's a big deal. But only on my scale. So don't eat the meat, it's already on the shelf in the packet. Obviously you can't stop everyone from eating meat right now, but if every individual had your attitude, nothing would ever change, right? And there's still a direct connection between the animals that are slaughtered and killed to you. If I don't buy it though, someone else will buy it. And that will lead to your demand. What I'm saying is what you're preaching makes sense on a mass scale.  And you would be one of those individuals. I'm saying you make a difference for the animals slaughtered for you. Imagine if I said you can't stop all the dog cruelty in the world. Impossible. So what does my abuse of dogs matter?  I haven't got the right attitude to make change. And it's your personal accountability, bro. I'm sure your conscience holds you accountable for direct violence. Yeah, but not like minor. Disconnect, but you've got to make the connection. That's happening because of you. You pay for that. Did you used to eat meat? Yeah, how old are you? 24 today. Happy birthday. So I ate meat for 26 years. Really what, like McDonald's? What happened to make you? My life story is a bit... You watch a Tristan thing? I was in gangs and I was in drugs and I went to prison, got sober.   I wanted to make a difference and I decided to go vegan. It's one thing to go vegan, but what made you like f***ing take it to the max? So basically, imagine I'm going out in a neutral kind of position. So right now you're abusing animals and killing them through your choices. Being vegan means you remove your support for that. So you're neutral at that point. So what I do now is one step further, I try to help animals which is a positive position. It's like I'm doing something more. I'm stopping you and you and you and you through helping you make the connection to maximise my impact. And you can do that too. If you care enough. I f***ing love Tristan. Do you know what I mean? When I was watching it and he went I don't care about animals. I don't care about this topic at all. I've got friends that I don't agree with some things they do but they're still your friend.  So what he does is he says I don't care about all animals in every case and I only care about humans because God created us. I would say a human is made in the image of God. That's what I would say and a pig is not made in the image of God. I love dogs but if you wanted to eat your dog I would not have a problem with you eating your dog. You didn't watch the discussion then? No, I did. I talked to him about that.  I'm not going to tell them not to and I don't care what they do with their dogs. I don't care what other people do, essentially. So there's a very dismissive kind of attitude about it. I've been to Hong Kong, talking about this.  No, but I plan to do it. I don't see it as morally different to pigs in cages. It's a f***ing odd though eating a fish seems so insignificant but do you think that's cruel or not? Do you think a fish is feeling that? I'd feel it, yeah. So they're having their own experience and you might feel like it's insignificant compared to your experience, such as human, robust and you have all these different concepts in your mind and future and past and present.  They're quite intelligent, they build their own communities, they communicate with it, they use tools. There's a book called What a Fish Knows Fish are. It's not the fact that they're smart, it's the fact that they're sentient. There's a difference, right? Sentient being like, attached to like, I don't know, I don't know what it is. So basically, like you see how you're experiencing the world, you've got vision, you've got thoughts, you've got this sensation of feeling. You know, this is all sentience. Consciousness. That's what has moral value, not intelligence. Which is like a logic and computer, f*** it like that. Human rights is a smart human, don't they? That's what human rights are, universal. So these animals are also sentient. Their experience matters to them, so they deserve rights as what? Do you believe they deserve rights? I suppose everyone does. Without rights, they're not protected. Yeah, true. I did see a pretty wild video the other day.  So they experienced happiness, joy, sorrow and making and suffering and fear.  So I watch it and I'm happy that they're happy, at the same time I eat f***ing birds.  I'm pretty sure of your Piers Morgan one. Didn't you say like, if you eat... I don't know, I don't know. Not consciously know, that's what I said.  No, not consciously, you don't know, but now you know what happens to animals in slaughterhouses so you can have milk in your coffee and what are you going to do about it? Because if you don't know, where is your duty to act, but those who know have a duty to act, you know. So now you're at war with your conscience. Yeah, do you resent people that he may or do you think he's that choice? I resent dishonest f***ers who try to justify it with lies. As cool and callous as Tristan Tate is, he's honest, he's not technically as hypocritical as other people because he has this standard across the board. It's the one to know how bad it is and lie and try to justify it with bulls*** saying grass fields, pain, don't you think?  But when someone like, you care, you know what's wrong and you just haven't made that change, that full connection yet, I don't resent those people because they're people that I only hope, they're animals only hope.  Which I think is a more honest approach than I think it's insane. I honestly think it's callous, it's cold. It is. But you are, I don't know man it feels like you're wrestling with this idea like you see animals happy and you feel it but it doesn't matter how you feel man. It's hard man, I always help an animal or like save an animal and not eat me. Have you seen a sanctuary before where they rescue pigs and chickens from there's one there where I live called race day I believe. Well they rescue those animals from from meat eaters because your demand is causing them to be in these places and to get slaughtered. They've been offered sanctuary from the choices of meat eaters. Yeah, yeah. How does that feel? It's just, it's the meat industry, billion pound industry.  You think they like actually come propaganda 100% they do. There's actually people they pay. It's the matrix and you know and Tate's all going about the matrix and all that.  If you look at this display of the matrix then this, all you get to see is green grasses and pastures and humane happy labels and when you look behind the label this is what it is it's decoding the matrix, it's the matrix. It is not some like a pack of eggs and you see a chicken raising. Free range eggs here most of them are free range it's 55%. When you look at a free range farm it's just a factory farm dude. It's a big factory farm. The eggs still say free range. So that's not free range then. To the guidelines, yes. They have to have access to their outdoors but they don't get it because there's so many in this pecking order and they can't get out but they have to give them access so they just have to have windows there and they don't have to open them.  It's wise. It's good what you're doing anyway. Are you going to continue to make my job harder because if you don't go vegan I'm going to have to work twice as hard. Do you know what it is as well? I know a little bit about nutrition and stuff. You know like the carnival diet and how nutritious meat is. You know like on this, I don't know what it's called now but you know like Fakern and like the fake beef and the fake burgers. On the ingredients list it's got like it just don't... There's more words in an ingredient list than if you buy beef. It's just all processed. Well wait a second just because there's ingredients in a plant based burger that doesn't mean they're unhealthy because when you break down beef into its chemical components the ingredients list is actually bigger. Every food is made up of chemicals. So if you take a beyond burger it might have pea protein, isolate, sunflower oil, yadda yadda yadda it'll be a list like that. Beet root juice because they are making that plant based meat out of its components. That's why they put it on the label. Now beef has beef but if you broke it down to its chemical components the list is like massive. Shout out to a lifting vegan logic. Now what these carnival quacks try to do is say oh there's this one ingredient in this food and therefore the whole food is poisonous. They are unscientific there's no evidence to support a carnival diet in fact the only evidence is bad for the carnival diet and there's plenty of evidence to support a whole foods plant based diet. I've heard all this like you know red meats, carcinogen, red meat. That's a flip side to the coin though that they say that because they don't want you to actually eat it because it is neutral. What do you mean? All you see is meat advertisements so they didn't want you to eat it they would ban meat advertisements. Why don't they ban meat if they don't want you to eat it? Why has everyone always eaten meat all the propaganda is meat focus all the advertisements meat focus it's a much bigger industry than veganism is. In fact they want you eating meat. It's a much bigger industry there's a much more vested interest in meat, dairy and eggs it's in everything. It's a class 2 red meat It's a class 2 B unprocessed red meat. Processed meat is a class 1 carcinogen.  No. It means it's a probable cause of cancer. I'm not a health promoter I'm not here for your health. You can be healthy and plant based you can eat meat and also if you increase your fiber eat a lot of whole vegetables you can displace the negative components of meat. But not of saturated fat in the meat that's a heart disease risk but you can be reasonably healthy if you have a so I'm not here to say if you ate fish I'm not that kind of liar. What I'm saying is that fish are sentient beings cows are sentient beings they deserve respect you can be healthy and plant based if you supplement with B12 which is easy and how is it justified for a sandwich these are f***ing living beings imagine if we were doing this to people That's my argument and the people go with this all f***ing chemical eat a f***ing bag of beans then dude I don't know how the f*** is that justified anyway look it's cut in this f***ing head off for a lamb chop dude that's right how the f*** could you do that though yourself I couldn't actually do that the only time I can do something like kill is if I really had to but not an innocent being man not an innocent being I don't understand how someone could not you for trying to do a positive message but you know like I think it's amazing what you're doing I'll stop an animal from being directly abused I'm not going to stop eating meat but obviously you probably will present for that yeah I think what you're doing right I think you're deliberately I think you're not ignorant now I think you've got enough information to make the change what I think is you're being purposely cruel now well purposely deliberately careless towards beings that can't defend themselves from your choices right 10 years goes down the track that's a lot of blood man that's a lot of suffering, that's a lot of killing and you know sometimes you've got to stand for something man to be honest I'm quite like pushing Andrew Tate's agenda positive Tristan said it's got enough to do with being a man I'm not one of these guys who says oh you have to eat meat to be a man being a man is about living by your own convictions standing up and saying this is what I believe in being ready to challenge people in their beliefs being able to stand up say what you mean mean what you say that's what being a man is about standing up for what you believe in saying truly your principles that makes you a f***ing man defending your family if you need to defending your partner if you have to these things that make you a protector these things are innocent so it doesn't make us a man to abuse the innocent you've got a level disconnect but you're doing something horrible to them they can't defend themselves from you I'm indirectly so but if I pay my mate to come and bash your head in right now that's indirect but I f***ing know what's going to happen if I pay him what happens all the time and they get just as much sentence for a hit, for a conspiracy if I tell someone to bash you if I go hey mate go get that f***ing I'm just as much in trouble so you're just as much in trouble morally for the disconnect you're doing it I get what you're saying I think maybe this might sound really f***ing stupid and I might get saying it but maybe if there was like a bit more awareness on what's actually happening if I put human meat on that product you would not buy it in a second you'd be like f*** that if I put dog meat on that packet you would not buy it in a second why do you hate these animals so much think about what the word hate means it can mean anger but it can also mean prejudice I think it's just kind of like growing up with that's the norm isn't it it's how bad it's disconnect but now you wouldn't buy it would you because you don't eat dogs and why don't you eat dogs there's something about your dog what's your dog's name Nova and if I killed your dog you'd be upset and you'd probably want to kill me or something bad to happen to me because I've taken someone from you and what happens to Nova matters to Nova if I boot Nova it's not bad to boot them look at her as an individual if someone killed her or harmed her I'd happily sit in a cell these are all individuals here like Nova should you sit in a cell every time you pay for them to get what if I paid someone to kill Nova should I sit in a cell the excuse doesn't work really you have to either say look I know that I'm a f***ing cruel I don't care enough to f*** just buy something else walk five minutes to buy something else have you ever eaten meat before? yes he has I used to be a massive meat eater I was also a violent gang member I was eating steaks and I was disconnected but if I seen someone like my mate showed me a hunting video once and I was like why are you showing me that f***ing poor bro and I was eating meat eater then but I was like do you think I think that's f***ing funny dude yeah because there's videos on Facebook of people chasing like they like hogs around with like assault rifles on trucks they're like shooting them and they're like squealing the sauce is for pleasure you like the taste of chicken? I don't eat meat for pleasure I do it for nutrition seriously? do you seriously don't do it for taste? what nutrition are you getting out of a chicken? that you can't get from plants? I think it's just the low fat high protein there's low fat high protein sources of plant protein as well of course you've got guinea beans no no bro you've got tofu, you've got satan you've got mock meat to be honest, nutrition wise I've meant more red meat to be honest with you copper, iron, magnesium, potassium phosphorus, selenium I can't pronounce some of these chaline, omega 3 yeah you can get all that f*** on plants but that is a nutritious bit of food though I mean there's also a bunch of fat cholesterol there's a bunch of other things in it and red meat's got bad associated with bad health outcomes when you look at population data some of the best data has been done on red meat itself and that's why it's something they teach you to minimise but yeah you can get a bunch of nutrition from human meat as well I'm sure it's got exactly the same minerals maybe more yeah I agree with you yeah keep up what you're doing anyway it's good work I know you don't want to hear it right but it's better than nothing I can't stop eating meat but I'll try a bit of tofu that's all I can promise you that's really nice talking to you you'll be vegan soon