Mijn naam is Philip, een paar jaar geleden begon ik te spelen met de DNS, want ik dacht dat het een echt cool lieverde was. Er zijn er dus plenties van talen bij Willem over DNS, maar dat is meestal hoe het met de overal DNS-infrastructuur gaat.  De andere prootjes, en op hetzelfde tijd ben ik echt interesserend in de DNS-locale vervaring en ook gebruik van stornige knie-materiaal. Dus ik dacht, ik moet dat in één tal verbinden. Ik dacht dat mensen weten over de DNS en de DNS-sack, dus kippen introducties op dat. Ik maakte een comparenslijt over hoe de DNS-sack is, compared met het normale PKI systeem. De DNS-sack heeft een route van geluid, een geluid van geluid die alles schijnt. Alle geluid draait uit dat, waarom het PKI is. Je hebt veel individuele CIs en je kunt gewoon meer hebben. Ik denk dat het echt leuk is dat je de route van geluid, de register en de registreerd moet geluiden.  En dan heb je het gedaan, want met PKI hebben je zoveel individuele CIs. Dat kan alle certificaten voor je domain zijn. De hele communiteit is er constant vervaring de CIs om te zorgen dat je het goed doet. De andere ding met de DNS is geluid is volledig delegatie. Als je je domain hebt, en er is een trustanker in de parentzone, dan kan je wat je wilt doen in je domain.  De DNS heeft een praktische, een soort van disadvantage, is dat je moet praten met verschillende partijen om alles wat je nodig hebt.  En PKI heeft dat gesloten in een praktische manier, waar je een certificatieverkant krijgt. Waarschijnlijk heeft de DNS een andere security advantage, maar het requiert meer werk. Recentelijk kwam het op de DNS O-Ark mailing list, ik was in mijn naam, dat is hoe ik het heb gevoerd.  Het is zoals, waarom hebben we een toysystem? Ik denk, kunnen we een reale systeem hebben op zo'n punt? Je kunt niet zeggen, ik heb de keymateriaal in het DNS-sec, en ik zeg, oh ja, maar ergens in dit link is 1024-bit RSA-kies, dat is niet de manier waar het moet zijn. En daar, at least de web browser community was veel meer regressief en zei, well, this is insecure, we're just not doing that anymore, and the world has quickly basically deleted all insertive secure ciphers. So, if you do DNS-sec, then you have to verify stuff, and you want to do that locally. I mean, if you let your ISP's resolver do a DNS-sec verification, then do you know if they're a good job, do you know? I mean, it's from a trust point of view, it's just not a good idea. So, you can do two things, the easiest way is to re-cursor a resolver locally, but it has very limited functionality, because in the DNS protocol, basically, you can say it checks or it doesn't check, and you don't know anything else, and it's usually set up, it's very fragile, because then your application reads et cetera, resolve.conf, where there should be a local IP address, but if it can change, suddenly you lose your security. So, I think it's way better if you want to do a DNS-sec and rely on the results to use a DNS-sec validating library that is integrated with your application. So, a number of years ago, people wanted to have a new DNS API, not specific for DNS-sec, but basically to have a more modern way of sending and receiving DNS packets. And the other thing is that most existing DNS solutions have no support for LibEvent or other event-based systems, and that was added to the GetDNS API, but it does support DNS-sec. It's more modern in the way that it's modeled after Python and JavaScript, so you get lists and dictionaries, and basically those data structures can always change without having to introduce new API calls, and that makes it extensible. And the thing I want to point out is there's an API specification and there's a reference implementation, and I, well, just mix the two. Assume that the reference implementation is the API, even though in theory, of course, you could have multiple implementations. Well, lots of text, but to highlight, that's at the top. You create a context, but that's a normal way of doing things with libraries. The way they solved the LibEvent thing is that there can be multiple routines for specific event libraries that hook into the GetDNS API to make the event stuff work, and then there's two sort of the most general calls. One is called GetDNS General, and the other is GetDNS General Sync. Sync is the synchronized version for if you don't want to use an event system and you just gave it a name and the DNS type that you want, and then at some point it says, well, I'm done, and then you have a result structure. And because result structure is like, say, JSON, the API is basically, we'll get an element from an array, look up an element in a dictionary. There's two basic data types. One is an integer and the other is binary data. So basically these calls here are the only other calls that you need to know to get DNS to know how to use the API. So from that point of view, it's very nice. It can be just a simple, stable set of calls that you don't need to change ever. There is, however, I mean, we always get complexity back in a different way. And so GetDNS creates a rather complex namespace because it's a namespace, you can always put extra stuff there. Because well, if you just don't look, then it doesn't work. So you have sort of the traditional DNS things like a decoded DNS reply packet. But as you can see at the top, it has a field, just address answers. And then it gives the addresses as a convenience for, well, if you were interested in the answers, the addresses, then you find them there and you don't have to know about what actual DNS looks like. And of course, you can always make changes to that and say, well, if we want to have a simple way of getting key material or something like that, you could just say, well, this is what you're actually looking for and then here is also the DNS stuff in case you want to know that as well. But it will never affect the actual API calls that you need to make. So, then I get to my first example. Well, I definitely do it. I mean, you see, well, there's an unknown fingerprint. What do you want to do? Do you want to continue? Yes. Who do I ask whether this is the right or not? Do I have time to ask anybody? No, I just continue. If it looks like the right server, it's probably good. So, and you can come up with all kinds of ways of storing fingerprints in a web server or a local wiki or whatever. But my practice is nobody actually does it. So, the obvious thing is, well, why don't you store it in DNS and then try to use that? So, the SSHFP record was created a long time ago and there's actual code in OpenSSH today that uses that. And that's, by default, enabled in FreeBSD. And so, I thought, well, I should play with that. I was really excited about that. And then I found out that it really doesn't work at all. And if you read the code here, then you can see this is sort of a more traditional way of having a DNS library. And then you see at some point that you have an LDNS RR list, RR count thing. It's very much tied to how a DNS packet structure works. You get it from this section and you get it there. And only if you know what a DNS packet is, you can use it. But the code itself, I mean, this code is perfectly fine. I mean, I don't know who wrote it, but there's nothing wrong with it. But it still doesn't work. Yeah. And if you look at... So, then I started, after I found out what the issue is and why it's not going to be fixed, I thought, well, this really cool new thing at DNS, I started working on that. And so, this is basically the first piece of get the DNS code that I wrote. So, there's a couple of things at the top. You can set some extensions and that goes in a generic way because you give a string and you put some values in the string. So, you can always add more strings without changing the API. You do the actual call. And then when I started using get DNS, you actually had to walk the data structure one step at a time. So, you say, well, there's a thing called reply tree. So, get the object of the reply tree. And then that has a subfield and et cetera. So, then I complained about it and then later more support was added to say, well, I want directly this field and know what it is called. So, but basically... So, it does the same thing as the LDNS version. But the difference is that you don't have to know as much about DNS. You have to know about how the get DNS reply structure looks. But if there's convenient functions, then you don't have to understand DNS at all. But now we get to where it went wrong. Is that the LDNS library is just completely standalone library. It doesn't know anything about the world.  But if it needs to do DNSSEC validation, it needs to have a trust anchor. So, LDNS library says, well, tell me where the trust anchor is and then I will start doing that.  I mean, it should know where to find the trust anchor. We cannot set code in OpenSSH to tell it about trust anchors. So, the net result was, even though all the mechanics were there because of the lack of the trust anchor, it would never do any DNSSEC validation. And then there's an environment variable that LDNS listens to. So, if you have actually set the right environment variable, then it will actually do the validation. But yeah, that's a completely ridiculous way of doing things. So, fortunately, getDNS has a completely different approach. And it's like, well, it should work. And you can set some extensions where it works harder to make it work, like the roadblock affinities feature. But one thing that getDNS also added, sort of around the time of the route K is rolled over, is to directly fetch trust anchors itself from IANA, which is operated by ICANN. So, there it's always feel a bit uneasy about it. It's like, it's something that really works. But you just go to a random website and you get some stuff. And there's some certificate that signs that it's true. En, well, the thing is, it's a certificate. At technical point, it's all fine. But I don't know, maybe anybody at ICANN can edit this file. But at the same time, you have to trust ICANN anyhow, because well, they operate the route. So, yeah. But it shows the difference in philosophy. So, getDNS, it tries to make it work where LDNS and sort of most of the other older DNS libraries, they're very strict in like, well, we do DNS type of things and we're not go out of the way to make it work. A second example is TLSA. I really like TLSA for mail servers. Because one thing is, well, there's no real way that you can enforce a secure connection between mail servers. Because, well, how would you tell? I mean, you can ask the mail server to support start TLS. But yeah, men in the middle can just do a downgrade attack. But if you put a TLSA record in DNS, then it's a very clear statement of the operator of the receiving mail server. It's like, yes, I want TLS to be used. And this is my certificate. That's the other problem with encrypted email because everybody started out with self-signed certificates. There's no way that you can now simply transition. All certificates need to be signed by a CA because their mail would break. And if you do it in TLSA, then there's explicit statements like, well, this is my certificate or this is my CA. Then, as a sort of a hobby project, I'm trying to contain, isolate security things in separate programs. It's mostly for if you have small one-off things and you don't want to figure out, for example, how SSL works. So I created what I call the TLS helper, which basically does all the TLS encoded, whatever, and you just talk to a pipe to it. So then when I started thinking about Dain, it was like, well, maybe I should also create some sort of thing that does the same thing for TLSA lookups. So what it basically does, it tries to gather all the relevant information you need to send mail in one output JSON. And then the mail client can then use that information to set up a secure connection. Or it can know that it is not going to be a secure connection. So this is, well, FOSDEM doesn't have a TLSA record. But it does have DNSSEC. So in all of the fields, it says, well, DNSSEC secure is true. But then you get, for the TLSA record, you get no such domain. So, well, my favorite example for some people who do that.  So there you see at the end that you get the key material right at the bottom. So as an application, if you would use that, then you don't have to be aware of anything related to DNS anymore. You just look at the right fields. But mostly this is also a hobby project to play more with this kind of stuff. A few code examples, this code is written, I don't know, a year and a half later than the first one. For some reason, so now I have a lot more extensions that I enable, because there's certain information that I get the DNS gives back by default. And if you want to have more fields, because I want to check the DNSSEC status, then you just pass in more extensions. En because it's all strings, you could basically store it anywhere. So that's the thing that you use to configure it. And then the way you extract stuff is you can see here at the bottom the change that was made. Because now you have a path to something that you want. So if you directly know, I want to look there, then you can ask for that. And so this is all straightforward. It's like, well, you have to have the conceptual model, what the DNS output data structure looks like.  But I've found one piece of code. And I remember that it actually took me quite a while to get right. En dat is, I also need to know whether the call succeeded or not. And apparently something succeeding or not has an endless number of possibilities. So I don't have no clear idea how to prove it. Maybe this is just how it should be. But it feels like this code is too long. So you basically have to first look, did the actual DNS call succeed or not? Because well, it's possible that you didn't get a reply packet. And you have a timeout or something like that. Then you have the issue that the DNS result could be no such domain or some other error code. And then you have a DNSSEC status, which could be secure, insecure, bogus, et cetera. And you have to have so many combinations that you all sort of have to figure out. It's like, well, if the DNS call didn't work at all, then the others won't be there. I think there's one case where you don't get an answer, but you have to treat it as an answer or continue anyhow. So I found this to be a bit of a minefield. But other than that, it seemed to be all very straightforward. So that's the end of what I wanted to show here. I think that if you would write any sort of DNS client code, then it's definitely worth looking into getDNS, because it's a very stable API, very extensible. Well, I think that definitely from the security point of view that the world needs more DNSSEC. So everybody who doesn't have DNSSEC signs on should immediately do that. Well, one thing is how do we get rid of the weak keys that are there? So one thing could be that if maybe the open source community says, well, one thing could be if getDNS has an option to say, well, a 1024 bit key is not secure, then the DNSSEC validator could say, well, this is either bogus or insecure or something like that. I mean, can easily be added as an extension. But the community could also go further and just say, well, in so many years all validators will just say, well, 1024 bit is just not secure. Well, it's just the default. We don't care anymore. Please registries do something sensible and have real security. Then I think that if those issues can be fixed, then I think DNSSEC and getDNS provide really cool opportunities for key distribution. There's probably a lot of other key distribution applications where people create really complex systems to distribute keys. But if you could rely on DNS to just do it, then you put them in DNS and you're in business. So that's my presentation. On the JSON earlier, I saw you had them in two fields. There was a DNS insecure and a DNS secure. Just to understand what's the reason behind the design. Does that just have the one field in total? No, because. Oh, so the question is, on this JSON output, I have two fields, one called DNSSEC secure and the other one is DNSSEC insecure. And that's because you have three values. If it's secure, then there's a signature chain. If it's insecure, then at some point there's no DS record. So you know that the child zone is not signed. But there's also the bogus status where you can fail to verify it. And of course, if you want to send mail, then you probably, well, it's up to the mail client to decide what to do in a bogus status. And so I wanted to have an explicit test like, well, is it secure that we could test for that? Is it insecure that we know? And then, yes, you have the other stuff. For this one, maybe we have to talk about this one, that this somehow can get incorporated in the library, the ICPB, if it's. Yeah. But we thought of things like for TLSA, there's extension return unsecure answers. Because if it's insecure of before it gets bogus, or if it's insecure, then the TLSA is not secure. No, but the thing is, I wanted to have sort of the generic helper program, dat zei het, well, if you want to know something about this site for sending mail, tell me about this site. So I could just say, well, it was not secure, so you don't get any answer. I want to say, well, it's insecure or bogus, and these are also the answers, because you may still decide to send the mail anyhow. But the flag is basically, if it's not secure, then you don't get any answers or something like that. It was, it did something other than what I wanted. Yes. The question is, did I reach out to the OpenSSH community? Yes. So I put my code on GitHub. I mean, I'm really slow in tracking new OpenSSH versions. I sent them a pointer to that, but I think that OpenSSH maintainers don't like this concept at all. So for them, it's like, yeah, there's no priority in changing the code. Well, the weird thing is, is that the LDNS code is in OpenSSH. So at some point, somebody managed to get that merge, and then it's like, let's make it really work. Nah, not really. So that's the question I just answered. So the question is again, why is there two fields, DNS secure, DNS insecure? I can answer it again. It's because there's a third state at this bogus where validation failed. I hope that doesn't happen. Maar zoals je kunt zien van de code dat is nog op de schrijving, waarschijnlijk, ik initialiseerde alle variables properly to zero. Er zou geen code moeten zijn, maar dat zet ze aan.


Alright guys, so we're gonna start off today's video. We're gonna be using Paul Mitchell Blonde skylight with 40 volume plus olaplex We're gonna do a 16th of an ounce of olaplex will not dilute the formula What we're gonna be doing is I have my model grace and we're gonna be popping in some chestnut Highlights so my goal is not to get her super bright. So I'm gonna do a technique that's gonna be nice and subtle Very salon friendly techniques. So I think you guys will be able to use this right away on a lot of different guests We're gonna be doing some teas highlighting, but also some hand painting. So Paul Mitchell skylight has a nice smooth feel to it It's a clay-based lightener. So you get a smooth polish Definitely good coverage great lift. So we're gonna start off by painting Hand painting and like I said, we're gonna tease up the section. So as I work up the side What I wanted to do is not take it all the way to her base So I'm teasing the section and then I'm hand painting what's left basically mid shaft to ends Throughout there. So that's gonna give it more of a diffused look. That's what the teasing does I also don't like to tease the hair a ton Sometimes you'll see people really tease the hair up definitely not against that but for quickness in the salon This is more of a salon reality technique I just tease enough to give it that broken Broken up feel in the mid shaft and then I hand paint it to give it an even softer effect So just going through there soft teas. That's just to diffuse it a little bit Then I go through and hand paint the rest So basically what I've done is I've taken that rectangle off the top and everything underneath that basically from parietal ridge down is Going to be hand painted and teased Now I'm gonna do the same thing on the opposite side. So while we're working on this side I'm gonna talk about the product a little bit the Paul Mitchell skylight is a great clay-based lightener Those of you that are not currently using a hand painting lightener to do this kind of stuff You can attempt it with other lighteners But the great thing about a clay base to gives it a smoother application It also you use the clay to help incubate the section which will give you that lift So as a lot of other lighteners what they'll do is they'll go on the hair They'll dry up so they don't lift as much with a clay-based lightener It's gonna go on the hair it's gonna create kind of an outer shell around the hair shaft and Help incubate that lightener so that it doesn't stop working throughout it Okay, so now we're just finishing up that second panel again Don't do a lot of teasing if you do a ton of teasing you're gonna spend a lot of your time in the bowl Trying to get that teasing out it doesn't make that big of a difference because we're hand painting through it So even though I'm teasing a few times. I was about four times through it. I'm very softly pushing that hair up So now what I'm gonna do is I'm gonna take down. This is like the piece right at her part line So what I want to do with that piece is I want it to be a nice strong chestnut color a little bit brighter right around her face So I'm gonna take that little triangle out of the corner I'm gonna paint right to the scalp in the very front and then feather it back off Towards the back that'll give me that nice bright pop piece, which you'll see in the end result But you can see how it kind of fades back Now we're gonna work through the top like I said, this is a quick technique for in the salon This is not something that should take you a lot of time These are techniques that I really love doing because it has a maximum result with less Time in the chair, which is great for your guests because not everybody wants to spend hours in the chair But also not every guest wants to be completely transformed and blonde either Some people just want to have that little pop or that little bit of richness put in their hair And this is what the technique is gonna do for us. So we're working diagonal forward throughout the very top section So diagonal forward partings over directing everything back What I'm gonna do is I'm gonna work almost like a V shape with the lightener So what that means is I'm gonna come up a little bit higher on each end of the panel and then a little bit deeper through The middle so the middle is not gonna have as much lightener on it Which is gonna give me a more shadowed effect So now we're gonna continue on through the top section and I'm gonna continue doing the same thing So creating that V look to the highlights Maybe get a little bit brighter a little more saturation towards the front But you'll be able to see that V look so see that depth in the middle And then I paint everything else nice and high Towards the top just giving me a more shadowed effect It's almost like back in the when we would do a weave foiling which some of you guys might still do it This is an alternative to a weave foil a little bit more of a natural effect But it's just not painting all of the hair blonde allowing some of that shadowing to be in there So the last little bit I'm gonna paint this is her fringe area So this is really a key area and I'm gonna go much brighter on the very front So you'll see right through that center. I paint I'm gonna paint a lot of this section Completely throughout because that's gonna give me that brightness in the very front of her hair So that is our application the only thing I'm gonna do now is I'll go through the back And I'm gonna do a little bit of hand painting just to Add some light pieces to the back, but I don't want a lot of stripiness or anything like that So we're just gonna go through. I'm gonna take a horizontal panel I'm just gonna hold all over here at once and Basically paint some highlights on their freehand. So wherever I want to see that brightness happen That's where I'm gonna paint and I'll really work the product in so we talked about Saturation the more saturation you get the lighter your highlights are gonna be so in the very back here I'm not giving it as heavy of saturation because I don't want it to be as bright in the back You might see a little more product going on there, but it's also a lot bigger section But I am painting it to a much finer consistency. So it's not really incubating it as much So we won't get as much brightness So the cool thing about this video in when we talk about salon reality is the product that I'm gonna use at The end to create the chestnut color it's actually a product called color craft by Paul Mitchell And the cool thing about this product is it's a customizable conditioner and it has a professional purpose and also has a Take-home purpose as well. So We're gonna be using a couple different colors to customize this color for grace She wanted a chestnut feel to her highlights. There's our end result. You can see there She wanted that chestnut feel so what I'm gonna do is I'm gonna add in two different colors I'm gonna mix them together. So there's two different ways to mix this product The first way is the professional way so we go in here And I add two of the tubes of the custom color treatment And then I'm gonna add about two pumps of each color to the bowl and mix it together The take-home version is that bigger tub of color craft Custom color treatment that you see over there. I'm gonna put a lot less pumps of the color But customize it mix it in there and then sell it as a take-home product that my customer can take home and Maintain their color craft at home. So they use it in the shower. So you have a much more diluted Customized version here and then you have your take-home version as well. So what I want to do is walk you guys through that application now So we're gonna start off. We're gonna rinse Grace's hair. We did put Olaplex in the formula That's our bond multiplier. So I need to finish that process. So I rinse her hair Tau dry it do not shampoo it I apply the step two and I'm gonna let that sit in her hair for about five to ten minutes Depending on the timing in the salon five minutes is fine That's gonna help multiply those bonds keep her hair from breaking and then after you leave that on for five to ten minutes You're gonna rinse it out once you rinse out the step two then you're gonna shampoo with whatever Shampoo you're looking to use for today's video. We use the ultimate color repair shampoo It's an anti-fade quinoa repair shampoo. So it's got lots of protein helps keep vibrancy and strengthen The hair so definitely a good choice and then we go in and we apply our custom color craft treatment So I'm gonna paint that from scalp to ends really heavily saturating the entire Hair because there's a lot of different benefits Not only is it gonna color the highlights that we put in but it's also gonna color the base and it's going to add Condition and shine to the hair check out the end result grace loved it You could see that brightness around the face that little piece that we painted and then just a little soft Movement in the very back. So hope you guys liked the video. Let me know in the comments below


All right, today I wanted to talk about command line pandas killers. So I have a collection of utilities that I turn to when I want to do some simple data manipulation. Now none of these are going to be pandas replacements or dply replacements or data table replacements. None of them are going to completely replace those libraries. But it is interesting to see that in 2021. There are not just one but many tools that can be used to do data manipulation on CSVs and they do it really well. So I'm reviewing some utilities that I think are very good at manipulating CSV files. So I do have a little post here on my website. I'm going to skip some of this if you're interested in what I ended up finding works well and what doesn't work well. You can take a look at some of these tables. One usually means present, well one always means present, zero means it's not present in that library and I also wanted to look specifically at some summary functions just to get a sense of what these different packages can and can't do and just get side by side comparisons. Now I say that but the idea here is that I don't want to discount a package just because maybe it has fewer features because features isn't really the only thing and speed isn't really the only thing. I didn't even look at speed here just because I didn't really want that to affect the results. Sometimes with different tools you might be attracted to it because of more than just its feature set or its speed. Maybe there's a way it expresses operations that you might like. So I just wanted to instead of do a ranking by features or ranking by speed I wanted to do something a little different and give these different tools awards or maybe they're almost more like personalities like what would I how do I feel when I use the tool like what does it make me think of. So the first one on the list and I guess I could go through this really quick that the tools I'm comparing and looking at is XSV, CSVTK, TSVutils, MLR also called Miller and Q. Again all of these are great picking and maybe you shouldn't just pick one I think it'd be useful to to get to know a handful of these and explore them. All right so the first one XSV I gave this the Stonehenge Award. The reason is that this tool just feels really rock solid. And the source code is clean, compiled, memory safe and I think that this tool definitely will stand the test of time. It's not going anywhere. It could do the standard things that I would normally look for especially when it comes to things like summary statistics, filters, selects, joins. It can do all of that. Really quick I do have a file on my GitHub repository where I take a dataset it's a data slash weather dot csv. I also have a shell script that just pulls this but I take this file and I manipulate it with these different libraries so you can get a sense of how these different libraries have their own options, flags, grammars for manipulating a csv file. So for example if you want to do some summary statistics on a csv file which I'm calling df so dollar sign df is environment variable. All you do is xsv stats use this s option and select temperature and humidity and from this I'm getting a handful of summary statistics. And this is really concise which I like and pardon me kind of doesn't like. The thing I don't like is that I feel it would be nice to use a very specific summary function not just this everything option but that's just what xsv gives you. So that's xsv for summary functions. I have a few other things that I really liked with xsv. One thing I thought it did very well was joins. I thought it did joins probably it felt better. Joins felt better with xsv. So here I have a left join with xsv all you do is type this join option with this left flag and you say which column number you want to join on for two csv's is really straightforward. So I liked xsv because it was really solid really great code source code. And when it comes to things like filters join selects it does everything in a very clean concise way. So that's xsv great tool. All right the next one csv tk the word for this one is the order. That's that's like someone who likes to speak or elaborate or expound. The reason why I gave it this word is because I think that it gets a lot of things right in terms of the verbs it uses. So if you if you're familiar with a deep player library you'll be used to verbs like summary, rename, mutate and csv tk has these same verbs. So for example doing a summary statistic csv tk summary dash f for the field you want to select and then you get to say each for each column and each function you want to do a summary on you can specify that in csv tk. But it's not just the summary function that's cool. There's also let's see maybe I should just do a search here. All right so mutate mutate to. So this lets you make a new column and I think the way it does it is really clean.  I'd have to look at the documentation remember I did this I did this a few days ago and I can't really remember the exact name or purpose of the dash flag. But what I'm doing here is I'm sitting I'm using this dollar sign temp and saying hey if dollar sign temp which is a column you call columns with a dollar sign and csv tk if it's greater than 32 degrees call it liquid else solid and we're going to name this new column water phase. So I thought that was done really well. There's other examples of things like that in csv tk so it also does joins well. The thing that you have to watch out for on csv tk is how it handles n a's and so this is the one thing I wasn't too much of a fan of with csv tk but csv tk does not remove n a's by default. It does have a flag that allows you to do this but it's not default. So in xsv ignoring n a's in every case that I tested was a default operation with csv tk it's not so you have to just use a flag to say hey if there's n a's remove them. But yeah I just really liked the way that csv tk expressed its operations. So that's csv tk tsv utils so the award here is the ranger and I call it the ranger for two reasons. One is that the utilities it's not just one package so csv tk is part of csv tk. You call csv tk and you say something like I want to do a summary or I want to do a mutate. With csv utils you call something like csv select that's its own binary or tsv filter that's its own binary and I actually really like this. I thought it made it I thought I thought it was a good way to look through things like the documentation the man pages had a good sense of what I needed to do without filtering through too many lines of of text and in a man page or read me. So let me see if I can find some examples where csv tk seemed to shine. All right so here's one csv summarize so this is going to be a summary function. Header is just saying we're using a file that has a header group by is saying what we want to group by so this is pretty pretty clean and we're grouping by origin and what we want to do is a count and that's it. So so that would be a count grouped by origin for this for this weather data set. Let's see. There's some others down here tsv select. I mean this is this is very clear. And I don't think I mean this is probably if I were to implement a a similar package this is how I how I would imagine implementing it. It's just so clean tsv select with a header file using these fields origin year month temperature from the tsv data frame. And then we can pipe it to another utility they have which is tsv pretty tsv pretty just makes the csv file look nice. And I guess I should mention here I'm looking at some of these other lines. Lots of times I'm piping the output of the csv to a application called TV which is short for tidy viewer and this is also a csv pretty printer. Since tsv has its own I use tsv pretty for for when I was working with tsv functions but when it came to the other command line utilities I was always piping to TV really nice really nice package for for csv pretty printing. So that's tsv select and the tsv utilities package. I thought that it was really unique that it was able to group different binaries in their own in their own package. So I like that a lot was the way I thought of it is like a rangers quiver of arrows. You know that's that's kind of the idea. These are the different binaries that the tsv utilities use uses. The thing I didn't like is that you cannot use csv's on tsv utilities. They have to be tsv's and they have some good reasons for doing it. But this is specifically looking at csv data manipulators. So when I wanted to use tsv utilities I used a function that they also provide another arrow in their quiver csv to tsv. So you have to convert your csv's to tsv and then you can use tsv utilities on on that file. So that's tsv utilities the ranger because it's got different packages for its different functionality. And I think it veers off the the standard path by having all of its operations work only on on tsv files which has its its pros and cons.  Probably I didn't even look at source code but if I had to guess Miller would be probably an order of magnitude more more code than the others. And it's because it does a lot of stuff and it does a lot of stuff very well. So let me let me pull up Miller. And I guess first what I should say I don't have an example of using some of the functions that gives it the scientist title. But the examples that I have here is that you can do linear regression with MLR. You can also bootstrap with MLR. And that's just the beginning. MLR is kind of its own in its own league. And for many applications you may not need all that power right. And maybe you know something like CSV TK or XSV or TSV details is is perfect for what you need. Miller is when you want to do something a little bit more heavy at the command line and I think it does it feels its role well. It feels that niche well. So let's see where could I find a good Miller example. OK so here's one Miller with Miller you have to add these flags. I guess yeah flags. You're saying the input is a CSV the output is going to be a CSV and you're going to filter and most things in Miller are expressions. So you'll have things quoted and the way you select columns is with this dollar sign a lot like a lot like with what we saw with CSV TK. So you do I want to filter where origin is equal to LGA month is equal to six days equal to one and then I just pipe that to TV. That's one example. Let's see what else we have. This is this is a sort sorts are a little bit different on the command line utilities. Well they're different from what you might think. If you're used to using libraries when using command line utilities there's these flags like dash and our dash and and these different flags will say you want to do things in reverse order or you want to do something reverse order with numerical values or you want to do something reverse order with lexical values. So there's this this nr flag is something that's common within the CLI data manipulation tools but maybe something you aren't used to seeing with dplyr or pandas. So a little bit different there. Let's see. What else do we have. I really liked how powerful Miller was with making new columns making new columns isn't something that every command line utility has. So and Miller does it very well. You name the new column and then you can also input different columns as as inputs or arguments to a function. So here I just have a feature is equal to temperature over humidity. And in my mind that was really clear very concise. I think it does a great job at making new columns. So again I didn't show the full power of Miller and it's just because I'm trying to compare similar features across all these command line utilities. And if I were to go into some of the features that make Miller the scientist it would be the only tool that would have things like bootstrapping linear regression covariance calculations things like that. So if you are interested in doing more statistics type of analysis at the command line definitely look more into Miller. But even even without that even without the statistics portion it does lots of other things really well filters new columns things like that. Miller is great. Q Q is very popular and I've seen Q mentioned a number of times on my YouTube channel. People seem to really like Q and so I gave it the title defense of the agents. Dota right. It's a it's something that's using a grammar that already exists. And I think that's really cool. I don't think it's anything to scoff at you know it it's it's it's unique in that way. No other tools are really using SQL like expressions to do data manipulation. So let me show what I mean by that. So let's see if you know SQL Q is going to be a breeze to use. OK. So here I'm I'm doing a replace for for NA. So this is just said looking for NA's and replacing it with nothing. All right. That's it. That's an easy way to to take care of NAZ. It's not something that you that's unique to Q many different command line utilities need this search and replace for NA's. But anyways once once I do a search and replace for NA's I can pipe that to Q with Q I'm just saying hey I'm using a header file the delimitator is a comma comma and what I'm going to do here is I'm going to select these different columns do a sum or sum in max average count and group by origin. So this is you can see this is just just like SQL. This is how you'd express in SQL. And the cool thing with Q is that lots of times I didn't even look at the documentation right. I just tried typing what I would do if I was writing SQL and most of the time it worked. And so that's really cool about Q that that's a cool feature with Q. Filters are just like they are in SQL. So use that where clause. There's another one. Let's see sorting just like just like SQL. So joins Q does well also. So inner joins left joins. It seemed to do it seemed to do great. Surprisingly I there wasn't like a union clause. So in SQL when you want to do something like append more rows onto a data frame or append columns onto a data frame. So maybe if you're or you can think of this as bind columns or bind rows Q didn't have a way of handling that. And frankly most of tools didn't have a way of handling that. When it came to these table to table operations it seemed like XSV was the one that was that was able to do some of these appending by rows or pending by columns. So that wasn't available on Q. So Q doesn't have everything SQL has but for lots of things you would normally do Q has a lot. So I call it defense of the ancients because SQL is one of the first ways we had of expressing operations on data sets and Q sticks to that. And I think that's great. So that is all I've got for these different command line utilities. Again every single one of them are great. They're open source. The developers you can tell spend a lot of time on each of them. So I would say if you're interested in doing command line data manipulation pick a handful you know maybe don't just try one but try maybe three or you can try all of them just pick a handful and give it a shot. See how it goes and if you've got any feedback or you feel like there's something I might have missed in these tables provide that feedback in the comments. And that's it. Thanks for watching.