Okay, so this talk is about the verification of mixed net and today verification is a company, but the terms verification mostly people, most people think of verification as the mixed net, so feel free to use that. So before I say something about the mixed net I want to say something about electronic voting in general to give you some context where this fits in. So most electronic voting systems that you hear about, they basically consist of four different components. You would have a way to generate a public key somehow, a way to authenticate voters and check eligibility for voting and a way for forming your encrypted vote and then some way of tallying. And I wish to say something about this. So there are two basic ways of encrypting your vote, so to speak. One way is to have somebody else do it for you in advance and then print it on paper and so on. In this case you need a trusted postal service or a courier or something like that to provide you a piece of paper or in the polling station to provide you a piece of paper. So there are proposals like this, for example, the Prätta Votere system. And the other way is to have a device to encrypt it for you, so like a computer or your phone, in the obvious way. And of course we should worry about viruses and people are worried a lot about that in the e-voting community. So there are a bunch of different clever ways for the voter to verify that the vote was encrypted correctly. And since people are involved in this and notably people that are not so technical, these methods have been studied also with respect to usability. So one way to do it is to have a separate device to basically perform the encryption again and check that the result is the same. This is what they have done in Estonia. Another way is to challenge the encryptor. So you would have a machine that produces a ciphertext and then you can either use that ciphertext or you could tell the machine to show that it encrypted it correctly and verify that. So this has been used in the Wombat system in Israel and this is originally a proposal by Josh Bernalow. The third approach is to use something called a return code, which is basically like a personal MAC of your vote. So you submit your encrypted vote and you get back a private MAC of your vote. So you can see that your intended vote was actually encrypted and submitted. And then you need to verify that it was recorded so that the authorities don't remove your votes. So this can be done basically what you need is a bulletin board. And today we have all these cryptocurrencies which are perfect for doing this. So you can have a local chain that you now and then link into the great chain online. And this allows voters to verify that their vote is already there. What they submit will actually be counted. And finally, we need to tell you. And the result of all these input schemes, which I tend to call it, is some way of submitting a ciphertext that has to have some additional technical properties for proofs to work. But basically what we need is an algamal ciphertext or a malleable homomorphic ciphertext. So the one that is used in my code is the algamal cryptosystem, which has this nice homomorphic property. And I think that most of you know this cryptosystem, where at least you can recognize that this is basically the Hellman key exchange where you use the joint key as is as a one time pad. So instead of doing key generation and using some symmetric stuff, you just use that group element as it is once. And the nice property is this homomorphic property. So you can multiply two ciphertext and you're given a ciphertext which contains the product of the messages and the sum of the randomness. And the way we use it here is that we set m2 to 1, which means that we have re-encrypted the ciphertext. So keep that in mind. Okay, so David Chum came up with a beautiful idea called Mixnet. So here the idea is that we use multi-party computation to distribute trust and provide privacy. So the server somehow generate a joint public key. They accept some ciphertext, a list of ciphertext. All they do is decrypt and output the plaintext, but in random order.  This is what this is used for. Unfortunately, his particular construction was not particularly efficient or practical because we also want correctness property to be able to prove in zero knowledge that the whole thing went correctly. Okay, so here is an ideal Mixnet. So you have these nodes that you can view as senders or submission schemes used by voters and the result, they take the public key and the result is the list of ciphertext, each voter encrypts its vote and you get the plaintext in random order. So this box here is what we wish to accomplish with a protocol and we don't want to trust any single party so we distribute trust by multiple parties using multi-party computation. So here is the protocol. This is basically this original idea by Park Ito and Kurosawa to do this in this way. So here the servers generate the public key, the voters encrypts their votes and here I have ignored a lot of technical details and then the Mixervers take turns. So first the Mixer will re-randomize and permute the ciphertext. Then the next Mixer will re-randomize and permute the ciphertext and re-randomize and permute the ciphertext and so on. I think all of you know how Tor works where you have multiple layers of encryption and you strip off one layer after another, each server does this and randomizes. I this case instead of flipping off a layer we have a special envelope that we can kind of shake such that it looks entirely different after shaking. This is what re-randomization does. So this is what avoids the growth in length. But the main thing here is not really the length problem. The main problem that we wish to solve is to prove that this was done correctly. This is called the proof of a shuffle. And it turns out to be quite difficult to do this if you use layers of CCA to secure encryption. But for this construction it's much easier because you can use algebraic properties. Okay, and finally you decrypt. Okay, so this is basically the protocol that is implemented. Then there are lots of lots of details and before I go in to give you a little bit more about what this whole project has been about I want to give you the history here. So this actually started in Syriak. I was sitting at the ETH office and I had to check some equations in the proof of a shuffle, not the one that is used but similar. And those are very nasty. So I figured that I write a hack to check it. That at least it was correct. Okay, so I did that. And then when I went home I liked the program so I started working on this a little bit as a hobby. And then I started talking to Alon and Amnon and they wanted to use this. I talked to Ben Adida, we had these various discussions and I was hacking along during the evenings and so on. 2010 I released the first open source version and then this was used in the Wombat system for elections in Israel. There was an election for the Meretz Party leader, for example. Similarly a company run by some Spanish guys. It's called Envotes Now, previously Agora. They have used this for a long time and tallied 2.5 million votes. So they run a company using this open source version. Then we found some flaws in a system provided by the company Seitel to Norway in 2011. So they wish to replace the shuffling part of their system. And they didn't have anything to do that in reasonable time. So they asked me and then I had to improve things and start a company to protect myself legally and so on. So I was kind of dragged into this project in a way. And then I figured I'll try to go all the way. I started talking to people, I think some people from this more or less theoretical community and it was a humbling exercise. So I had a lot of advice and I improved the code and I did a lot of testing and everything. And now recently in 2017 it was used in Estonia. So this is the history. And here is a list of implemented protocols. But it's very important, I think, to remember that this is just the top layer. So there are some people here in the audience that are authors here. But there are also this huge literature of things that led up to this. So although the implementation is mine and a lot of tricks are mine and a little bit, some protocols, most of it is not my theory. It's not my ideas, let's say. So this is provably secure, it's well understood, but this is not really enough. This is an exact proof that gives you secure concrete security parameters and proves composition of all of these protocols for exactly the protocol that is implemented. Not some slightly different theoretical protocol, but exactly what is implemented, the real world crypto protocol. So this is something that I'm working on. So what about usability? So the functionality is really a black box. You can ask it to generate keys. These servers running as a box, okay. It can shuffle ciphertext without decrypting, it can decrypt without shuffling, it can do both. And every time it will output the zero knowledge proof that it was done correctly. And you can have different number of ciphertext processes as one block. If you have longer plaintext, you have to embed it in multiple blocks. So it's very flexible, I would say. So to deal with this, I wrote a user manual. And so everything is really well documented, I would say. All the commands, everything, this is very polished stuff. And when I say this, you should be very sceptic. But when the people who actually run it in Estonia, this is the response. I asked for feedback, now you cannot really see here. But it said, very forgotten work like a Swiss watch. This was the complete feedback I got from Sven. I asked, can you give me feedback? What should I improve? And what is wrong? What were obstacles and so on? That was the complete feedback. So it's a usable software. And then when you say that something is, you know, you have this zero knowledge proof in the e-voting community, this is sometimes called universal verifiability because this means that anybody can verify that the whole election was tallied correctly, which is of course very important for integrity of the election. But it's one thing to have a theory paper saying this in abstract mathematical language where things are field elements and there may be some arrows and things like that. It's a completely different thing to actually specify how these things are stored on files in different directories and how they are labeled and named and how they are packaged, all of these little elements, okay? So this took a long time to develop this document. I started in 2011 and had a lot of help from Alon and Amnon and their students in the beginning when we run this project in Israel. And now this seems pretty mature.  Because for a hash function you can have a test vector, you can print it in a document in a standard and you can check it. But here there are so many parameters that you cannot really do that. So the reference implementation allows you to print test vectors for intermediate results in all of these calculations, okay? And to give you an idea of why that is very nice, I should also say that we have tried this with multiple undergrads. So that's the level of knowledge you need to implement a verifier. Undergraduate in math or computer science or something. Okay, so this is proof of a shuffle. So that's just one of them. And then there is a similar blob for decryption. And all of those are stored in files. So you see the problem here and then you have session identifiers and various padding and stuff like that. You have to say how you derive every little element and how you store it. So it's very hard to get all of those things right in the first attempt. It really takes time for this to mature and you need other people to try it and give feedback and so on. Also, when you apply something like this, you need to have an idea of how long it takes. Is this feasible? Do we need new hardware? Can I run it on the hardware I have and so on? I see it's purpose, there's a benchmarking suite. Okay. And this has been optimized in various ways. Some are perhaps rather obvious, but the first one is what seems to be a novel protocol. I haven't seen it somewhere else, but it's kind of nice that the number of servers doesn't influence the time it takes to decrypt or to verify. So it's very efficient in this way. And there is also a use of rewriting and algebraic tricks to allow more parallelism between the servers because sometimes they wait for each other. The naive description is that they wait for each other. But you want as much as possible of the computations to take breaks simultaneously. Yeah, okay. So what about the source code? Yeah, so this is something that I am very, very concerned about. The way the programming community works today is by patching, patching, patching. And the adversary in my case is other governments, foreign agencies. And we don't want to reduce dependencies as much as possible. We want to achieve something that is nearly stable static code that is not changed.  I'm totally aware that this is naive. We're standing on this tower where there are lots of code below us. But at least at this point we can do as well as we can, okay? So minimize this. That's my idea. So I really struggled to not use any scripting languages, not even bash, okay? It might sound crazy, but I think it's actually a sound approach because if you try to do it everywhere, then at least you know that in those few places where it might matter, you didn't do it. You didn't make that mistake. In the same way, comment everything. I really mean everything. Because then you know that you have commented also those places where a comment is needed. Static analysis. I'm sure that you who work in the industry have access to more expensive tools than I have, but this is what I had and I used all the tools I could use. And the union of the complaints is bigger than any one of those. So that's what... And not accept anything, right?  And again, don't believe me, believe Joe Kinnery. He's an expert in this stuff. And he basically gave these things to me as homework. Like, have you done this, Douglas? Have you done this? So I learned a lot by doing this, you know, coming from the theory area. Input and output, the same approach, be very strict with parsing. Like ridiculously strict. Verify the grammar, verify every field with regex and then verify again, you know, with code for every field. Simple data structures. So don't need... So I don't need any parsing, special parsing software because it's very simple, byte-oriented format. Easy to parse. Nobody here could write a parser for that format in like 15 minutes. Okay? And allow different variations by pluggable tools. Okay, so we also need a client. And so my idea was I wanted to have a demo and I didn't really get that far, unfortunately. I wanted a demo voting system, something at least simple that could use the MixNet. But I got stuck because I have very hard time using what was available. So I wrote this from scratch, JavaScript code. And I think the main lesson here is that if you just take... It says hack here, that's a handbook of applied cryptography. If you take the textbook algorithms and you implement those cleanly, you do a little bit of like intermediate level arithmetic algorithms, and then you optimize, then that's good enough. And then you have something that is also like commented with references that you can use for your students when you're teaching this, which I think is also nice. Yeah, and also something that is... I should say here, it says... The next, the last point here, it says that does all the crypto and syronage proofs of knowledge abstractly. This is very important to deal with the absolute nightmare i terms of configuration and software. When you think about multiple ciphertext processed by the MixNet, multiple ciphertext encrypted, and now you want to do syronage proofs for different sizes, are you going to have four loops all over your code, or are you going to collect this? So the right approach here is to basically go back to basic math class and say, I can view this as a product group, I can view Algamal as over a product group, and then syntaktically everything looks beautiful. So if you look at the code, you won't even see that this actually handles much more general notions than simple. So acknowledgments, this is very hard for me to say because I've been doing this on and off, I've done other research for almost 10 years now, and I've talked to many people from different communities to learn and to try out my thinking to get feedback. So these are just the main people, I guess. So in Krypto, it's Johan, and Björn was my former student, he's now at Google in Sweden. Charom was a postdoc, Alon, and Benadida, Olivia Pereira, these are people that I've talked to over the years about crypto stuff. And then for coding, Tobjörn Grandon, who wrote GMP, he is very helpful to make things fast. And Joe Kinnery for code quality, also Kerstin Schürman, Markus Lagergren and Joel Borghen, they wrote the Jrocket JVM, and they studied with me, so it was like I've had really, let's say, from an industry perspective, I think, very cheap, extremely good advice. I could never afford to pay these people for the help that they gave me. And then you need somebody to read all that documentation and try things out. This is really hard to get people to do. That's my experience. So summary, basically what I have tried to do is, starting from my theory bubble, I've tried to look as far out as I possibly can and learn the techniques that I know that some of you are much better at than me and do that correctly by the book. This is the good stuff. And another good thing is that I'm a single author, so if I want to re-factor something, I just take the code and I smash it on the table and put together it in a different way. And I don't have to care about anybody else. This is an advantage. And the bad thing is that we are still, like I said, there are lots of other code floating around in an operating system. There is the JVM and the libraries that I still have to use and so on. So this, I have no control over this. There is this formal verification notion that you can either verify that code behaves correctly at various abstraction levels or you can derive code from type-based languages. So maybe that is the way to go in the future. I don't know. Side channels seem not so relevant in this context due to how these protocols are set up and the fact that they are typically run behind an airwall anyway, the servers. And of course the bad thing is that I'm the single author. So the code is kind of very, very consistent, but probably it has like a dog-less-ish style all over the place. And I'm sure that better programmers can have opinions about some things. I don't know. Okay, so what is the future? The future is if I have the energy or if I have a sponsor or some kind of funding which I lack, is to glue together a complete voting system because I really have all the code that is needed for the client, the authentication. It can be so different in different context that it's not really meaningful to try to implement something on your own. And we already have all these cryptocurrencies, cryptocurrencies. Other possible applications. There was a talk here about messaging. You could ship messages through this thing and it would be verifiable. You could have proofs that things were mixed properly. But mainly what I need, I need sponsors to make it open source because right now I can't really continue doing this on my free time because I don't have real funding to do this. I have small pockets I've received from. And this is a massive project. And it's all online so you can try it out and it's free research. So it's more liberal than open source licenses for researchers, but for real use I have a fair fee. You can look at it if you like. So we have time for one question perhaps. Why don't we go up to the top, yeah. Hi. Thanks for your talk. I was wondering, you mentioned that there's a... Where are you? I'm here. Okay, hey. You mentioned that there is a JavaScript crypto library as part of this project. Yes. It implements a big integer library. Yeah. Is there any sort of formal claim on the functional correctness? Is that formally verified in any way? Formal methods is not my strong side, but the test suite is rather, how should I say, aggressive. So it actually... It compiles, so you can compile it to use different word sizes to find corner cases in the word boundaries, for example. And it does all... I'd be happy to discuss it and explain more. But it's not formally verified, no. Thank you. One more question. Yeah. Hello. I have some questions for the underlying protocol. So what is the verifiable shuffle that you actually end up... Growth shuffle? No. It's the tedious week-stream shuffle. And what's like the asymptotic complexity of that? Or is it linear? Okay. So I hesitate to answer you like this. So on the one hand you can look at that complexity, the theoretical complexity, but on the other hand it's irrelevant for two reasons. Först, the running time of Jens fast, if that is what you think about, the Stephanie Bayer and Jens Groot paper. So the approval of that protocol has about the same theoretical complexity as ours, the prover. The verifier is faster. So this means that at best you can have a factor of two speed up. But do you not care about the communication complexity, is that not that relevant? Running low on time, it sounds like this is going to be a long conversation.  Let's thank him one more time.


Welcome back to another episode of Code Club, I'm your host Pat Schloss. We're going to be talking more about the Group Buy and Summarize functions in today's episode. Although we've used these functions previously, we haven't talked about a new argument in Summarize that was released this past summer in the Plier version 1.0, that is, the Dot Groups argument, that is, the topic of today's episode. After playing with it more on some other projects, I find it's actually pretty helpful for making the code in my analysis more explicit and readable. Perhaps you'll recall that when we group our data by some categorical variable, like the genome that the sequence came from using Group Buy, we could then use Summarize to calculate a summary statistic across the observations for that genome. In the last episode, we used this to calculate the number of copies of the 16S RNA gene in a genome and the number of unique 16S RNA gene sequences in each genome. We've been calling those Amplicon sequence variants. Before the Plier version 1.0 was released, the default behavior was to remove the genome-level grouping of the data, removing the last grouping variable. Now Summarize outputs a message telling us what Summarize is actually ungrouping. The change is that Summarize now has a Dot Group argument that allows you to tell Summarize what grouping variable it should be ungrouping by. One argument is Drop Last, which gets you the Pre-Version 1.0 behavior, and it's also the default behavior in 1.0. But it also allows you to use Drop, which removes all grouping for those cases where you've grouped by two or more variables. Another option is to use Keep, which tells Summarize to keep the grouping that was set with the Group Buy. And finally, a fourth option, Row-Wise, that creates a group for each row in the data frame. For today's episode, we'll take a deeper look at why we might use Drop Last, Drop, and Keep behaviors. Although the Summarize documentation indicates that Dot Group is an experimental argument, my sense is that it's probably a good thing for making it more clear what's going on in our code, because it forces us to be explicit about what we're doing. We'll see how this plays out in today's episode, where we'll figure out how many copies of the 16S RNA gene can be expected per genome at each taxonomic group in any taxonomic level. Even if you're only watching this video to learn more about R, and don't have a clue what a 16S RNA gene is, or what taxonomy is, I'm sure you'll get a lot out of today's episode. Please take the time to follow along on your own computer. If you haven't been following along, but would like to, welcome. Please be sure to check out the blog post that accompanies this video, where you'll find instructions on catching up, reference notes, and links to supplemental material. The link to the blog post for today's episode is below in the notes. So I've already filed an issue for today's episode, issue number 28. And what I'm going to do is we want to look at trends in the RRN copy number across taxonomic ranks. And so I'd like to have on the x-axis the taxonomic rank, and on the y-axis the average number of copies in that taxon. And so, because many of our species have a lot of genomes, but many of our other species only have maybe one genome sequence, what I'd like to get first before we look at higher taxonomic ranks is an average for the species. So you'll recall that E. coli has something like, I think, 800 genomes in the database. And so that's going to really kind of swamp out the signal from other groups. So if we get one number from each species, we can then look at the averages for those higher taxonomic levels. And that, again, will allow us to control for uneven sampling of those different species. So again, this is issue 28, and I'll come over to my terminal, and you'll see that I already created a template for today's episode to make an R Markdown document in our exploratory folder. Let me go ahead and create an issue, get branch, issue 28, get checkout, issue 28, and we're good. I'll go ahead and open up R Studio by opening my R Proj file. And if I look in my exploratory directory, you'll see that for today's episode, I've got this template already created, quantifying the number of RN operons across taxonomic ranks. I think I should fix my date. What I say was 10.5. And again, I have my editor set up to output the chunks to the console. You might prefer to have it embedded directly in the document. For whatever reason, my quirk of how I've worked with this in the past, I really like it going to the console. And then I also have the output set up so that when we render the document, finally it'll be, it'll look decent or good on GitHub. So what I've also copied in is the code chunk from the previous episode. What I like to do in these exploratory analyses is kind of like build up a bunch of different analyses, and then figure out what works, what doesn't, what's going to go in the story, what's going to get cut out. And as we do this, we'll find that there's chunks of code that get kind of copied between different, our markdown documents, or different iterations of our analysis. And in the end, when we try to pull everything together, then I clean up that code and make it dry so that we don't have repeated code across our project. We're called dry stands for don't repeat yourself. We only want code represented once and then we want to call it in different places, maybe using functions or by creating files that we can then use as input. OK, so again, this was what we had before. I'm going to start by commenting out this part from end of line 21 here before the pipe line 22, 23, 24, 25 and 26. What these lines did was they took our wide data frame and made them tidy. But before I do that, I want to get an average for each species. So I'm going to get the average for each species and then I'm going to make it tidy. OK, so we'll go ahead and run these different steps. Ah, and I have to run my libraries and then it will know what the pipe is. And then ASV metadata ASV. And if I come down and we get our data frame that we know and love, where we've got a genome ID, kingdom file and class order family, genus species, strain, the region, the ASV name and the count. And the count is the number of time each ASV shows up in each genome. So we'll have to do two things first. So we will need to get the total number of copies per genome. We'll need to focus in on the V1, 9 region, those two things. And then we'll also come back and get the average for the species. All right, so metadata ASV will pipe that to filter to remove the V1, 9 region or to get the V1, 9 region rather. So this will get rid of all the other regions. And if you're paranoid that that didn't work, you could always add a count region. And we see that everything is from V1, 9. So we're good. Okay, next we need to get the count on the total number of copies per genome. So what we could do, or might be tempted to do, would be kind of group by, and this was genome ID. And then summarize and do say NRNs equals sum of count. If we do that though, one thing that you'll notice is that the summarize function creates a new data frame, in this case with two columns. One column will be the group by variable, and the second column will be the summarize column. But if we do this, then we lose all of our taxonomic information. We could always do an inner join back to metadata ASV, but that just seems kind of silly. What we'll do instead is we will add our different taxonomic levels. So we'll do kingdom, class, kingdom phylum, class, order, family, genus, species. I'm going to leave out the strain because I really just want one representative per species, and this should get us where we want to be. And now you'll see the difference in output is that again, we have all of our grouping columns as well as our summary column. And what you'll notice, of course, is that it also got rid of a few other things, right? It got rid of the strain, and it also got rid of the ASV column. So good. So this looks great, and what I now want to do is I want to group by the species, right? I want to get a summary for each species. And so it's already grouped by that, I see. And so I can go ahead and do summarize. And I'll do mean rn equals mean nrn. Running that, I now see that I've got species and the mean number of rn's for each of those. Maybe what I'll do is I will do a filter on species, Escherichia coli, to make sure everything looks right. Then we get 7.01 copies, so there's perhaps one or two genomes in the database where there's maybe eight copies, who knows. But that looks good, right? That's the kind of output we want. We want one line, one row per species. And at this point, we see that the output is grouped by Kingdom Filem class order family genus, which is great. And what I would normally do would be to say, let's do ungroup. And so if we run the ungroup function, then we no longer have any grouping variables for our data. But now what I'd like to do is make the data tidy. And so I'll do pivot longer. And I will use minus mean rn's. So I'm going to pivot everything except for that mean rn's column. Names of the columns will go to rank. And the values in those columns will go to taxon. Let's see, argument four matches values two, sorry. And so now we get three columns, mean rn's, rank, and taxon. But I also have some na values in there. So as we saw up above, I should probably just copy this down and paste this down, right? So I'm going to also bring down that mutate with the rank. And I'll get rid of all this other stuff. Paste that in. And we will use our drop na, as well as our mutate, all right? And so if we run that, we got rid of our na rows. And we now have mean rn's, rank, and taxon, okay? Great, so this is already pretty quickly turning into quite the pipeline. I now want to group by my rank, right? No, I want to group by my taxon. And so, of course, we will notice that the output of this has no grouping variable, right? So we'll go ahead and do group by taxon, rank and taxon, actually. Because if I just did taxon, we would lose that rank information, right? So we'll do rank, taxon, and we'll pipe that to a summarize. And then we will do mean rn's equals to the mean of the mean rn's column, right? And so now what we see is that we've got a column for the rank, the taxon, and the mean rn's, and that we've got seven different ranks here, right? So we're in good shape. And we see that the average for bacteria is like 4.33, for archaea is 1.61, before I did this, without correcting for numerous copies of different species. And the bacterial average was actually like 5.06 or so, I want to say. So that's no doubt due to the overabundance of things like salmonella and asherichia coli that tend to have like seven or so copies. So again, this is a number that we can think of as being corrected for overrepresentation of each species. So this is good, right? This is the data that we can then plot, and I will then call this data frame, taxon rn's, maybe it'll do rank, taxon rn's. But a few things happen in here that I want to point out. That as we ran this pipeline, this is how I would create the pipeline up to about, say, five months ago, right? But in the last few months over the summer, the developers of dplyr put out a new release of dplyr, and dplyr is where we get these group by and summarize functions. That they, you'll notice that when we run a couple lines like this, where we're getting the total number of rn's per genome, the output at the top of this data frame tells us, summarize regrouping output by kingdom file and class order family genus species. Right, so it's telling us how it is ungrouping or regrouping the data. I think of the group by function as basically making my data like an onion, right, with different layers. So in this case, we've got how many, four, five, six, seven, eight different layers, right, of our onion. And then we ran that summarize and appealed off the genome ID layer, right? So that layer now is gone. And our output, as we can see, is kingdom through species, right? Genome ID is gone. We now, we then got the mean. And if we, if we add that line to the pipeline, we now see that it tells us that it output, it regrouped the output without the species, right? So it removed that species layer of the onion. And we, of course, then see that the output is here, right? So this is the default behavior. This is what group by did prior to the release of 1.0. But it tells us that we can use the group's argument of group by, of summarize rather, right? And so we can be explicit about what we want to group our data by. And so for example, this default behavior we get by setting, using the value drop underscore last to the group's argument. And so we run that, we no longer get that message because we told it explicitly what we wanted to drop the last to drop the genome ID, right? So we could do the same thing here as well on this next argument, right? And so, of course, we run both of those and we no longer get the output message about how is regrouping because we told it how we wanted it to regroup. But, so this actually makes me notice that I forgot that I had this ungroup function. And so the ungroup function following summarize is actually the same as using drop, right? So if I use drop, then I get rid of all grouping information. And so you'll see in the output of the table, there's no grouping data there. And I really like to be able to ungroup my data because when I get into things like pivot longer or some of these other functions, I tend to see just weird things happen. And really, this gets to the important point for today's episode. It's that I like to be really explicit in how I am grouping my data. And I like to be explicit in what I'm doing in my code just in general, right? So if you knew R and you knew what was going on with the summarize function, you would know that this first summarize is doing it on this full list of things I was grouping by. You would then know that the second summarize did everything from kingdom through species, right? But if you weren't so familiar with R, or if there's a lot of other stuff going on in here, perhaps between lines 38, 39, and 40, it might get a little bit confusing. And so at first, when I started using this new version of DePlyer, it kind of annoyed me that it was giving this output because it was just, I don't know, I don't like the warning messages or the notifications. I like for the code to just be clean in its output, right? So as I've played with this and other projects, I've come to realize that I kind of actually like these different options. And if you look at the help page for summarize, you will see that there is actually, there is the groups argument. And you'll see that life cycle experimental, which tells you that they're not totally sure this is what they want to do. Well, I think I kind of feel like maybe drop might be appropriate to be the default. And I can imagine there are reasons not to make that the default, but I'm kind of feeling like that's a default that I might like to use. Because if I put drop there, that will force me to be explicit about what I'm grouping by, right? So I would be basically doing this, right? And that way, if you come to my code and you read my code, or if I come back to my code in a month or three months and read my code, it'll be very clear what's going on, right? And perhaps the example that I have here was already clear. But imagine if you have a more complicated pipeline where you've got other steps intermixed and those three lines were split up. It might not be so easy to track what's going on, right? So again, we can run all this and we get the same result. We see that we have one complaint down here. That's because we have the summarize on this step. But what I'm going to do is I'm going to do groups equals drop, and it'll get rid of that, okay? And so then, again, if I look at my data frame, which I called rank, tax on RRNs, I see the values that I had had before, which is great. There's a couple other arguments to be aware of with summarize, or not arguments, but values to groups. So we talked about drop last, which again is like peeling off that outer layer of the onion, drop, which just takes off all the layers. And then you can also do keep last, or just keep rather. And if I look at running these four lines of code, what I will see is that I still have everything grouped, including all those layers grouped, including the genome ID. So basically each row of this data frame becomes its own group, okay? So that's another option. I'm not totally sure where I would use that, but it's good to know it's there. And then similar to this is row-wise, which I don't know that I've ever used before. And so now I'm going to start looking for places to use row-wise. But group-by, we can think of as grouping on columns and summarize allows us to do operations on those grouped columns. Row-wise basically treats each row as its own group. And that then allows you to do operations on individual rows. Again, I've never used this before that I'm familiar with. There is a row-wise function, just like there's a group-by function. I don't know that I've ever used it. So I'm probably now going to start looking for opportunities to use it and learning more about it. But again, those are the four options that we have. We have drop-last, which is the default. We have drop, which gets rid of all the layers. We have keep, which keeps the full group-by listing. And we have row-wise, which then treats each row in the output as its own grouping variable. And again, I'm going to use drop because I like the fact that it's going to force me to state what I'm grouping by. And again, I typically in a case like this last one, or even at this last one, right? This last where I'm going to end up having things grouped by rank in the end, I would always follow this with an ungroup function anyway. So putting in the dot groups equals drop to me is kind of a nice convenience. And again, makes my code a bit more clear. So if you think this is all much to do about nothing, know that there is another option to all this, right? You could do de-plyer, summarize, info equals false. I think that's the argument. Yeah, summarize and form, sorry. And you can set that to false. And if I remove this dot group, groups argument, and run all that, maybe summarize does need an S. So usually they're very good about using both spellings. De-plyer, summarize, inform. Yeah, for some reason, it's still telling me what it's doing, even though I have de-plyer, summarize, and form set the false. So I don't know what's going on there. But again, I generally don't like to turn off warning messages because I think the warnings should mean something, right? And if there is a problem, then I wanna know about it. So I'll go ahead and add back in my groups equals drop. We run that, we're good. And again, if we look at rank, taxon, RNs, we get our three column data frame, the rank, the taxon, and the mean RNs. And what I'm now gonna do is I'm gonna go ahead and do rank taxon RNs. I'm gonna pipe that to ggplot, AESX equals rank, Y equals taxon, not Y equals taxon, but mean RNs. You'll see that rank is a factor, and we put that code in up here, right? To put things in the same order that we had desired. I have strain in there still, but it doesn't care. So then we can do geom jitter. And I'm gonna let's give this a run, see how it looks. And we see, yeah, we see a mess. But this is what we're going for. I'm gonna go ahead and put in width equals 0.3. Maybe make that cloud a little bit tighter so it doesn't bleed over. Maybe I'll also do alpha equals 0.3. What alpha.3 does is it makes the points transparent so that they then overlap on top of each other. They get darker. It kind of blends in with the background a little bit. So I'm a fan of theme classic and that cleans it up a little bit because I can't help myself. I'm gonna play with the labels a little bit. So labs, X, I'm gonna make null because in cases like this where I've got categorical variables on the X axis, I think it's obvious what my variable on the X axis is. So null then will remove that line that contains rank and why I'll make mean number of RN copies per genome. And then title, I will say is what? There is gotta be in quotes. There is wide variation at the species level in the number of RN copies. And maybe I'll put a line break in halfway here. And I'll go ahead and put in a subtitle which will be each point represents a single taxon within that rank. I'll say numbers based on an average species copy number. Kind of guess about where to put line breaks in here. So I'll save that. I think that's good. My deep thoughts will be what? So bacteria have more copies than archaea even after correcting for number of genomes per species. There was wide variation in the number of RN operons per taxonomic group. Okay, so that's good. We'll save that. I'll go ahead and knit it. Make sure everything works well here. This outputs what it will roughly look like on GitHub. I'm gonna clean up this tab a little bit. See what's going on with this and re-knit and see if that improves what's going on there. Not really. All right, so that's good enough. We accomplished our goal for today's episode. Let me go ahead and commit these changes. So I'll git add exploratory, 2020, 10, 05, put the star to get all those files, git commit, construct strip chart, of RN copies per taxonomic group and level. Closes number 28, close quote. We're good. Git checkout master, git merge, issue 28. Our studio is freaking out in the background because I closed stuff. Should I close this? Yes. And we're good to push. And that's all going. And this is closed. And again, we can go to our exploratory directory now and see that we have a good number of reports in here. Don't know what's going on with the tabs. That's kind of annoying. But we see that our plot looks pretty good. Maybe we didn't need all those line breaks, kind of played with our title here. But again, we can see the average number of RN copies per taxonomic group when we control for even sampling of the species. So wonderful. What we'll do in the next episode is perhaps draw a line across this to show the average for that rank. And so one of the really powerful things about GG plot is the ability to think in terms of layers and put a plot on top of a plot on top of a plot. And so you can make some really nice effects that way. So there's a lot we're gonna play with on the analysis we're working with using what I'll call exact sequence, amplicant exact sequence variants where we're taking the data exactly as it is represented in the genome without accounting for any kind of error in the sequences. Eventually we will come back and we'll entertain some noise and then we'll talk about say, amplicant sequence variants where maybe that we allow for a couple base variation to fold sequences together into the same amplicant sequence. So I hope you found this an interesting discussion of the dot groups argument. It's always a little bit jarring when there's a new argument to our functions that we know and love. I couldn't find a lot of description online for why this decision was made, but hey, if you have any thoughts about the dot groups argument, feel free to leave them down below in the comments. I'd love to hear what you're gonna do with this dot groups argument and how it's gonna change your workflow and the type of work that you're doing. So, and also if you can get that de-plier summarize and form thing to work, let me know, see what I did wrong, I don't know. So anyway, keep practicing and we'll see you next time for another episode of Code Club.


Hey guys, it's Monica. I'm going to share with you a little bit of my birthday highlights, some new tretinoin that I'm using and some gifts that I got. As a lot of you know, my birthday was this past Valentine's Day, February 14th, and I turned 65. And I kept thinking about, you know, what have I learned in 65 years? And I kept thinking, I've learned an awful lot in 65 years, and I've learned too much to actually try to even talk to anyone about it in a video because it'd probably be very boring. But I think ultimately, at the end of the day, at the end of the game, if you want to call it that, 65 years of living, I have really learned to treasure every moment I have with my loved ones. It's true because you don't know. You don't know when that last moment will be. You don't know if that last moment's going to be when your loved one drives off to go to work. You don't know what's going to happen overnight. And the last thing I want to do is have regrets and say, I should have wished I could have, you know, all of that. And yes, everyone is going to have some regrets in their lives for sure. And everyone's going to have moments when they say, oh, wow, I should have done it this way.  That's what really matters at the end of the day. YouTube is a wonderful outlet for me. I enjoy every moment of my YouTube time. I enjoy sharing things that I've got, things that I'm thinking about, things that I'm doing. I enjoy all of that. And if I didn't, I wouldn't be here with you in a video week after week after week. And I know sometimes my videos can be really long and rambling, but 65 years, I can remember when I was 15 and I thought 30 was old. I thought, you know, 30 was, it was ancient. And now here I am 65. And I'm like, oh, wow. How did that happen? Where have the years gone? How did that happen? Where have the years gone? It's like, it's, it's something we all, we all ask. And I can truthfully say that while my 65th birthday might have been extremely low-key, it was really a wonderful, wonderful birthday. Valentine's Day is always tough. It's tough for us to get reservations because every place is booked. It's, it's just really sometimes a tough holiday to go out and celebrate a birthday. It's the day of lovers. So this year on my birthday, Jay and I went out with my mom and dad, which we do almost every, every year. We go out with my mom and dad. Sometimes my children can join us. Sometimes my sister joins us. Sometimes our husband, if he's home. But we kind of go out for a family dinner. And we did it this year. We stayed somewhere local and we just enjoyed each other's company. I think it was like amazing to me that I was home maybe a half hour after my birthday. I had all these plans for the weekend.  We're going to do that. And about 30 minutes after arriving home, after dropping mom and dad off, I got violently sick. To this day, I don't know if I got a case of food poisoning. I don't know whether it was. I got a bug, but I got violently, violently sick. I was sick all the night of my birthday. I was sick all of that next Saturday. I got a little bit better on Sunday, but then I had a slight relapse. So it was like, it was like just a really not a good way to celebrate birthday. But I still enjoyed myself. I still had some great, great, I got some wonderful gifts. I had been wanting an adapter for my camera. I have a Canon M50. And this is my vlogging camera. And I'm also recording with a Canon, a bigger Canon. But this is my vlogging camera. And I had wanted an adapter to put a wider angle lens on. And so I got the adapter. It was my birthday present. I had ordered it and it arrived before my birthday.  That made me really happy. So when I'm doing, if I'm going to ever get back to doing the vlogging and I'm like this, I have the wide angle, I don't have to worry about really stretching my arm out. I can actually hold it pretty close. And the wide angle adapter will be on here. And it'll be, it'll be really cool. So I've been wanting that adapter. I just haven't wanted to spend the money on it. But I did, I got that from my birthday. So that was like a really wonderful, wonderful birthday present. One of the other birthday presents. And I had talked about this this actually arrived a little bit earlier. And I actually am wearing it on my eyes today. I've wanted on, I've worn this palette on my eyes since I got this more, more often than not. I received a PR package from Pixie. And they had two palettes, one that I used and one that I haven't touched. So they had two palettes in there. And I was using that one. I used that a couple of days, but then I went right back to this one. And this is the Hootie Beauty Mob Obsessions.  And you like softer colors that you can build up like me. I was really surprised at how much I like this. Now prior to this palette, it's the first time I've ever had a Hootie palette. And this one, I'm gonna try to cover up my hands a little bit. Prior to having this, my, my all time favorite palette has been my Visiard Amethyst palette. It still is. I love the colors in there. I love everything about that palette. But I have been wanting something that I could create more of a smoky or I look. And the brown shade in the Amethyst palette kind of always made me feel more like a baby poop color. If that's the wrong way to say it, probably. I didn't really like that brown shade. These, these, um, this one has a nice, nice brown shade. And I really like it a lot. So I have been wearing this palette a very, very, um, almost every day, often. I really, really like it. And this was the birthday gift from Sheila. And thank you so much, Sheila. It was just an awesome gift to get. I also got a gift from a subscriber. I'm not going to name, I always feel funny because I don't know if people really want to be named or not. But I got a gift from a subscriber and it came in right at my birthday. I considered it a birthday gift. She really didn't. I think she just considered it a gift. But it was a Tretinoian cream and it was the 1%. Which comes in super handy because I'm on my last tube of Tretinoian. And this, the expiration date's like, I don't know, a year and a half out. So I have plenty of time for this. But this is an open sealed Tretinoian cream. 1%. It's the cream as opposed to the gel that I have. So I received that. And that was like an awesome, awesome gift. Thank you so much. I truly, truly appreciate it. The other gift that I got was actually from my sister. And she had given me an Elta gift card. And she had also given me a Hydro Canone 4% Tretinoian 0.5, 0.05%. This stuff, I'm going to insert a screenshot here. This stuff, I don't really know what this goes for in the States. To be honest with you, but this stuff is really awesome. So what I've been doing is I have a dark sort of an age spot on the side of my face and over here on the corner of my eye. And also on my hands. You know, I don't, I don't know if the camera's going to pick up, but I have like the spot right here that in a couple of small ones. So my plan is to use this, this Tretinoian with the Hydro Canone in it on these spots and on my hands and see how, how it does. The, I've used it probably, gosh, I don't know how many times already, but I tell you the peel action is unbelievable. So I'm really excited to give this a try on those brown spots. So it says, it's, you know, if you read the, you know, the directions and all that, it's, it is something that will work on your, your brown spots and your melasma and stuff of that nature. So I'm really super excited about trying that. And then I also got in, this is from Maline Fab and Glam. I don't know if you know Maline, that sexy redhead. I'll probably get demonetized. But she sent me a whole box of goodies. I was like, unbelievable. I couldn't believe it. I loved the box that came and take a look at the little lips. Anyways, she sent me a palette. No, she sent me samples and lipsticks and, you know, other things that are not just samples, but she sent me a VCR palette. And this one is the nuts. So for, as you said to me, I could return it and get something else because she thought that maybe the colors might be too dark. And when I first looked at it, I said, they are a little dark, but I absolutely love all of the shades in here. Now I've played with it a little bit and it builds really nice. It is really blendable. The darks are dark, so it does take some, oh, you're not going to be able to see it. It does take some getting used to and some blending for me, but I took a little brush and I just popped the outer corner of my eyes and I was able to darken it beautifully. So I was absolutely thrilled when I opened that up and I said, wow, you know, I couldn't believe she did that. She also sent me an Urban Decay 24-7 Glyne On Eye Pencil in Zero and a number of other things, but the things that I really liked was the Marc Jacobs.  And when I say I really liked them, I really liked everything, but I've never tried these before. So this was really cool.  There's nothing as relaxing as just putting on a sheet mask for me and purity, made simple, philosophy, little sample of that. Jewel up, love your bare face. And a sign from the Elta Pencil and she gave me Sydney Grace. This is just a little tiny pop of color. So this is pink of perfection.  Pink of perfection and this is eye cream, shadow cream. So this is really cool. I really, really liked that a lot. I was like, whoa. And she also put in here a Hangover RX primer. So like, whoa. So I'm going to try this primer out. You know, I've been on a primer hunt. I've used many different primers. I'm going to have to do a video and update everyone on the primers because I've used so many of them. It's amazing. So that was like a super awesome, awesome box, a gift. I got all the palettes, the Huda palette, the Visiart, Visiart. How do you say that? Visiart, Visiart. Visiart, Visiart. I absolutely love it. I think these are just really awesome. So my new favorite palettes, yes. And now I have my eye on one of the other new Visiart palettes that came out. Was it liaison? Liaison, that looked pretty good. But I think the colors look very similar to the Amethyst, except there's a few more choices. And I'm thinking that would be really nice. I think I would like that. I think I would like that. And I'm wearing my Henry, Margu Hayden wig. This is the lighter color one I have been wearing in some of the other videos. I've been wearing the Raquel Welch and I really like it, the Fleur de Luret. And I've also been wearing, I have two Hayden wigs. I have, this is the lighter shade and then I have the darker one. And that's the one I'm wearing tonight. I really love the wigs, you know? I'm like really happy with the fact that if a wig is, like when you get it out of the box, if it's too shiny, you can do something to curb the shine if you want. I love the fact that they're so easy to take care of. They're so versatile and that your hair, you can have a good hair day even when you're having a crappy hair day. And I have lots of crappy hair days in real life. So not only did Jay get me the adapter for the camera, he also bought me some luggage. So I have two new pieces of luggage. I'm going to insert a photo here so you can see them, because we are going on a trip. We have a trip coming up to Las Vegas and the Grand Canyon for two days before we go to Vegas. And that's coming up really quick. So I'm super excited about that. And my mom and dad, they got me some money.  We're taking a really big trip next summer. And I, you know, I don't like to rush time. I really don't. But I can't wait for that trip. I can't wait for that trip. So, but for the rest of this year, this luggage is going to come in handy. We have Las Vegas and then we have a trip this fall. And I may have a conference that I'm going to have to do in Tennessee. I'm not sure yet, but that may be on the books also for the summertime, but we'll see. So I just wanted to share all that with you. Thankfully, I am feeling so much better after being sick that, um, yeah, that was really awful. Whatever I had, flu, bug, whatever it was, it was really awful.  So anyways, guys, I hope you all have a great day and thank you so much for spending some of your valuable time with me. Now go hug a loved one or go tell someone you love them or reach out to somebody.  I don't know. There are people all around us that are hurting and I think sometimes we're oblivious to seeing the pain in people. I'm sad. Now I'm being melancholy, melancholy, melancholy, melancholy. Oh well, anyways, never mind. Thank you so much, guys. Hope you all have a great day. Bye-bye.


Huawei P30 and P30 Pro, everything you need to know. Huawei's impressive P series is better than ever, and this is an exciting article from the folks over at Android Central, but I don't think they give you everything you need to know. And let me talk about that a little bit more in detail and from a security aspect, and that's what I wanted to cover here. So, CIA issues warning over Huawei. Politics, politics, politics. Did they give us any detail? Probably not. The US shared claims of Britain and other parameters and the Five Eyes Intelligence, et cetera, with UK entering final stage of review next generation and I'm not gonna go into depth on that because I care about the facts. I care about what we can see. What is public information that's available or that we can use tools that we have at our hands to dig into these pieces of information. And that's exactly what PE3ZX Huawei Blocklist. This is on GitHub, I'll leave links to all this. Doing some security research this person was. Very fascinating here because they took a brand new purchase from Thailand, Huawei P30 Pro phone and noted all the Chinese including .gov connections that are made from this phone. So domain names on master.txt are captured from DNS requests from Huawei P30 Pro purchase in Thailand, all domains located in China or has IP address with China's ASNs. ASN is the system by which we understand who owns what IP address. The block list may include domains from, may include test domains from Baidu and QQ.com by relying on Technique and update PY to verify location names. See domains such as Tencent Cloud and HWCDN are not flagged. I will find a way to fix this issue later now. This is one of the things important about when you're doing any type of security is being absolutely clear to the methodologies that you're used. So there's even a follow up which is provide environment factors related to this research, which they did. So you can read through and cover exactly how they came over this list. And this is obviously where things get concerning is what is these phones doing? Much of the data that gets sent is encrypted over TLS these days. Therefore, we may only see the connections. We're seeing that metadata but not the actual meat that's going on there. But it's obviously the first level of concern is why does my phone talk to China? Should my phone talk to China? It has a lot of access. What attack vector is this? What risks are there to have nation states that have full access to something on the phone? I think that's quite concerning. And obviously you can have your own concerns with Google and places like that. But I figured why not do this test myself on my Pixel 2 phone and figure out where it's contacting? So I did. And I wanna cover the methodologies that I used as well to get this information and figure out where my phone is going. So first is I'm using PF Sense. Don't make any videos on this. And I did went over here to Diagnostics and I chose LAN, because that's the LAN where my phone lives. It's my P address of my phone is <IP_ADDRESS>. It's a Pixel 2 XL. We set the count to zero and then we started a package to capture. But just before I started the packet capture, I turned my phone off. That way there's no connections and then started the packet capture, powered my phone on and downloaded the capture. Next step after that is Wireshark. So here we can see the Wireshark capture of the phone. So this is only, you know, not over a long time, but over what we got here, about 102 seconds of capture. So the Pixel 3 XL if you didn't know, or Pixel 2 XL if you didn't know, boots up really fast. So, but the packet capture starts at the time of capture when it started seeing traffic. So it sees graphically quickly. Long story short, this is a packet capture, but not for over like a day's time to see if there's anything that happens. Like it pings at 11 p.m., some weird server. I'll do something more in depth later, but nothing, I've so far researched my phone and found no weird connection, especially not to China. And one of the other things I want to point out right away when you start looking at the protocols, TLS 1.2, TLS 1.2, G-quick, G-quick, these are all encrypted transmission. So there are a few pieces that may not be, but for the most part, everything in here is encrypted, but we can collect that metadata. The metadata we're collecting here is the actual queries of where it went. So we're gonna go over here, statistics, endpoints, because that's the ultimate one. Where are all the endpoints that this contacted to? So here's just a nice summary of the packets and the IPs that went through and let's dig into them. So nice thing is, you just go here to copy, and we're gonna copy it as a CSV. Move my head out of the way that the CSV's right here. So we copy it as a CSV, and then we just open up Leave Your Office, drag it on over here, and we'll just go edit, paste. And these are comma separated, so right here it changes it to easy import. So there we go. Now we have a summary of IP addresses. Let's run down here though real quick. Purge out IP addresses that aren't public because they're not relevant. So these are the local connections that it's seen, local lookups. I did find it interesting, my phone on boot checks two IP addresses that are relative to my home in the 192.1681 range, because we started at the three range at the office here. So interesting that I picked those up, but they're not ones we wanna look up for ASN. And we don't really need, but I'll leave them in here anyways. We'll give an error on those couple broadcast addresses. Then we're gonna go over here to this handy website, bulk IP to ASN lookup. I'll leave a link to this. This was a handy utility. I know I could have all done this from the command line because at CSV I could have parsed it and run a script that does a reverse lookup and pulled the same information. You're not wrong. I just wanted to make it easy and accessible because a lot of people have access to Leave Your Office and this website copy pasta to get ASNs makes it real easy. So now we can dig into where do those IP addresses that Tom has go. All right, here's all the IP addresses. Google, not shocking, 888, that's Google's DNS. Amazon, yes, I have the Amazon app loaded on there. So maybe in fact I got a couple of them loaded. Amazon Prime, the Amazon Fire Stick Controller. So I'm gonna expect these Amazon ones to be on there. But the first ones that caught my eye that weren't Amazon were Linode.  I have the Facebook Messenger loaded but not the Facebook app. So it does reach out to Facebook. Google, Google. The one Comcast one I'm going to let you know to that's a Plex server, so I know what that is. But these Linode ones, what weird thing would it be contacting from Linode on there? So once again, we can just go ahead and copy, open up Showdan, just be an investigative here. Look up this, you found this and we will zoom in if I can and read what the, because it does have an SSL cert, which those can be really handy to figure out where things are. And it turns out it's, the reverse DNS lookup is for pocketcast.com, which is an app that's on my phone. So okay, that seems legit. And nothing else of really interest here. Like I said, it's a short list here of IP addresses. And of course, like I said, it doesn't look up those broadcast addresses that are at the bottom. So those are just sitting here with no result because of the broadcast. But this is an interesting way you can dig into and look into the things being looked up and what might be where your phone's going. It's kind of a fun little dive into the start of research and how someone came up with this block list. Here, obviously I'll note the exception that we don't know some of the 4G information. So if the phone were to send out a separate packet over 4G, we would have to take out the SIM card and force it then, but maybe it has, maybe we go nefarious further and say it detects when the 4G's often doesn't make these queries cause it's trying to hide things from you. Yeah, when you don't have clear access to full source code, this is a problem. But it is concerning that Hawaii's, you know, obviously they're not being secretive about this because they would mask it in some other way or set up a line node proxy that sends it back over to China, but they're doing nothing of the such. It openly reaches out to these IP addresses. So I found this interesting, but I wanted to make sure you understood some of the methodologies that could be used to detect this in case you wanna know where does your phone go or any device really, IoT, whatever device you wanna test. This is an easy way to test it inside a PF sense. You just go to diagnostics packet capture. I'll leave a link to this cause this is just handy between the tools used, you know, leave office, packet capture, wire shark, all these are for utilities you can use. And of course, if you're already running PF sense or whatever you wanna do for the packet capture, it's a great way to do it and makes life a lot easier. That's about it. And I will know I did full in case anyone didn't notice I had full packet capture when I did this cause I wanted everything, even though we're not really tracing out too much in wire shark. But always, you know, never be afraid to take a look at your devices and ask that question where they're going and stop and do that, you know, even once more. Maybe you get an app on there that is doing this, but you know, it's through these security researchers that we start to discover as we start to dig into what these things are doing. So it's always interesting. It may not be nefarious at all. Maybe just their policy to have statistics because there is such a mix of government slash private business with any of these companies in China. They don't have that separation of private company versus government. They're kind of one and the same. And maybe it's your policy that these have statistics so they understand sales better. Maybe it's not nefarious at all as we don't know the traffic is going across but we do know that it's going there. So that raises at least some levels of concern.  Maybe there is something to what the CIA is saying and the US government is saying about Huawei, but hey, keep your tinfoil head tight and feel free to continue this on in our discussion forums. Thanks. Thanks for watching. If you liked this video, give it a thumbs up. If you want to subscribe to this channel to see more content, hit that subscribe button and the bell icon and maybe YouTube will send you a notice when we post. If you want to hire us for a project that you've seen or discussed in this video, head over to laurancesystems.com where we offer both business IT services and consulting services and are excited to help you with whatever project you want to throw at us. Also, if you want to carry on the discussion further, head over to forums.laurancesystems.com where we can keep the conversation going. And if you want to help the channel out in other ways, we offer affiliate links below which offer discounts for you and a small cut for us that does help fund this channel. And once again, thanks again for watching this video and see you next time.