Hey folks, welcome back for another episode of Code Club. As you probably know by now, I am in the midst of a series of episodes where I am trying to build a data visualization of the world, showing the level of droughtiness for say the previous month relative to that same window of the year for the past 100 years or so. To do this, we are downloading a large dataset from the NOAA website. It comes to us as a tarball and it's 3.3 gigabytes in size, compressed, decompressed, it's 29 gigabytes. The engineering challenge that I have is that ultimately I want to be able to put my entire project up onto GitHub to use GitHub actions and there they limit me to 14 gigabytes of drive space, 14 gigabytes of RAM. I get three processors and I think I get some amount of time. But the challenge that I'm running into is this tarball, right? And so if it's 3.3 compressed, 29 decompressed, I clearly can't decompress it and then work with the files individually. In the last episode, we talked about using the archive R package to read individual files from that tar archive. The challenge though is that with a single processor it was going to take about 17 or 18 days to get all the data out of the archive that way. Sure, I could parallelize it to use those three processors, but we're still looking at about a week just to get the data out, right? And so I think that actually is beyond the length of time that a GitHub actions can take to run. So this again is a constraint and I was really annoyed by this because it's just something I didn't want to have to worry about. But I kind of like constraints in a way because they force you to think differently and to really think about your tools and think about your problem, try to kind of think through things differently. And so this got me thinking about the tar tool in the command line, the bash system that I'm using here on my Mac system that you can also get for Windows through the bash subsystem. And that obviously will also run on Linux and thinking about tar and whether or not there are options that I can use with tar to get the data I want in the format I want without having to decompress the entire tar archive. So I'm going to head over to my virtual studio code and kind of walk you through some of my thinking. As we go through this, the fundamental tool that we are going to be using to achieve the successful outcome is a pipe. And again, we've talked about pipes in R, but now we're going to talk about pipes in bash. Go ahead and open up a blank document to do some work in. We might save this in the end, but for now, I want to be able to have some place that I can type the commands I'm running down below in the console. Okay, so let's think about the problem we have. So again, we have 122,000 files that are all connected together in a big tar ball, right? That file then was then compressed using gzip to make it smaller, right? And so what I'd like to do is to get out of that tar ball, the individual files that are, you know, in that whole tar ball, but I want each of them to be compressed. My hope then, with that compressed, they'd be about the same size as the compressed tar ball. That's not exactly the way the algorithms work, but it should be pretty similar. Certainly, it should not be 29 gigabytes like the decompressed version is. Let's go ahead and look at the documentation for tar. So we're going to tar hyphen hyphen help, we can then see the arguments for the tar function, right? And so we see C for create. So we can see F for the file name we've used this before V for verbose, right? So in the past, we did like tar X V Z F. So V for verbose F for the file name where we give it the archive, Z is for compressing it with gzip, right? And then tar X is right here of extract, right? And we've also seen TV F, right? And so T is to list V for verbose and F for the file name, right? We could also create an archive, where we might do like CV Z F. So create verbose, gzip compression, and then the file name being the F, right? So again, there's a lot we can do in here. Something else we've seen in here is the hyphen C to change to whatever directory you state before processing the files. As I look through here, there's nothing that really sticks out at me as a way to get individual files out, right? And so to remind you, let's come back here and grab one of these dly files, any of them will do. And so we can do tar X V Z F data, and then ghcnd all dot tar dot gz. And I'm going to extract the ghcnd all, and then that deal of y file, I guess I didn't need to type that in there. So running this then will make a directory on my project root directory called ghcnd all, and then the name of this dly file. And so you can see how long this is taking to extract a single file. It wasn't just the functions from the archive package that was slow, but this process of removing a single file is actually quite slow. So this then extracts the file. Again, I now see I've got that ghcnd, ls ghcnd all, and then I can look in there and I've got this file. I'm going to go ahead and extract a couple other files so that I can have something to play with here, as I'm testing out different approaches. So I'll go ahead and grab these other files. And so I'll do I'll repeat my tar, and I'll give those three files. And so we'll extract these three files for practice into a ghcnd all directory that's in my project root directory. So I've extracted these three dly files, I'm going to go ahead and make a practice tar ball. And so again, we'll do tar cvzf again that C is for create this x is for extract so C create x extract, and we'll create a practice tar.gz. I could also call this tgz. You'll also see that as a shorthand for tar gz. And then I'll give to that ghcnd all. And so that then adds those three files to my tar ball. And I get I can do the listing. And so the listing again, do you remember that argument? It was a T. So I get tvf on practice tar.gz. And so then I get my ghcnd with those three files included looking back at the arguments for tar. Nothing really stood out as a way to kind of sequentially go through and extract each of these files in turn, right? So again, my vision was to pull out a file and gzip it pull out the next file gzip it and so forth, right? But if I think back about what I really want, what I really want is a concatenated file where I want all 122,000 files concatenated together. An advantage that I have is that each of those DLY files starts with the name of the weather station, right? So I've got this name here US1 IACW0005 daily. If I do a head on that, I can see right here, if my screen stops moving, right here, I've got the same name as I have right there, right? So again, if I can somehow get all of the files out, concatenate them together, and then compress it, that would be successful too, right? And so again, looking back at these arguments, there is one that kind of sticks out at me as being interesting, hyphen O, it's not a zero, it's an O will write the entries to STD out, don't restore to disk. So STD out is standard output, right? So if I were to do tar and then capital O, xv, zf, so the same arguments I would use to extract on my practice.tar file, this should output the contents of my entire tar ball to the screen. So running that, I see sure enough, I get everything outputted that is in that practice.targz file. But the next thing that I want to do is somehow write this out to a file, right? So I can write this to a file, right? By redirecting it with that greater than sign, I think we talked about this a few episodes back. So I'll call this practice.output, right? And so what this means is create a file called practice.output. If it already exists, recreate it as a blank document and take this stuff on the left side of the greater than sign and send that to practice.output. If I have two greater than signs, then that's going to append to the file if it already exists. And so now I've got practice.output. So if I now look at my practice.output file, I see that I've got all of my different data files in here. So there's the one with the 008, 006, 005, I think these are relatively small data files, right? But you can see that they are concatenated together. Cool. So let's come back to our terminal, right? And what I might like to do then is gzip, right? So gzip is applies that compression algorithm to any file I give it. So I can give it then practice.output. And now if I do lslth, I see that I have practice.output.gz. And this is a compressed version of practice.output. Again, I can do gzip-help to get the help documentation of the gzip function. And to kind of help us to see what's going on, I can use the hyphen k to keep the input file. So again, I can do gunzip to unzip the gzip file on practice.gz. Sorry, it's practice.output.gz. And again, I can repeat that gzip giving it hyphen k to keep the input file. So again, if I do lslth, so I see the concatenated file practice.output is 47 kilobytes. So it's a little bit more than 10 times the size of the table. But then when I take this and zip it, right, it then becomes 3.8 kilobytes. So that's actually smaller than the input file. So that's pretty cool. All right, but you'll notice that we did this with an intermediate file. I don't want to generate this file because it's too big to store on the computer that I'm going to be using. So what can we do instead? Well, let's come back and take the different commands that we've already entered. And so to see the history of the commands I've run, I can do history. And history then outputs all of the commands that I have run this far in my session. I guess it goes back quite a ways because this is the number of commands I've run. I haven't run 531 for this episode, trust me. All right, so what we can do is go ahead and grab 523, this one. And we'll go ahead and add this gzip hyphen k. So if I didn't want to have that intermediate file, I could remove the hyphen k. I could again paste these down in. It's saying actually it's warning me that practice.output.gc already exists. Do I wish to overwrite? Yes. And again, I could look at gzip hyphen hyphen help, and there's a hyphen f to force. So I could do gzip hyphen f to force it to output it. So I ran that and I no longer get that warning message. We've got two steps. We have this practice output as an intermediate file. That's not what I want. So something that is cool about Linux commands is that you could actually pipe them to each other. So as I showed you, we could do like ls ghc nd all that shows me there's three files in there. But say I have more than three files, or I didn't want to look, I didn't want to have to count, right? How would I count the number of files that are in there? Well, there's another cool tool called wc. So to show you a little bit about wc, we can run it on one of these dl y files ghc nd all us one I all that running that then gives me four columns of output. This 44 is the number of lines in the file 1971 is the number of words in the file, and 11 880 is the number of characters in the file, right? So I could do wc hyphen hyphen help. This complains because it doesn't have help with it. So the alternative would be to do man for manual on wc. If I make this larger, I see the manual page for wc. Not every program from the command line has help documentation that you get with hyphen hyphen help. Not everything has a man page. So it kind of depends. But what you'll notice is that there's a variety of different arguments that you can give wc. It has clm and wc is the number of characters, l is the number of lines, m ends up being the same as C for most purposes, and hyphen w is the number of words, right? I can go ahead and do q to get out of that. And so what I want to show you that is if we can do wc hyphen, l ghc nd all us one I all that, I get 44, right? So what I can do now is if I have ls on my ghc nd all, right, I can now send this to the wc function. And that is with a pipe. And so this vertical line in bash is the pipe character. This gets confusing, right? Because in R, the percent greater than percent is the Magridder pipe. And the new pipe in base R is a vertical line with a greater than sign, right? In bash, it's a vertical line that actually looks like a pipe. Whereas the others don't really look like a pipe. Anyway, I can take ls ghc nd all pipe to wc hyphen l. And that then will tell me three, which means that there are three entities here in this directory. Okay, so we can take output from one bash program and send it to another program. That's pretty cool, right? So again, what does this have to do, Pat, with what we are doing here, using the tar and gzip function, come back to our script here, I'll go ahead and close this dly. And what I could do is because this pipes out with capital O to the standard output, I can pipe this into gzip. So if I copy that and run it at my prompt, it then says gzip standard output is a terminal ignoring. So I need to redirect this to a file that I'll call practice underscore pipe dot gz copy and paste this down. And so now I didn't get any of the errors that we've gotten the previous execution. If I ls lth on that to again, list it out and get the sizes of the different files, I see that practice pipe and practice output are the same size. And you'll notice that nowhere in here did I create the intermediate file. I think we know enough now about tar, gzip, the pipe, and some of these other great tools from the command line that were ready to go ahead and do the processing on our Noah data to get that concatenated and compressed file without first going through an intermediate file. So I'm going to go ahead and save this script to my code directory. And I'll call it concatenate dly dot bash. I'm going to come to one of my other files and grab the shebang line there. So I can paste it up there. And we can now save that. Before I forget, I'm going to do a chmod plus x to make it executable on code concatenate bash. And so now it is executable, which means I can do code concatenate dly dot bash. And that will run everything in my command line without me having to copy and paste things back and forth. But of course, we want to process our ghc nd all file, right? So again, the file is in data. And so if I put the star, I'll get the full path. And so we see we have the data ghc nd all dot tar dot gz. I'm going to go ahead and remove those lines and plop the path here. So we'll again output everything to the standard output, we'll compress it, and then we'll export it.  And so this should work. And I'm going to go ahead and run this and I'll keep track of how long it takes to run to see if it hopefully is a lot faster than 17 days. Okay. So that took about 12 minutes to run through. If I do lslth, I noticed that ghc nd cat I accidentally left in the project root directory. That's because I didn't have data forward slash here, right? So if I go ahead and move that, again, move another program from the command line of ghc nd cat to data. And I do an lslth on data. I now see that I've got ghc nd cat, again, 3.3 gigs, ghc nd all the tar ball is also 3.3 gigs, right? So the concatenated and then compressed file is the same size as that tar ball. So as this was running, I was thinking about some other ways that I could make this even smaller. So one way that I could make it smaller is by noticing in one of these dly files, that one of the columns that we've talked about in previous episodes, again, this is a fixed width formatted file. And so there's a four character or four column variable here of the type of data being stored, right? So here we've got PRCP, which we want snow, which we don't want DPR, I don't want, right? And there's a variety of others in these files. But really, all I care about is PRCP, the precipitation, because I'm looking at drought. What I could do is add to this pipeline a grep function. So I could do grep prcp. And that then will take the stream of data coming out of tar, run it through grep, which will return any line from that stream that contains PRCP, send that to gzip, and then compress it as the outputted file, right? So I think this couldn't actually make it quite a bit smaller. So let's go ahead and run this. And it might take another 12 minutes. But let's see what the size shrinks down to after we've run this. So that only took about four minutes. So that actually ran quite a bit faster by getting rid of everything except for the PRCP lines. Again, if I do LSLTH on data, I now see ghtndcat is 850 megabytes, which is quite a bit smaller about a fourth the size of the ghtnd all tar.gz file. Great. So I think we've done a good job of getting out a small concatenated version of the file much faster than three weeks, or even one week. And that's a lot more attractive. So again, I know I often get people in the comments that say, why are you doing all this stuff in Bash when you could be doing it in R? Well, this is a case where actually I can't do it in R because it's going to be too slow. Instead, doing it here in Bash makes things a lot faster. And with, you know, knowing a few little tools and concepts such as piping things through and redirecting the output to a file, we can get a pretty good result fairly quickly. I'm going to go ahead and save this. I need to add this to my snake file. And so let's go ahead and minimize that. And we'll come down and I will add to where I have on git all archive. I'll call this rule. concatenate dly files. Right. And then my input, as we've seen in the past, we'll do script equals code forward slash concatenate dly dot bash, right? And then my output. What did I call it? The output was this, right? So it's always easier to copy and paste. I don't make so many mistakes that way. The other input is that Tarball, right? And so I'll call Tarball. And so even though I'm not going to use Tarball as an argument, because I've used script as an argument name, I have to also name the Tarball, right? That's a convention within snake bake that they enforce. Otherwise you get errors. And also if you have two values within a directive, they have to be separated with commas. And so then for my shell, I'm going to go ahead and copy this and kind of clean up my my spaces here. And I don't have a params file. So I'll get rid of that. And so now I will add the output here to my targets, right? So I can add that. And so again, when I run snake make, it will also generate this file if it needs to be updated. So let's go ahead and update our R script, right? So I'll go ahead and remove some of this code that I don't need anymore. I don't need this stuff with the map. So I'll go ahead and remove that, remove this sys time. And the read fwf can read directly actually from a GZ file. So I'll do data forward slash ghcnd cat dot GZ. And so I've got an extra parentheses at the end here. So I'll go ahead and run everything to that point. So that took a couple minutes to read in. Again, it's a large data file. And it's compressed. So whenever I talk about reading things in with big data files, I generally get a couple responses. So one is the suggestion that we use the f read function from data table. My investigation of f read is that it will not read in fixed width formatted files. So that won't work here. The other approach people suggest I use is to use the functions from the room package room room, like a car or motorcycle revving its engine. Well, it turns out that the developers of room are the same as the developers of read R. And if you look at the code inside of read fwf, it's using room already on the back end. My understanding is that they're pivoting from room to read R to make kind of a seamless transition there. So that more than likely we're already getting the benefit of the benefits of using room. So that's all good. One thing I want to check is the memory footprint of my data. So if I look at top and top is a handy dandy function, another program from the command line that will show you information about the different processes running on your Linux based workstation. If I hit the o key, this will then allow me to type in different keys that I want to sort this table on. So if I sort on mem, that will sort on the memory column. So we find then that R is using close to 10 gigabytes of RAM. You recall that I said that with GitHub actions were limited to 14 gigabytes of RAM. And so we're getting kind of close to the ceiling. And so we'll have to see what happens when we use the rest of the pipeline to go ahead and pivot this longer. Again, our days now are in the columns and we're going to pivot that longer so that we have a column for the station, the date and the amount of precipitation. So I'm a little bit worried that we will go over the ceiling of memory utilization. So if you want to find out the exciting conclusion to that story, please be sure that you check out the next episode, which I will link over here. Keep practicing with all this. I hope you enjoyed learning more about bash functions, and we'll see you next time for another episode of Code Club.


This is a peristaltic pump and in this video I show you how to use an Arduino to control it all while we build this test rig for future videos Where we're gonna test water level sensors So in this video, I will show you how this pump works on its own How to control it with code with an Arduino and how to make a switch to reverse the polarity So if I put it here It'll suck water out of this cup that it's been putting water into and bring it back into the main cup And all I need you to do is give this video a like and subscribe to the channel if you enjoy this type of content Recently I figured out that 95% of you aren't subscribed to this channel and you're watching videos It doesn't cost you anything and it helps me out a ton So as I said before these peristaltic pumps squeeze a tube and force the liquid into a direction By causing waves in the tube these waves cause directional pressure and a vacuum on the back end Just like at a beach with waves that come crashing down They come and they push everything towards the end of the beach And then when there's no more water they go back and suck it all in kind of like a curtain The nice thing about these peristaltic pumps is that there's no chance of contaminating whatever's in the tubes Because all the mechanisms that cause this pump to work are outside the tube So they're really popular in medical fields in foods In dangerous chemicals because the actual mechanism has no contact with the chemical inside the tube It only has those rolling wheels when you turn them like this It's kind of hard to do with your fingers When you turn it like that you could see this one right here is just going around and squeezing the tube As it comes around in either direction But there's actually no contact with the liquid I could just take this out Take the tube out and it's still the whole the whole tube system is still intact regardless of this pump So it's kind of like an add-on. It's also extremely easy to do maintenance on like if you need a new motor Just take this out and then put a new one onto these two pieces and actually goes together like this That clicks in there That clicks in there and then that's it those three pieces now made me a pump That will not contaminate the liquids usually in these type of videos we take an arduino We connect it to whatever we're reviewing we make it work and that's it But in this video, this is actually going to be a project for the next five videos on the channel What we need to do is make this arduino and this pump pump water it in out of a big container like this And then we have all these different types of water sensors that we're going to review in each video individually And we're going to use this rig to test them So the goal would be that with arduino and the pump we put the water into the container until the level sensor says the water is too High and then we cut the water Uh, you know, we cut the pump off So all we really need to do is put water in and out of this container We're going to need to reverse the direction of this pump and we're going to do that all within arduino If you want to actually copy this build You're going to need a relay to use the arduino to control the pump and then i'm going to be using this Rocker switch to control the direction of the pump So for example, this could be left and this can be right and this could be like an emergency shutoff in the middle So the arduino will plug into the relay the relay will plug into the rocker switch The rocker switch will plug into the pump and then the pump will pump liquid in and out of our big Trainer here and we're going to need a water source which will most likely just be a cup on the side of the screen That you can't see filled with water. Oh, you're also going to need some tubing Now since this uses water and I have a computer right here off screen. I'm going to be using this little I don't know what to call it like lunch tray Just to make sure you don't get liquid on the computer You're also going to need a power supply for your water pump in this case This water pump is a 12 volt dc water pump where at least the motor is so I have here a power supply I think I took off of an old laptop It's 12 volts 2 m's which is going to be perfect for what we're doing. So first off, let's build it out on the board here Let's connect all the cables all the tubing and everything and then we'll go and actually write the code to control it all So to keep this organized. I'm just going to start heart gluing stuff to this Lunch tray just to try and get it Kind of clean So there's our Arduino board It's going to need some more glue for sure I'm not going to glue the rocker switch yet because we are going to have to Sotter up these pins in the back before we glue it down so we can you know control the directions So for now, I'm just going to place it right there. But these two look good And then same thing with our pump. We have not finished yet with connecting everything So I think we're just going to wait out on that For a little bit. I don't want to spend too much time in this video showing you how I made this reversible switch So I'm going to keep that for another video in the series But for now I'm going to put a picture up on the screen if you want to follow along to do it yourself And then on the right I'm going to put a picture of what it looked like when it was finished All right, so now we're done with the switch. Let's connect it up to this dc motor It's got this one on the right here with that little red dot tagged as positive And this pin is going to have to be negative So I'm going to grab some cables from my trusty box over here So I'm just going to grab these two cables here and make those connectable for the water pump All we have to do is repeat the process of Stripping them down twisting them up Sticking them through these pins just like this And that's what it should look like Then just go and repeat the same thing for the other side All right, so back to the test rig we go. You're going to need your hot glue gun Your newly soldered h-bridge reversing polarity switch whatever you want to call it and your water pump Just to be safe, I'm going to put my water pump inside this little tool container It's not going to stop a massive leak because we're going to be working with way more water than can fit in here But at least if I see a little water leakage or won't leak over to my electronics or to my power supply That's connected to the wall or my laptop. I'll just be able to unplug everything and Take care of that leak issue. So first thing I'm going to do Is just get some glue And then get that down onto our board Now we're going to take our water pump going to lightly glue it I don't need anything crazy because it probably will come off at least once get some glue where those bolt holes are And then I'm just going to do this We need a good amount of glue because I will be pushing this switch a lot And then motor side will go to the motor And relay side will go to the relay just like this I don't want to touch the hot glue with my fingers. So I'm just going to scrape it off with this All right, so we have our do we know we have our relay We have our polarity switch or h-bridge switch whatever you want to call it And then we have our pump the pump connects directly into these two cables coming from the h-bridge switch Then on our h-bridge switch, we have our power supply, which is going to be our relay We're going to plug in our positive terminal from our switch To our relay now this is going to be the the one thing we really really really don't want to get wet So this right here we just cleaned up here Is our negative from the power supply or 12 volt power supply that's going to go directly into the polarity switch Now this is going to plug into the middle Port which has co-m on it. So now we have our positive and negative from the power supply connected to our relay and to our switch Which both go into our pump Our relay isn't connected to the Arduino so we could do that but first Let's just clean up this cables for that. We're going to need three different cables We're going to need a black cable to go to the minus And that black cable will be going to ground on the Arduino Then we need another cable. I'm just going to use brown to our plus on Arduino Which is going to be a five volt supply And then we're going to need another colored cable, which I'm just going to use green That goes on to s and s is for signal And we'll do signal on three Three on the Arduino board now that we have our rig built out We have our Arduino connected to a relay our relay connected to a power supply Which then interrupts that power supply to a switch, which allows us to reverse the direction of the motor So we'll do one side will be to pump water into the into the big box right here Then one side will be to pull water out and then eventually in each specific video The Arduino will be connected to a different type of level sensor Like let's say we're using this big float sensor And when the sensor detects that the water level has reached the highest it should be It'll tell the Arduino and that pump should shut off. That's the goal of what the videos will be And that's why we built this whole test bench So let's connect the Arduino board to our computer and then write some code to make this pump work When you upload this code I put up on the screen right here The relay is going to flash a light and make a loud click for one second And then it's going to turn off and repeat over and over there we go There's that light on and off. So I just changed up the code I'm going to have it initially turn off for a second Then I want it to turn on for one second and then turn off for five and repeat So that's one second on one two three four five Back on for one and then repeat. So I'm just going to Comment this out and upload it to the board. So only this code runs right here So the relay should stay off We're going to uncomment that. So I'm just going to plug in the power supply into the wall No sparks no smoking. I think we're good I don't feel any heat. Now let's upload this code to the Arduino and see if it does what it's supposed to do Now it's off for five seconds and then the motor should turn on for one second Now that we know that it works and it's not burning up Let's change that five seconds to one second and let's repeat the process I don't know if you can hear this so I'm just going to move the microphone All right, so we actually got this working if you put it in the middle switch here It's going to stop the motor from working and if you put it in this one it will turn the motor the other direction I'm just feeling it to see if it's hot. It seems like we're good Now the motor is back in an emergency off. I guess you can call it But if you put it in here, it's one direction put in here. It's the other Let's plug our pump into some tubes get those tubes into a cup of water And maybe another cup to dump the water in and then let's see if this thing actually works We know we can activate it But we haven't figured out yet if we can make it do what we're trying to do And hopefully it will not make a big mess and fry everything So let's go over this one more time. Arduino goes into relay relay controls a power supply Which goes to a reversing switch to reverse the motor in and out So that water can be sucked out of the cup or put into the cup And then That's pretty much it So i'm just going to run the code that turns the water on and off And i'm going to put it five seconds on and then five or we'll do five seconds on two seconds off All right, so we're we're getting bubbles in this cup right here, which means we're trying to push out this way Now it should start sucking water. There it is It's going down the tube. It's stuck right here Keep going It's in our pump For some reason there's red stuff in our pump. That's pretty cool And now our cup our little mug here is filling up with water Well, that's it for this video in the next couple videos We're going to take water level sensors like this one right here where it has a ball on a long stick And as the water level rises the ball will rise with the water level telling us where the water is We're going to use sensors like these inside of water containers And then we're going to use this system to bring the water level up until the sensor tells us It's enough or until it overflows and burns everything I have If you enjoyed this video one more time, just give it a like and subscribe to the channel and I'll see you in the next one


It's a windy day on the Black Sea, as the flags of the USS Vicksburg wave violently. Since December of last year, the USS Vicksburg, along with other ships assigned to the Standing NATO Maritime Group, are conducting training exercises in the region. We'll be conducting communications, maneuvering some gunnery and some air defense exercises with them. And what we really hope to do is to build the integration and the interoperability among the units. The Standing NATO Maritime Group is a multinational integrated maritime force. As a member of that team, the USS Vicksburg is working diligently with its counterparts to ensure mission success. Are coming here to train with them is just as important as training with any of the member nations of the Alliance. From shape Belgium, I'm Army Sergeant Vincent Pettikin, AFN News.


Welcome back to VMworld 2013. This is our special live coverage live in San Francisco, California. This is theCUBE. It's looking at Angles flagship program. We go out to the events, extract the signal from the noise, talk to the tech athletes, talk to the entrepreneurs, talk to the customers, talk to the partners, talk to the execs, join my co-host Dave Vellante. And we're here with a very special guest, Sanjay Pune, the new GM of the End User Computing business unit at VMware. Welcome back to theCUBE. You're a CUBE alumni from SAP, of which we've done for four years as well. It's our fourth year at VMware. Great to see you on theCUBE. Nice to be back. Thank you very much. Executive at VMware. So what a year it's been. The whole theme has changed in 10 years. A 10 year anniversary of VMworld. Defy convention, change. In our past conversations, you can handle that. You have so much experience at SAP. Defying convention, that's me. Defy convention and VMware needs some help at the end user computing area. Strategic area. So over to my data center, hybrid cloud, end user computing. Pat simplified it down to three areas. You're heading up end user computing. Strategic, to say the least, for VMware. What's your take of what that is today? Where has it been? And what motivated you to come to VMware from SAP, which had great edge, mobile, mobile analytics. Everything's happening at SAP. With mobility, you know, I think. What got you here? I've always loved being in your show. So first off, thank you guys for having me back. I looked at the way in which end users have been using computing. And as you know, I've been in the business user space practically all my life. Analytics, big data, in the last 15, 18 months in mobile. So I saw this as a great opportunity for a company that was looking to extend their brand from the data center to the desktop and where the desktop is going. My views of the desktop obviously is not just the laptop, but the future of these mobile devices. So the desktop of today is clearly going to move into the cloud, there's mobile, and then there's machines.  These are all the ways in which we think about them. These are all going to be end user computing devices. As I looked at this- Personal computers. Absolutely. Personal computers. There is more software today in your car than there was in the first spacecraft in 1970. Believe that or not. So from my perspective, if you think about the types of players and technology infrastructure players that are going to be strong, I believe that need to be three core disciplines to what that company either have. In terms of technology, and I'll cover the other aspects. One was management technology, the other was security and virtualization. And quite frankly, I felt that VMware had much better assets than any of the other companies to make that happen for what end user computing. The other aspect of it was VMware has always been a company that's valued innovation. You look at the roots of this company, it's an innovator. Product and go to market, it's an innovation company. I wanted to be close to an innovating team. The culture is very much of innovation from product to go to market. And then, you know, from a joking perspective, thirdly, my commute got 50 yards shorter. I used to turn left, now I turn right 50 yards shorter. That was a factor to me. And the campus is nice at VMware, and this beautiful campus is where I was at SAP. But I got to ask you about the end user computing. Because, you know, one of the things Dave and I were talking earlier, we love that stack that Joe Tucci, Pat Gelsinger, and Paul Meritz laid out in 2010. You know, four years ago, and it was something called the software mainframe, cloud computing, and it was beautiful. Nice, we all recognize that, yeah, fill in the stack. But there was some misfires. Virtualization expanded, you got flash, you got data fabric, all kind of was happening. Software defined jumped in there and changed a little bit. Didn't change the direction. But then, at the top of the stack, there was some misfires and just didn't feel right. Was that a factor that come in with a clean sheet of paper? Was it a turnaround situation? Was it just, okay, reboot, reset, grow? What was the dynamics at the top of the stack that you felt?  So that's always a good place to start. You're not creating a brand, you're building on that brand. That was very, very important. But as you looked at where the data center was going, it's moving to the cloud and cloud computing. And we have a clear strategy there to play in cloud computing. That was very, very important to me. But in terms of where the end user computing opportunity was, I saw this as just being the first inning, the VDI space, virtual desktops. It really is a two player market there. And I think our competitor isn't innovating very much. We have an opportunity to be the innovator and gain market share and overtake them to be number one. I'm here to be the number one player in VDI. But mobile, we're just in the beginning of that inning, if you would, the first to second inning. It's a wide open market, social computing. We've got social cost, it's a great product. Tim Young's the CEO of that company. He's the real deal in social computing. We think that those areas, desktop, mobile, social, and where they're going to go are at least the first three aspects of where end user computing has a lot, a lot of opportunity. So Sanjay, the first three shows we ever did, we did EMC World was the first one we ever did with theCUBE. And right after that we did SAP Sapphire and then VMworld. It was always interesting to contrast the sort of infrastructure shows to Sapphire. And we saw the transformation of SAP pretty substantially, particularly after the side-base acquisition. So you had serious management, this top management thrust toward mobile. And then you also, the acquisition of side-base gave you some of the management technology, the security, and maybe not the virtualization piece. But you really began to change the discourse and the customers bought into it. So that was, I felt a catalyst just as an outside observer. Is that true? And then what's the catalyst here? Was it cleaning up sort of what I used to call the misfit toys, creating the pivotal piece and getting more focus, bringing in a new leader? What is that catalyst? You know, I think, fortunately I was very blessed to see a transformation of SAP from systems of record to systems of engagement. I was very involved, as you know, in the analytics, the data, and the mobile business. Much of that catalyst is still valid. Systems of record to systems of engagement. So a lot of what's happening in the end user computing area is those systems of engagement. That continues, that's abroad. But then as I looked at the IT trends, of big data, of mobile, of social, of cloud, there's at least three or four of those now that we have an opportunity to play here in a big way. Yes, there was an opportunity to redefine things with almost a clean slate in many areas. And a fantastic team. Both Pat and Joe Tucci were involved in recruiting me here. Have a grand vision and a great vision of where this is going. They're fantastic executives. And then the rest of the executive, from Carl to Jonathan to everybody else here, the world-class people that I felt I could just culturally fit in very well with. And in this area, as you know, business user stuff in end user computing has been my passion. So it's a very natural area for me to hopefully extend what I've done in the last 15, 20 years into something that could help this company. And then from a revenue perspective, if we can double the revenue of this company with hopefully a good part of that coming from end user computing, this will continue to be a very, very, hopefully well-marked, capitalized company. And also you mentioned SAP about the cloud. They've had some cloud issues. We've seen some shifts and some things there. But the mobile they were kicking butt on. We saw that right away and we were in that. But I'd like to get your view on the competition because David and I were talking yesterday on the first day about Balmer's resignation and just did a reorganization, go to the intelligent edge of the network. So the competition was Disruptor or Sustainer? You know, who are you, right? I mean, VMware, a technology-driven company is not sustaining anything, it's still disrupting. Microsoft obviously kind of sustaining the status quo. So the question is, who's your competitors? Is it a Microsoft? Is it someone else? Who's in your rear-view mirror and who's, who are you looking to laugh on the field? I mean, clearly, if you look at each of our businesses, they're different ones. But in the core data center stuff, traditionally completely Microsoft, but we are far and ahead. A market share leader, you know, in that at least 70% plus. So that's a very strong position there. And where the cloud is going, it's open mark, it's a complete open opportunity. I think we heard from Pat Gelsinger a message where we're going to embrace an appropriate place as open stack. So you're going to see this not be a combative, but a place where it's collaborative in some places. In the end user computing area, we are in some areas, number two behind Citrix, but I view it as a huge opportunity. We're bigger than them and as a brand, as a company. And my ambition here is to play to be number one. And in the areas of mobile, there is no clear leader. It's very much an open area, especially as it relates to mobile management and mobile security in the enterprise mobile world. And the good news I think in the end user computing area, especially as you think about the new aspects of mobile, there is no one device operating system. In the laptop world, you guys both have Macs, but Windows is still about 80, 90% market share there. In the device operating world of mobile, it's a heterogeneous world already from the get go. iOS, Android, Windows, mobile, and that actually creates much more of an opportunity for a Switzerland type of player to be a leader in management, security, and virtualization. So much of those dynamics I think requires to be innovative. But my general attitude on competitors is, you don't obsess about it. Clearly, you have to have ambition to be number one and competition keeps you honest. But I'm much more focused not on who's in the rear view mirror competitors. I described some of them in that league, but a lot of where we believe we should be going.  Cisco calls it the Internet of Everything. I love theCUBE because we get to do all these great events. We were at the GE Industrial Internet launch and I know Pivotal is a good part of that. So I wanted to, it was great, because you get to talk to all the smartest people like you and extract their knowledge. So I want to understand the roadmap for the customers. So you take down the enterprise of what used to be known as VDI. Is it a parallel path to the Internet of Things or the Industrial Internet? Talk about that a little bit. Yeah, I think, first off, you've got to think about where the desktop is today and where the desktop is going. Because that's the primary aspect of where people have traditionally been using, their computing time has been on the desktop. 10 to 20 years ago, you probably did all your email on a desktop. Today a significant part of that is on a smartphone. As little as possible on a desktop. So the world of the time spent between a desktop and a mobile device is changing. We've seen it, especially since iOS came out. Now, the fact of the matter is that computing inside machines is not a new concept. It's been there for a while. But the ability to manage them, to secure them, potentially virtualize them is something that's very nascent and one where I believe there'll be an opportunity. So my view is that whether it's the desktop, whether it's a mobile, whether it's a machine, the technologies are going to have some common substrate or fabric that are going to be common to all of them. But there's going to need to be some miniaturization. You can't apply potentially the exact same software you have in the data center to a tiny thermostat. And the beauty of it is we've got smart engineers. We understand the miniaturization of where this technology needs to apply. And we're going to play this out. I think the machine-to-machine, internet-to-things opportunities, well ahead of us. We're in the beginning of that. But the good opportunity there is it's 50 billion plus things, tens of billions of devices, and so on and so forth. The other thing that I think is important is that there are other technologies. I mentioned management, security, virtualization. There are other technologies that I think that are also going to play. Social computing. What is the appropriate place for how people are going to collaborate in processes? I don't think that's been, it's well understood in the consumer world because you see the Facebooks and the LinkedIn. But there is some mirror of that in the enterprise, which I think is a wide-open opportunity. There could be machine learning, big data, those kinds of trends will drive a lot of that. And how you're going to collaborate not just between man-to-man, man-to-machine, machine-to-man. Unsupervised, yeah. Unsupervised machine learning. That's the automation piece. That keeps on coming back as the automation. I got to ask you since you brought up the thermostat thing and you were kind of riffing on that. You had a lot of experience and certainly SAP in the verticals. It's interesting. Mobile has different verticals. You talked to financial services, government. Is that a constraint or an opportunity? I think it's a great opportunity. I think that any company that grows horizontally, which is what we've done. You know, we have five, I mean three, the most baffling stat when Pat and Joe talked to me was that VMware has 500,000 customers. That's huge. But most of that growth has happened horizontally. There's been gradually an industry verticalization. One of the things we did very well at SAP was industry verticalization. Now, SAP is a 40-year-old company. You'd expect that. What we're going to start doing in end user computing is starting to verticalize some of our aspects of how we sell. For example, in VDI, we're finding a number of healthcare places where they want to equip physicians and clinicians with either laptops or devices now where you can get to their clinical records, medical records on a device that's completely locked down and virtualized so that you don't have to worry about that device or that laptop getting locked. That's a real, real opportunity for virtualization in healthcare. The same in government. We're seeing the same in many of the very, very sophisticated branch and local type of remote office type of settings. So increasingly, we think both the desktop and mobile, they're going to be vertical use cases and we'll start building out technology and both solutioning of them that allows us to go once in a while. Absolutely, by industry. And I think some of them that are very relevant are healthcare, banking, federal, and then several of the other sectors where there's lots of employees very distributed in sense and also ones where you've got to ensure that the data doesn't leave their device, laptop or mobile. Sanjay Pune here inside the cube now with VMware, GM of the end user computing group. Thanks for coming inside the cube. I'll give you the final words I want to ask you. Obviously you're a senior executive, great leadership move for VMware. You can come in when you're running shoes on. There's no real delay there. So I'm expecting to see some great stuff from you but what's your objective? You're going to get ingratiated into the new culture which is not going to be hard for you. Again, left turn versus right turn. You understand tech. What's your personal objectives over the next couple months as you get in and start to put together the plan? I mean, obviously the metrics are clear. Objectives are clear. SDDC, hybrid cloud and user computing and peer piece. What are you going to roll out? I think it all, as you know in any software company your assets are your people. So my first focus is to really understand the fabric of our team, our people. It's a smaller team than SAP. I don't worry about 65,000 people here, it's 15,000 but in my own team, getting to understand how we can continue to retain and grow great talent. We have a fantastic team, not just at VMware but also in my immediate team and we're going to continue to grow that team. And in that I think you'll find great innovation. Secondly, we're going to define goals that are revenue and market share related that are both short-term and long-term to be number one, the undisputed number one of every aspect of our market. In some of our markets, there are existing competitors in other ones that we're completely creating new markets. And third, we're going to go and make customers enormously successful. I think when you make customers successful with great innovation and great people, you have a fantastic business. And I could envision over a multi-year period, imagine that VMware is twice the size that we are today, five billion dollar company. If a significant part of that could come from end user computing, that's going to be a great morale boosting.  Rent down 23,000 people estimated here. And it's just growing right up, straight up. So VMworld. But you're going to envision that 50 billion dollar TAM, 39 billion dollar market cap. I mean, I could see that within the next five years. I hope there's a double the revenue. You guys do the estimate where the market cap is. But double right along with it. And then you got to figure out, we talked to Pat, you got to figure out the bigger TAM, and then it's got to go for the $100 billion TAM. We double the revenue in triple the market. Those are all good problems TAM. Now we're dreaming a little bit. But I think you got to dream the future before you create it. These are open markets. The most inspirational thing we could do is help our employees and our customers dream that future. And that's what I'm excited to do. Sanjay Poonan, we're here at VMworld. We'll be documented. It's our fourth straight VMworld. We'll be continuing document when we're here in theCUBE. Thank you for coming on your tech athlete. Looking forward to seeing you with your running shoes on and user computing needs a big lift. They have the right guy for the job. We're right back with theCUBE, with our next guest, our friends at Andresa Horowitz and a hot startup in the space in virtualization. We'll be right back after this short break.


Hey folks, I'm Aaron Massey. I'm a software engineer with Google on the Chrome OS embedded controller firmware team. You may also know me from being a maintainer of the Zephyr FieldGage API, as well as of emulation in the testing framework as well. So we have a goal. We want to scalably verify peripheral drivers. Now let's dissect this a little bit. Specifically, we have, let's say some chip for IC. Let's say it's a FieldGage as part of a smart battery pack. Well, we have some driver that knows everything about that FieldGage. It knows about its registers and it knows how to talk to it. What kind of, like is it SMBus, is it not, what have you. We want to verify that driver, what the piece of code that knows explicitly and is specific to that chip. We are not interested in verifying the bus controller driver here. That's sort of out of scope for what we're going to be talking about here. So just the driver that is specific to a peripheral chip. Now we also want to do this scalably. So what does that actually mean? Well it means we want it to be cheap and terministic so that we can run in CI. And what we're going to see later in this talk is that we can get really good quality and scalable testing by combining emulated testing with hardware testing. So let's get into it. We're going to talk about peripheral emulators and we're going to talk about when or why we should emulate. Like, is it worth the engineering cost? How do we create an emulator? And once we've created one, how do we actually use it in a test? Then we'll have some final comments and I'll have some time for Q and A. All right, so what is an emulator actually? And by the way, I've gone from off-chip peripheral emulator to just emulator because saying off-chip peripheral emulator is a little long-winded and I don't really want to say that over and over in this talk. I also want to talk about this and talk about emulators and these peripheral emulators in an abstract sense right now because well, there's simulators, there's emulators, there's fakes, there's mocks and it's a little confusing which one we're talking about and it's a little overloaded. So let's start with this just so that we are all kind of on the same page about what we're talking about here. Okay, so let's start with a test. This is a test that has a hardware flow and an emulated flow and it goes from left to right. Cool, we have this config emulator here that basically controls if it's emulated or not. So let's step through this. So we have some tests. Let's say we're testing a field gauge driver. Well, that test is gonna call some field gauge driver API or it doesn't have to be a field gauge, it could be sensor, it could be a lot of other things. So it's gonna call some driver API as a test would, as any client would really. Well, that control and data flow is gonna go from that test into, well, the driver that implements that driver API. So that driver API or rather that driver is probably going to want to do some kind of bus reading or writing to its associated peripheral. So in this case, we're talking about I2C. It's gonna want to go ahead and do some I2C reads, just checking, hey, what's the temperature of the battery or what's our charge or capacity at? So, okay, cool. Our driver's gonna go ahead and talk to our field gauge. By the way, I'm starting with a hardware flow first. So emulation is disabled. This is a hardware test flow. I wanna start with this just so we can all be reminded about what a hardware test flow looks like. It's something I think we're all a little more familiar with. Anyway, so that driver is going to, well, try to do an I2C read or some kind of bus read. Well, it's gonna ask the bus controller driver to send these signals. The driver doesn't actually know how to break apart the I2C messages and send them on as little ones and zeros along the wires. That's kind of outside of the peripheral driver's purview. That's more of the bus controller driver, the part that we're not really interested in testing here. So, okay, the driver goes ahead and says, hey, please send this to my peripheral. And it does just that, right? In this case, for an STM32, it just goes ahead and it takes those I2C messages and sends them on the bus and goes over the wires directly to the peripheral. Cool, this is a hardware test. A lot of us are pretty familiar with it right now. So far, so good. Let's go through the emulation flow. Well, we have our driver test and we call some driver API. So far, no difference from the hardware test, actually. Well, that driver test, obviously, that driver API is gonna be implemented by some driver. And again, from the driver's perspective, there's actually no difference. It's gonna have to make those bus reads regardless. So, okay, the test, the driver. Now, here's where things get a little interesting, a little different from the hardware flow. We have emulation enabled. So what that does is that allows us to use a fake bus controller, right? The part that we said that we weren't really interested in testing here, that was kind of out of scope. Well, we're gonna include this fake controller that's gonna take these very abstract I2C messages from the driver and we're gonna forward them directly to the peripheral emulator. And then this peripheral emulator, it takes those I2C messages and based off, you know, is it read, is it write, goes ahead and modifies its internal data or just responds accordingly based off whatever its internal state is. And we're gonna get into pretty deep detail on these last two blue boxes on the right a little bit later. I just wanted to kind of go over an overview to make sure we all had like the right mental model of what we're talking about here. The main takeaway here, though, is that from the test perspective and the driver's perspective, like logically, there's no difference between a hardware test and an emulated test. So there's no difference in that source code. And the other thing here is that these emulators are emulating at a really robust transaction model level of abstraction. So they receive an I2C message of like a stop or a start or a write or a read. And that's the level of abstraction that we're operating at. So we have a loose abstract idea of like what an emulator it is in the context of peripheral emulators and Zephyr. So before we go any further, like is it worth the engineering cost here? Should we even do this? Here I have like pretty rough comparison of emulation and hardware and testing. So let's just go through some of the things that we get from emulation. So first, you know, it's hardwareless. Look mom, no hardware. I love writing that joke. But what's cool here is we can take the emulator and start prototyping the drivers against the emulator before we even have the hardware available. There's some earlier comments on that in earlier talks actually as well. That this allows us to at least get started and not sit there twiddling our thumbs while we're waiting on parts. Another nice thing here is with emulation, we get really scalable testing. We can run this NCI and it's pretty consistent. It's pretty deterministic, right? At least they should be. And it just works. It just runs. It's not slow the way hardware tests can be slow and hardware tests are also often non-deterministic because hardware is frequently full of non-determinism. Another nice thing here is code coverage kind of just works, right? You can just use L-Covered, G-Covered. It's a lot harder to get code coverage on a actual physical device. You might need to have a special test device that has extra memory to include and record that extra information. It's just extra setup. And again, we're looking to be scalable. So I don't know how scalable that is. Another nice thing here is when an emulated test fails, it's a lot easier to debug.  It's a little more involved to get debugging set up with an actual hardware board. You have to either simply have a J-Link or connect some probes or maybe all you have is like you are. So it's a little more involved, a little more work. And you don't get the same kind of control that you would in a emulated environment with GDB on your dev machine. Another really important item here is when you're doing emulation, it's a lot easier to go through and kind of fake a certain kind of test scenario or environment. It's a lot harder to do that sometimes in hardware. So as an example, right? Let's say you have a battery and you wanna test whether or not it's a drained battery or it's at some certain percentage. So you kind of have two options in hardware typically is you can use some specialized hardware that kind of fakes the battery readings or you can sort of just wait for the battery to drain. One requires specialized hardware. The other one lowers your life of your battery. Neither is really like I mentioned, scalable. Both emulation, we can just say, hey, pretend to have a low battery or if you have a temperature sensor, hey, be on fire. Be boiling. That's a lot more expensive to do your hardware. But we have one problem, right? We're not shipping the user an emulator. There's all these nice things about emulating. Good to know. Yeah, feel free to reach out to me a bit later and we can talk about that in scaling. So that is an approach. You just have to have people who can put that together and set it up, cool. So like I said though, there is a problem, right? We're not shipping the user an emulator, right? We need real hardware tests. We need real accuracy, right? We are shipping the user a hardware. But what we're gonna see is that we can actually use the hardware tests to inform our emulation and our emulators. And we can get the best of both worlds. We can get the control and velocity of emulation with some of the accuracy and fidelity of real hardware testing. So let's see what that might look like in a project. So here's a project lifecycle. We're gonna step through this both on a pre-hardware side and on a post-hardware side. When we don't have it available and when we do have it available. So first, we're gonna select some part. Maybe it's because it's what our user wants and something cool, but more often than not, it's because of some advantage in supply chain, right? We have something set up to actually get that part easily to us. Well, while we're waiting on the parts that would be put together and while we're waiting to actually receive them, we can get started, right? We have the data sheets. We know what other parts we're gonna use. We can start writing emulators and start prototyping our drivers against those emulators. Or maybe we have some stub driver from a manufacturer, but we can take that and we can actually just start prototyping and running some emulated tests with it to make sure it's doing what we want it to do. Like at least roughly, right? This is still before hardware, this is still emulation. I also want to mention right here, this is the best time to write an emulator. It's when you're writing your driver and you're most familiar with the chip and the data sheet. All right, so we get our hardware and we start actually using our driver. Awesome. Well, explosions. It doesn't work, or maybe it doesn't work as well as we'd want it to, right? We prototyped this driver with emulation and when we run against real hardware, maybe we're seeing some kind of problem. Maybe it's not being registered correctly. Maybe it's, who knows what? It could be a lot of things. Does that mean that, okay, this was a waste of time? Not quite. All right, well first we were able to get started and start talk about the design and development and prototyping of the driver before we got to that part, this part I mean. So at least we've got that covered. But the other thing we can do here is we can take our failing hardware test and we can use that to inform our emulator. We can use that to patch our emulator and our tests to actually improve our emulator and make it more realistic. So our hardware test actually becomes a test for our emulator as well. And so what we do here is we can patch our emulator to make sure it fails the emulated test the same way a hardware test fails. Well, once we have that emulated test failing for the same reason, at the same time we start patching our driver to make both tests pass. What we've done here is we've taken our hardware test failure and we've created a regression test. We've created a more accurate emulator and we'll catch the same problem before it actually makes it out to a hardware test lab, which is gonna be a lot cheaper. Now, something else that's really cool here is as we go through this process over and over because we're going to find more and more things we're actually becoming more aware of the types of mistakes we make in our emulators and we're using every cycle of this graph we're actually improving all of our emulators because we're learning how to emulate our hardware. What's also pretty cool is that this is basically test-driven development. We're taking or we're using the tests to drive our firmware development and our emulator development. So we have our red tests and we're turning them and creating and making them green. Okay, so now we have kind of a rough idea of some of the benefits we can gain from emulation and where it kind of belongs in an entire project really like the whole project. So we've been talking very much in the abstract, how do we actually create one? Let's get into it. Quick reminder, this is the best time to create an emulator. It's when you're creating the driver. It's when you're most familiar with the peripheral and with the data sheet. All right, so we want to create an emulator. Well, how hard is it? It's actually not too complex. It's about 400 lines of C code typically, documentation and whitespace included. We're not emulating an SOC. This is not QMU, all right? We're not cycle accurate to the peripheral. We're really just implementing a bus functional model to handle bus messages. Not going too complex here. Well, okay, so we've written the source code. We actually instantiate emulators pretty similarly to how you would instantiate a device driver in Zephyr. So we have the device DT define with its various parameters. Well, for emulators we have emulator DT define. And they share a lot of the same parameters. You've got the node ID and initialization function, some private runtime data and some static configuration data. But there are two parameters that are specific to emulator DT define that we also have to have. Well, actually you only need one. The other one's just very useful. That's the bus API and the backend API. So the bus API is the implementation of how that emulator responds to the bus messages. It's how do I respond to an I2C read? How do I respond to a write? Or basically, how do I respond? And the backend API, we use this to allow the test to reach into the emulator state and set up test scenarios like low battery. Let's start with getting a little deeper into the bus API. So when we're implementing the bus API, we're actually implementing this last blue box on the right here. That's what we're doing. It's this function here. Oh, that was really loud. And this is what that implementation looks like. It's only about 50 lines of code, really. Again, I'm keeping it pretty abstract, but we just implement a function. This is for I2C that takes the emulator, the I2C messages and some base furniture and we just implement, hey, is it handled? Is it read? Is it a write? Is it something else? Should I explode? What should I do? So that's the bus API. Now, what about the backend API? Well, let's get into that. So we've talked about how you can have a test that calls a driver API, which then gets implemented by a driver, which then asks a bus controller to forward messages to the peripheral, in this case, an emulator. That's great, but a lot of the time, in fact, I'd say most of the time, especially with sensors, our peripherals are telling us about the world. We're not telling them about the world. If we have a temperature sensor, I don't tell the temperature sensor, it's boiling. It tells me it's boiling, though that'd be a lot. So how do we handle this? Well, we provide an API to allow the test to reach into the backend state of the emulator. That's why we call it a backend API.  And then the emulator implements that API and implements, this is what my internal state and registers are gonna look like. And then when the driver next kind of pulls and talks to the emulator, that's going to get percolated to the driver. So this is an example of a test actually using the emulator API. So this is the BC12 test. Here up on top, it's including the BC12 header, emulator for that emulation API. And it's just saying, hey, emulator, pretend you have a BC12 type SDP charger connected. Cool, so it does that. And in the middle here, in which I've kind of hidden out, is that the driver is regularly talking to this chip and it's checking, hey, do you have a charger connected? And if it does, the driver goes ahead and updates some state here with a callback. And that's what this test is verifying. Did that driver actually work? Did it do what we were expecting it to do? What's neat here is nowhere here do you see anything that's specific to a particular kind of BC12 chip. This is at the API level. So we're not writing specialized tests for a particular driver in a particular peripheral. So this is just an example of what that emulator BC12 header would look like. Just an API struct and the header that you end up, or the function prototype that you end up calling from your test. And this is what an implementation would look like in an emulator. Not a lot of lines of code. It just takes the emulator and the partner type, set some internal states, which will then get reflected back to the driver when the driver talks to it. Asking the right questions, that is. All right, so we've talked about the bus API and we've talked about the backend API. Let's see how both of these get combined together. So this graph here shows kind of both at work. It says test there, but think multiple test executions, but the same test source code. What's happening here is a test talks to a driver API and based off the device tree, it's either talking to a MediaTek driver and thus a MediaTek driver or MediaTek emulator for the same chip, because the driver and the emulator are both specific to the peripheral and thus they're both coupled. Well, if my device tree says, oh no, I have a diodes driver, I have a diodes device, well then it just runs the same exact test but includes that driver and then that associated emulator. And then if it needs to test, oh, have this charger connected, both can just share the same API. So you can have multiple test executions with different device tree overlays, but not change your test source code at all. So that's how we create an emulator. Talked about the backend API and the bus API. So all right, we've created our emulator, we've done the work. How do we actually use one in a test? Well, first thing we gotta do is enable emulation, right? But we also need to include two device tree nodes. We include the bus emulator node, which is the part that intercepts driver messages to the peripheral and then goes and finds the right peripheral emulator to forward them to. And we need a device driver node for, well to include the device driver code, right? That's actually no different than a regular hardware test. So when you're adding this node here, the bus emulator node, you're adding this blue box here. That's what you're including. And all the work's been done for you, you just need to include this device tree node. So you don't need to write any C source or anything like that, we're done. You get this for free if you're using a native POSExport, that's why you'll see those tests very frequently used with emulators. But you can just as easily include this in a QMU. And I have some examples later that'll show links to different examples that actually use a QMU with a peripheral emulator like this. There's some stubbed values here. They're not actually used right now, but they're just there so this effort thinks this is an I2C controller. By the way, we have also SPI and ESPI as well. I just chose I2C because I think we're all pretty familiar with it. And then we need the device driver node. Just the same as a hardware test, so we just add that in. And when we add that in, we get the device driver, but we also get the emulator. Because the emulator is coupled to the device driver, when we include one, as long as the emulation is enabled, we get the emulator as well. So here's some examples I've got that I was alluding to earlier. I don't know if the PDF slides actually have working links now that I think about it. They were links, but I don't know if that survived the whole PDF process. But those are some search terms to get you started. Feel free to ping me too on Discord if you have any questions. I'm Aaron Massey. I think I'm Aaron Massey, underscore Google actually. Well, before I get into Q and A, I wanna talk also a little bit about some improvements that I'd like to have in the emulation subsystem with Zephyr. So this whole function that we need to implement to sort of parse the bus messages and respond to them, there's a bit of boilerplate there. It's not a lot, but it can still cause duplication of mistakes. So I created a GitHub issue on this. I would love some comments and some help. I think the initial idea is sort of emulators that are implemented device that uses SMBus. Maybe they can all use some shared abstract logic for parsing that. But just an idea, please comment. I could use the help, we could all use the help. Thank you. Another thing here is we have a lot of drivers, but we don't have a lot of emulators. Now I picked three drivers here. I'm not picking on any of you or anyone's driver, by the way. I just picked these pretty arbitrarily. I just wanted them to have different APIs and be sort of updated in the past year or so. It'd be great if we had emulators for them. And especially if someone who was like more of an expert on their associated peripherals would be really great if you could contribute an emulator for them so we can get CI and regression testing for these drivers. I think eventually, and Keith actually alluded this to this earlier in the keynote, it'd be great if we started asking contributors of new drivers to just contribute an emulator and emulate tests with that new driver. They're already pretty familiar with the peripheral and with the driver, so it's sort of the best time for them to create that emulator. It's when they know most about the peripheral. And then we get CI and code coverage just from the get go. All right, so one final thing, I'm willing to talk about how using emulators has been a success at Google. So as Keith mentioned earlier in his keynote, we started at 30% code coverage and then we went to 90% code coverage. We started with a pretty large pre-existing code base and we took drivers that already existed and we implemented emulators for based off our data sheets and the existing driver code. And then what we did, what we got from this, when we moved from 30% to 90% code coverage is we caught various bugs in drivers. We also detected dead code that was not link time optimized out, meaning through emulated testing, we were able to make our binary smaller. We also blocked regressions from changes. We blocked them before they made out the hardware testing. And we even went through the process of prototyping new drivers when we didn't have the hardware available. So here's that big, pretty graphic that I spent some time on. Does anyone have any questions? Yes. Ah, Marty. Oh, hey Marty. Nice to meet you. Nice to meet you. First of all, oh, yeah. First of all, thank you for this wonderful talk. I would have really appreciated knowing all this information when I was reviewing some of this stuff for the first time. So thank you. I really appreciate your work. I guess I have a question, and this may be something that's just sort of out of scope of this level of testing, but it seems like at the level of the fake bus driver, you're sort of intercepting at the level of like the struct ITC message. Yes. And so that seems to make it impossible to include test cases for handling situations where there's corruption on the bus that are not representable as a valid struct ITC message. Do you have any notion of whether that is something that could be tested via the backend API, or do you have any thoughts on how to address situations like that? And you know, I didn't get to writing this RFC in time, but we actually have started talking about this about, hey, let's send some corrupted messages, let's intercept the individual messages, and then send them over to the emulator. We've actually thought about maybe we can have some kind of stack mechanism where you can have multiple sort of interceptions of this bus messaging, and you can kind of change it on each side during the test. That's a fantastic question, and that's something that we could use to improve it. So all right, I'll get cracking on that RFC, or maybe I could use some help. Maybe you can help me out with that. Thanks. Awesome. Yeah, once again, thanks for the brilliant talk. I've also got one question. Any ideas, plans already on when peripherals would also have interrupts like an IMU that would trigger on the fall detection stuff like that? Well, you can include a lot of that in the emulator code itself. So in our emulators in Google, we actually have that implemented and then quite a few of my emulators were like, oh, this value changed.  Go ahead and set this interrupt as high. So you can totally do that. It's the emulators. They get compiled as part of the Zephyr binary, and so you can rely on device tree or what have you. Did I answer your question? Okay, okay, absolutely. Hi, my name is Hugo from BlueNetics. My question is, I saw some statements, Zephyr statements, so is this part of a Google fork or is this already? This is all in Zephyr. Oh, sorry, I should have let you finish. Is this publicly available in the Zephyr tree done? It's just we need to activate or use it. Absolutely, it's in the Zephyr tree now. Actually, all of Google's emulators are also open source. They're just in our Chromium OS tree as well, but it's obviously easier to use if it's in Zephyr. So, but absolutely, yeah. To make this easier for, as you mentioned, requests as we have new drivers and so forth, have you thought about sort of a way of from the tests, having a trace that, so we, I could see a version of this where you just have it like be a data, like a description file of register read address. There's some way of describing this for that peripheral as opposed to having to write all this code necessarily to kind of have a state machine and sort of emulate the whole device or do whatever that may be versus just saying, hey, okay, I've got my test. I ran my hardware. Okay, I got this version where I enable this K config in Zephyr that dumps out a trace log. I can take that trace log, put it in a file, and that's my emulator. Okay, so you're talking about, can we just take the logs of register reads and writes and then just say, okay, as long as we have these reads and writes, can we just say this is our emulated test? Yeah, exactly, right? I mean, you may want to put some additional things like you have, like for being able to signal right from the test if it's kind of saying, hey, I want to emulate or act as if, that I'm the low battery condition or some of that stuff, but some way where we could take a trace with some other information, and then that would just generate my emulated driver for me for a lot of cases, because then that makes it really easy for someone to do this as a low hanging fruit. There's still the option if someone wants to write a full driver, emulated driver, they can do that, but the quandary I have with us having to do this and expect this from people is, if I'm a silicon vendor, I'm doing this, I've probably got a bunch of other things I'm already doing for simulating my system. Now you're asking me to do yet another different one and trying to explain the cost of that to somebody that's, hey, do this Zephyr specific thing and then, hey, you're doing a system C model and I'm doing this model, I'm doing that. It's like, how many of these do I have to do because everyone's got some different way they want it to look? It's just, I mean, I've got to convince management to say, hey, do this 16 different ways, right? So I think we have to figure out how to make this cheaper and easier if we want vendors to actually provide that as part of driver support. I understood. So when it comes to the engineering cost of actually creating the emulator, we don't have to make a fully fleshed out emulator at first. We can kind of more iteratively do it. Like I was adjusting the trace, right? Right, so what I was going to add into, so you mentioned the trace. That's an interesting idea, basically recording what the trace is and then using that to generate an emulator, right? Is that what you were saying? Yeah. Okay, okay. He said yes in case anyone, man. That's an interesting idea, especially for prototyping or just getting started quicker. I think it's the very least worth exploring into an RFC, what have you. At Google, we didn't end up doing that. We found it easy enough to just create an emulator and sort of patch it as needed, but I don't see any reason why you can't just take a trace and then try to have a tool that then generates some kind of emulator. Obviously, when you have generated code, make sure it's at least somewhat readable because you may wanna have special implementation in your emulator that is, I guess, maintainable. And so generate code and maintainable doesn't always work together, but that's interesting. Let's talk more offline, like that's very interesting. Hi, thanks for the interesting talk. How do you run the tests in the CI? You run them, like compile them for POSIX or run them in QEMO? You could do either. In fact, in those examples, you can test in native POSIX or in QEMO, just use Twister or just works. Thanks. Great talk, thank you. One thing I suppose is worth pointing out here is when you have, you're just showing one driver, but taking an upper level, say you're trying to, you have a system which has a battery, it has a charger, it has temperature sensors, it has other things as well. You can just wire all that up and run logic on top of that that uses these things as emulators. So you can do higher level testing of your code basically without doing any additional work. Right, so Simon makes a great point. I didn't really talk about it too much in this presentation, but you can make the emulators for the peripherals and you can use the drivers of those emulators, but you can combine and use multiple drivers and emulators and create just a layer of testing above it. So if you wanna test, let's say your USB power delivery policy, right? We actually do that in Chrome OS in our open source code and there's like several layers. So, you know, good question, thanks. Hey, I have actually two questions. The first one is currently this relies on using the transport API in Zephyr for the different drivers. Is there anything that you've done to explore memory mapped IO? So like using some extra abstraction layer through Zephyr to do register reads and writes through memory to be able to interject that? I haven't looked into that myself. I don't know, that sounds interesting. I'm not sure. I haven't really looked into doing that. I don't think it's been necessary so far, but can we talk about this more offline? That's very interesting. What was your second question? Sorry. No worries, we all answered the first one, so that was perfect. Oh, I'm sorry. And the second one is can you actually use the subsystem also to do a high level application development before you have actually your hardware? So not just for testing, but also just running your application code already in your target devices. Is that something you can use for that? Absolutely. You absolutely can. You can just start up a native politics board and run it. In fact, we've also done that in Chromium OS. And yeah, so you can totally do that. You can just start up a board and you can have emulators as part of the binaries. And we'll just talk to the emulators instead of real hardware. Amazing. Thanks for the introduction to the emulation. Actually, a very simple question. Which bus types are currently supported? We have I2C and we have spy and e-spy. We can add more. So here's an example of the e-spy node that you would add and the spy node that you would add. These are all native politics for free and you can also just include them in a QMU. And we've got a question from the virtual audience. Would this also work on fully embedded devices such as the Nordic RP2040 or Raspberry Pi? I don't see any reason why it wouldn't. This is just some code that gets compiled into the binary and you can just run it. Maybe I misunderstood the question. Yeah, I'm guessing maybe they're asking could you do emulation of like an entire board? Oh, well you can. So this is specific about emulating the various peripherals attached via some kind of bus. But because you can run it under QMU, you can kind of do both and combine it with other tools. So, so you guys have two minutes left. Yeah, so let me get here. Hi, thanks. You mentioned about three different specific kind of peripherals that you were looking for implementations of. But it might be worth talking about other generic versions of like so you had an ADC, you had a GPIO, and you had a sensor. But are there not generic ADC emulators and generic GPIO and maybe even generic sensor emulators that you could just say give this sensor value with this type or give this ADC value? So there is an ADC kind of, I guess, vague. It doesn't actually use the emulation subsystem where it's doing this whole bus forwarding thing. There is something out there is the ADC, I think underscore emul dot C. And that might be, you might want to take a look at that. So that's a little different because it's not actually doing this whole, you know, parsing of bus messages, right? So that's a little, the parsing of bus messages is a little bit more involved than that. As for in terms of more generic emulators, I mean, that's part of kind of what I'm asking for in up here is maybe we can get some kind of at least generic emulator code to kind of improve the emulator development and we can get more emulators quicker. You know, I've been working on the fuel gauge API and just having a smart battery emulators, pretty nice. But I've been considering changing that into sort of a library so that folks can use the emulation for very specific chips that use, that implement the smart battery spec, but maybe they have some quirks or something like that. But we want to reduce boilerplate in that emulation. You know, that was just an aside. Good question, thank you. All right, yeah, we're out of time here, but thank you very much, Aaron. Thank you, folks.