Hi everyone, Joe for jazby's casebreaks.com, coming at you with 2022, Panini Donnerous Football, six box, third of a case, random team break number two, all teams in, all cards ship. Big thanks to this group right here for making this happen, and there are the NFL teams right there. Let's roll it, let's randomize names and teams, two and a six, eight times each. One, two, three, four, five, six, seven, and eight. Got Donald down to Nicholas, two and a six, eight times for the teams, one, two, three, four, five, six, seven, and eight. And final time, Buffalo down to the Seahawks after eight.  Let's order these alphabetically by team, and we're going to pause the video. When we come back, we're going to see if there's any trades, and then we'll have the break stick around, BRB. All right, welcome back everyone. Well, this is a little bit of trade chatter, but in the end, no deals were done here on a Wednesday. Random team two, the second third of the case, so Colton, you ended up with Last Bottom Mojo and Ravens. 70% of the time, Last Bottom Mojo hits 100% of time, allegedly, so we've got six boxes on the left and right side. We're going to roll the die and see what we're going to do. One, two, three, for the left side, four, five, six, for the right side. Two, left side right here, and this side we will save for next time, which will be random team break number three, which is on the site right now. Still tons of spots left on that one. 28 spots left. All right, good luck everybody. We're looking for an autograph, one autograph, one memorabilia card per box, on average. Good luck everybody. Yeah, they are James. This is the, I think this is the full strength Lakers squad right here. Kind of a musk wind game. The Pelicans are ahead of them in the standing, so they got to beat them. Still a lot of time left. 750 left, but they're up 101 to 84. This is what the Lakers need to do. Play some defense, be a fish on offense. They got some willing rebounders, some outside shooters. Kind of bounces out the team really well, so one thing they don't have though, and then a shooter with the three. One thing they don't have, unfortunately, is time. Not too many games left. There's already already passed the all star breaks coming out. This is their last game before the all star break. And then what, 24 games left, 23 games left in the season? They're not too many games back of a play-in, or even a non-play-in playoff spot, but there's all too many teams to leapfrog, so they got to work on that. No, sorry, Nick. Cardinal's not a not a good draw, but hey, if we get like a Kyler Murray one-of-one or something like that, the Andre Hopkins one-of-one or something like that, you know, then that's kind of what we're hoping for.  So maybe a Kyler Murray downtown, something to look at. Jolani Woods to 500. And a Justin Jefferson. 67 out of 99. Vikings. Scott with the Vikings. Highlights autograph. Jefferson Snacks scoring grab in week 15. Well, there are Super Bowl downtowns, but in the first third of the case, we did see a Micah Parsons Cowboys downtown. Could be other non-Super Bowl downtowns in these next boxes. I mean, yeah, this is going to be like a 40-minute break, so if you're not feeling it, you know, obviously you don't have to watch. You can watch the recap at the end of this video that I'll do after I upload it. Passing the torch, Marshall Fock to Jonathan Taylor, 129 out of 199. That'll be for Derek and the blue horseshoes. Yeah, James, I agree. They never should have traded for a Westbrook. What was that original deal? It was like Miles Turner and Buddy Heald. I feel like that would have been a lot more productive. Logan Thomas, three out of 25. Die cut for Jared and the Commanders. There's Dominators, Pat from Homes. It's pretty good. Box one down. Oh yeah, it gave. That's a good draw. I got the Niners. Cross your finger for some Brock Rock Lobsters. Whoa. This is a family show, Gilo. This message is held for review. Even YouTube knows. All right, next to box. Good luck. Strait London.  And Kenny Pickett, rated rookie card. Steelers.  Travis Kelsey. No name variation for Cleo Mack. All card ship, of course. Boston Knox, 51 out of 88 for the Bills. Donald. Trayvon Walker to 49. Rated rookie autograph. That's for Michael Gallucci and the Jags. Are the pieces falling into place for the Jacksonville Jaguars, ladies and gentlemen? It kind of feels like it, right? They got some good coaching in place. They got an edge rusher. They got some pass catchers. They got a playmaker and Travis Etienne. They get another good good draft going. Christian Kirk turned out to have a nice season. Here's Canton Kings. Eric Dickerson, 65 out of 139. Show up on local sports talk radio every once in a while. Goes to Andrew and the Rams. Tried to trade, couldn't trade mojo. Out of Clinton Portis, 177 out of 399 for Jared and the Commanders. Derek Singly Jr. Rated rookie to 150. Alexander with the Texans. Right, next box. All right, another box. Marquise Brown. There's Brian Robson Jr., tough guy Brian Robson Jr. Jared with the Commanders. John Mechie, rookie phenom. Jersey in autograph 56 out of 99. For the Texans, that's going to go to Alexander. Got a Kenny Pickett.  Got an Anthony Barr, 49 out of 50. Scott with the Vikings. That was Aaron Rodgers. He's going on his his retreat soon. He's going to figure things out. See what he wants to do in life. Speaking of Aaron Rodgers, there he is. All pro kings. What are you going to do, Aaron? 274 out of 299. What do you think happens with Aaron Rodgers? I know this is probably over covered on the sports media landscape, but I don't feel like we haven't really delved into it. Does he get traded? Do they move on? Does he retire? Kenneth Gainwell, 009 out of 264. All right, we are halfway through this third of a case break. Three more boxes to go. He needs to retire, says Gilo. I feel like after all this like hubbub, I feel like he's just going to go and play for the Packers for one more season and then retire. You know, or say that it's his like, his retirement tour or something like that. Yeah, Nick, keep those fingers crossed. Let's see if we can get that downtown. Grant, Joe, I'm missing something. Series one case breaks. Aaron Rodgers follows follows far. Does the Vikings for two years? Then the Jets for a season. I don't understand your question, Grant. Baseball series. Don't see them on the website because we sold them out. Sold out. And if you take a look at our break schedule, you can see that's why we do the schedule. You can see what we've done, what we're doing currently, and what we're going to do. You'll see that the series one we all did towards the beginning of the day. Yeah, me too, too. Although I had some help today, so it wasn't wasn't that bad. There's Kirk Cousins speaking of Minnesota to 372. That's for Scott. Yay, Lakers win 12102. Nice, good job guys. It's a good feeling going to the All-Star break. Here's another Packer. It's time it's AJ Dillon. Jersey Kings, two-color patch. 44 out of 100. It's going to go to, it's another one for Michael Gallucci and Green Bay. Scott Hansen fans of the game will randomize that to one person at the break along with any other non-team cards. And Amanra St. Brown is your autograph. Nine out of 99. St. Brown walks it off for the Lions. Remember that? That's Chad Tucker with D Shroy. Yeah, J-Lo doesn't want to see Trey Lant. He got randomized to Niners. He wants to see the Brock Party. The Brock Lobster. School of Brock. For those about to Brock, we salute you. Yeah, that is a pretty random one, right? Scott Hansen does red zone, right? Do we break full cases here? Yes, most of the time it's full cases. Scott Hansen. Not to be confused with Chris Hansen who catches predators. Scott Hansen catches highlights, right? Yeah, NFL Red Zone channel. I feel like he gets a lot of heat.  This game was so close. Why would you tell us to go here when the games aren't even on? It's like just not on in their region. Do we ever do upper deck product? We do. The cup, yeah, cup's allegedly supposed to come out. 2020, 2021 cup hockey is allegedly supposed to come out so. There's Mack Jones, Jersey Kings. 45 out of 100. I think so too. Gilo, I was just hanging out with my buddy who was a producer for the NFL Network and I was like, do you think maybe do a little red zone in the fall? There's DeForest Buckner Orange to 99. Yeah, it would be pretty chaotic. Yeah, Nick, and this is not a bad job either. It's probably less stressful than red zone. Gilo Gabe, didn't you call Skymore getting a touchdown like a week before the Super Bowl? I feel like Gilo made that call. Yeah, it's a whole thing, Rex. There's contracts to be signed. There's appearance fees.   It was a good prop though.  There's a Dulcich to 500 and here is a Tyrian Davis Price autograph. 61 out of 199. Not the auto that Gilo is looking for but but still Niners on the board. Thanks for getting in, Gabe. Chuck Norris. There he is, fan of the game. Dallas Cowboys. You think Chuck Norris really is a fan of the Dallas Cowboys or are they just saying that because he was in Walker, Texas Ranger? Alright, final box coming up. Yeah, we should fact check these fans of the game. Are they really fans? What was it? What's some of the other fans of the game? I think Mark Wahlberg, Patriotism. I guess that's pretty obvious. I mean, I would imagine that that there has to be some approval process where they just can't put random celebrities in here without some kind of agreement. So they're just like, hey, Chuck, alright, more like Chuck Norris' manager. Is it all right if we put Chuck Norris in here and say he's a fan of the game for the Cowboys and they're like, sure, how much, how much are you gonna pay us? X dollars. They're like, okay, that's that's an agreeable amount. Let's do it. Yeah, Nick, you think it's just Walker, Texas Ranger and then there's like, listen, we'll just pay you? It's like, well, Mr. Norris is actually an old Houston Oilers fan. Mr. Norris actually likes the New York football giants, but if you're going to pay us, you know, however much they pay you for this, we'll take it. It's on Haskins to 100, Titans, Tristan, and reach for the Sky. Three out of 99. Skymore. There's a snake in my boob. Oh, that's a different character. Skymore, Chiefs, that's gonna go to Jace, caught a touchdown in the Super Bowl. Is a Super Bowl champion, Skymore. Right, yeah, Chuck Norris probably a Texans fan, but Walker, Texas Rangers on your TV now, Nick, say. I feel like that was a fun show. I don't watch too much TV, but whenever it's on in the background, I'm like doing chores or whatever. I'm just like, yeah, that's pretty entertaining. Chuck Norris just solving crimes and kicking butt. Sony, Michelle, kind of a slightly older Chuck Norris too. 64 out of 75.  Someone's poisoned the watering hole. It's a good movie. Toy story is great. Malik Willis going to Tristan and the Titans. We got T-Law, Jersey Kings to 399. It's for Jackson to let it be for Michael. Remember Renegades. Yeah, I feel like that was on the USA Network a lot. Who was the, who was the main dude in Renegade? He had a, he had a long dark hair, kind of stringy. There's Logan Thomas to 199. Jared with that one. He had a motorcycle who was just, we would ride like a big chopper with the handlebars up to here and had a shotgun. Right? Is that, am I thinking of the right show? See, there's Mark Wahlberg. I guess he is a fan of the game. I may have seen probably a quarter of a dozen episodes of that show. A dozen different episodes of that show. Lorenzo Lamos was his name. What did, what did he do? Was he just, was he sort of a drifter that went from city to city and solved crimes? Did he solve crimes? Because he wasn't committing crimes. What was the premise of that show? The world may never know. I'm Joe for jaspyscasebreaks.com. That was your six box random team two break. Let's see where Scott Hansen goes. Let's go back to the list right here. Got Nick, down to Jared. New dice, new list. Name on top gets the fans of the game card after 11 times five and a six. Was he a bounty hunter? That makes sense. An 11th in Valentine. He's still got a shot at that fan to the game card, Nick Anderson. No, you did, you're in spot 10. Sorry man. Appreciate you trying though. A little consolation prize for Nicholas. I don't think I called your number very often, Nicholas, in this break. But poor Nick's in this one. But I appreciate everybody trying. Scott Hansen going your way, Losis. And I'm Joe for jaspyscasebreaks.com. Final third of the case in the store. Check it out. I'm Joe and I'll see you next time for the next one. Bye-bye.


Did you know that you can run other programs from within R in RStudio? Pretty cool, huh? Well, in today's episode, we'll do just that when we run the lefts of function from the mother software package as part of a script that will write to generate those classic lefts of plots. And then, well, maybe we'll discuss why those plots aren't such a great idea. Hey folks, I'm Pat Schloss and this is Code Club. If you've been following along in recent episodes, you know that I've been looking at a variety of approaches of representing relative abundance data for microbiome data for three different disease status groups from a study that my lab published a number of years ago. Those three disease status groups are healthy people, people with diarrhea, and people with diarrhea who also have C. difficile. We wanted to know are there biomarkers, are there bacterial taxa associated with those three different disease status groups so we could perhaps learn a little bit more about the disease? What we've done in recent episodes is that for each OTU or each taxonomic level, we did a Krosko-Wallis test. We then corrected P values for the multiple comparisons. And then for those comparisons that were significantly different across the three disease status groups, we then did pairwise Wilcoxon tests to figure out which of the different taxa were differentially represented across the three disease status groups. And then in the last episode, we went ahead and put those beautiful stars and bars on all of our data to indicate which comparisons were significant. Doing that requires a fair amount of sophistication in your use of R, right? You needed to know how to use Nest and MAP and Krosko-Wallis and Wilcox, pairwise Wilcox, all these different things, right? Well, one of the things that's been really handy in the microbiome literature is people developing statistical tests that kind of perform generic tests like a test across different disease status groups, say, and doing it in a packaged way. So you put your data in, you get results out and then you're good to go. These pre-packaged tools also often output visuals for the user that many times users will take and then use in their own papers without further modification. Hopefully if you're following along and you're interested in a video like this one, you realize that that's just not gonna cut it. So this figure was taken from the original paper that described the lefts of package. And we will see figures like these populate the literature now. On the left is a cladogram that shows each kind of ring of this radial phylogenetic tree is a different taxonomic level. I'm not such a big fan of this. We'll move on and talk about that another day if people are interested. If I look at this, you know what this looks like? It looks like a pie chart. The right side, I see this type of plot showing up quite frequently. And so what are some things to like or not like about this figure? First of all, on the x-axis is the LDA score which is in a log 10 scaling. So tell me, what is an LDA score? I mean, really, come on, tell me, what is an LDA score? Well, if you can't tell me what an LDA score is then please don't publish this figure because if you don't know what it is then why do you expect your audience to know what it is? This LDA score is a measure of the effect size. It is a measure of how separated are the two different groups based on this one taxonomic grouping. And so if you have a value more to the right then that is a more associated in this case with Tibet rag knockout or Tibet, I don't know what these are. These are mass genotypes. And if it's more to the left then it's a rag too knockout, right? And so we can then see that these biomarkers are skewed depending on the type of mice that they're coming from. The way Lefse works is to do a Wilcoxon test on each taxonomic grouping. It identifies those OTS or features that have a P value less than a threshold, say 0.05. And then instead of doing a correction for multiple comparisons like we have used in the past like Benjamini Hatchberg or perhaps you've heard of Bonferroni, what it then does is apply this LDA, the Littier Discriminate Analysis to then rank the features by their effect size. And so those features that have a larger effect size are then represented if the effect size is, say, bigger than two or three and those that are smaller than are kind of ignored. Again, this figure is showing a pair-wise comparison between two genotypes. In our case, we have three different groups. And so that would be three pairs of comparisons. As we've seen in previous episodes we've been working on developing this figure where we can show the actual relative abundances of the three different disease status groups. Their level of variation for each taxonomic grouping that we're interested in. Again, to represent the same information with these LEFSA plots, we would need three LEFSA plots. And I don't think the LEFSA plots are doing that much work, right? They're showing you the effect size but they don't actually show you like the actual relative abundances. So in this episode, I'm gonna show you how we can run the LEFSA function from the mother software package within R in RStudio. We'll create the files that we need. We'll call mother to run it and then we'll read in the output files and then generate a figure just like this. Let's go ahead and head over to RStudio now and we'll get going. If you'd like to follow along today's episode with your own computer, I strongly encourage that you do that to really help learn the material all the better. Down below in the description here is a link to a blog post associated with today's video where you can get this starting code chunk. Also across the top here, I'll put a link for instructions on installing R, RStudio, the tidyverse and getting the data that we're gonna be working with. You see that we load in the libraries. We then read in some taxonomy information, get nice formatting and things like that. We then read in the metadata for this read Excel and then we read in a shared file. I'll go ahead and load all this into my current RStudio session and we'll get going. One of the things that we'll need before we can proceed into mother is two files to run the LEFSA function. So we'll need a shared file, which is a mother formatted file that we've been kind of working with in recent episodes as well as a design file that tells LEFSA what kind of sample each sample is, what group or disease status group each sample belongs to. Also, in some cases, in some projects, you might have different samples represented in your shared file than you might in your metadata file, right? So perhaps you sequence things but not everything got fully sequenced and so you might have more things in your metadata file than in your shared file. So we need to make sure that those two files are the same that contain the same information on the same samples. So we'll go ahead and look first at metadata, which again contains all sorts of metadata, the sample ID and a whole bunch of other stuff. So I'll do a select on sample ID and disease stat. And so now if I look at metadata, I'll have those two columns and only those two columns. Also for my shared file, that is going to be a massive file with a whole bunch of columns and rows. It's got 338 rows and 2,491 columns. There's 2,488 total OTSUs that we're gonna be looking at. What I'd like to do is make sure again that my metadata and my shared file contain information on the same samples. I can do that using an interjoined function where I could give it a shared file and metadata. And because the columns indicating the sample ID are different, I need to use the buy argument. So I will give buy the column for the shared file first, group equals and then it's sample ID from the metadata file. If I do call names on this as part of a pipeline and then look at tail, I see that the very last column is my disease stat. So I'm gonna go ahead and save this as shared design. So with this shared design file, I can then specify different disease status groups that I want to compare. So I'll get two different disease status groups and then from that, I'll create a shared file and a design file and then I can run that into mother. So let's do an example, right? So we could do shared design and pipe that to filter and I could say disease stat equals equals non-diarrheal control or disease stat equals equals diarrheal control, right? Maybe I'll put a line break here just to keep things on separate lines and make things so they don't run off the edge of the screen. And I now see that I have how many samples. I've got 244 samples. And so this then is a shared design file for my non-diarrheal diarrheal control comparison. And so maybe I'll call this NDC DC. So with this, I could then create a design file and a shared file for this specific comparison. I can do NDC DC and I could then pipe that to a select and I will do minus disease stat and I can pipe that to write TSV. And I wanna put this in a directory that I'll call processed data. I have a directory called raw data. So I'm gonna put this in process data because these are not raw. I'm gonna keep my raw data, my process data separate and then I will output this as Schubert.ndcdc.shared. I run that and it complains because it cannot open the file, process data Schubert.ndcdc.shared. And so what I can write would be dir.create processed data and that then creates a directory that now pops up over here in my files tab. And so I can now run this there and I then put Schubert.ndcdc.shared into that file. I wanna do something similar to create the design file. So I'll do NDC DC and pipe that to select and here I'll get grouped as well as disease stat. And I will then pipe that out to write TSV. And again, this is going to be processed data Schubert.ndcdc.design. So again, looking in my process data directory I now have those two files and I'm ready to run this in mother. One problem is that I don't have mother. Firing up my browser. I'll go ahead to github.com forward slash mother forward slash mother. This brings you to the mother github repository on the right side down here. There's a link for releases. The latest release as of I'm filming this here at the beginning of June 2021 is version 1.45.3. That brings you to the latest release version. I'm running mine on a Mac. So I want this Mac OS X-1014.zip. If you are on Windows, then you want a mother.win.zip. And if you're on Linux, you want the mother one of these Linux ones. So I'll go ahead and download mother OS X-1014. So I've got my mother directory in my downloads and I can go ahead and move that into my code club directory here. And now looking in code club I see that I have the mother directory with everything I need to run mother. If you're in Mac land, they've got all sorts of security provisions going on now. To get mother to work, you need to right click on it and then hold the command key open. And that will then open up a dialogue that says, are you sure you want to open this? Yes, go ahead and open it. And then that fires up mother. And so I'm gonna quit and we'll be good to go now there. Coming back to our studio, there's two ways, at least two ways that we can run mother from within our studio. So first I can go up to tools and then do terminal, and I can say new terminal and that opens up a new terminal down here in the lower right corner. I could then do mother forward slash mother and that gets me my mother session from within our studio pretty slick, eh? So I can then at this prompt run lefse. So I could do lefse and I could do shared equals shubert.ndcdc.shared and design equals shubert.ndcdc.design and I can say input dir equals processed data. Close parentheses and we can run this and it runs lefse, right? And it then produces this output file, okay? Another way to run mother from the command prompt would be to quit out could then do mother forward slash mother and we can then run commands for mother from the command line. And so we do that by putting it in quotes. So we do a quote and then a pound sign and then I'm gonna paste in the actual lefse command and then a closing quote, right? So this will run lefse without me having to go into mother and run the lefse function directly. So again, this way of running it from the command line will be very helpful as we think about how to run lefse from within R. So I'm gonna go ahead and paste that lefse command here. Again, remember it was mother forward slash mother and then we did a quote, pound, lefse, all that and a closing quote at the end. So this is a long string and what I will do is create a variable that I'll call command and I need to then wrap all this, the mother function call in a single quote, okay? So I can't do double quotes because I've already used the double quotes and so it doesn't matter which you use first, the single quotes or the double quotes but we wanna load this command string. Now within R we can type system command and so system command will run that command from within R and so pretty slick. It's then running a mother from within R. Isn't that pretty cool? And then this will output this file which is Schubert ndcdc 0.03 lefse summary and so that's good. And so we can then take this file and read it in and make our nice plots, okay? But I wanna do this for ndc non-diarrheal control versus case and diarrheal control versus case. So what I'd like to do is turn this series of commands into a function so that I can give it the different disease statuses I wanna compare as well as a tag for what I wanna name the files. Getting going on this, I'm gonna create a function that I'll call run lefse and so we'll do run lefse equals function and I'll give it x, y and tag. X and y are going to be these values so non-diarrheal control and diarrheal control in this case, that was the example that we used. Of course, next time we run it, x and y will be whatever value we use when we call run lefse. I'll then call this x, y because I'm not very creative with names and then I'll need to use x, y here as well as here. For this tag, I can use glue to put in the tag but I need to write glue around all this so that what the glue function will do is insert the tag into my file name here. I can do the same thing down here with the design file by again doing glue and then replacing that ndc, dc with a pair of braces and tag in there and then I can do all this and here again, I'm gonna need to put in my glue but we'll get this going here and again, I can then wrap this in glue. Okay, so that will be good. I'm gonna go ahead and indent these lines over a bit and then we need to put in our closing curly brace and then I'm going to return the mother output file name, the lefse summary file and so if we look at our terminal, that was this and we'll go ahead and put that in quotes and I'll replace this ndc, dc with tag and curly brace and of course I now need to put glue around all this. I can then run this with run lefse where I do a non-diarrheal control, diarrheal control, gotta spell it right and then I'll do ndc underscore dc, run that. So again, we can then repeat this for our other three comparisons but for now I'm gonna call this ndc, dc and again, I can copy this down to non-diarrheal control ndc case where I'll then put in case here and then in my tag, I'll do case and then here I'll do dc underscore case and I can then do diarrheal control there and case and then dc case, right? Again, now we have these three variables ndc, dc ndc case, dc case that contain the file names of that lefse summary file that we generated in mother, right? Let's go ahead and make one of those classic if you will, lefse plots. So we'll go ahead and read underscore tsv ndc, dc and we get a sense of the layout. It's giving us a bunch of errors. They're not errors, they're warnings of parsing failures. So the problem is that like for row seven, it says five columns expected, three columns actual and then if we look at seven, what we see is that this is a case where there wasn't a significant value. Anyway, looking at this table, we see in the first column, the otu, the log max mean, the class as well as the LDA value as well as the p value. So again, if the p value is less than 0.05, then it calculates the LDA and then tells you which class has the higher mean or the higher median value. Again, let's think about this plot on the right side of the screen here. And what it is, it's a bar plot that's vertical. So the y-axis is the otu and the x-axis is the LDA score. Interestingly, the LDA score is positive for one case and negative for the other case. And so our LDA values are all positive. And so those that are associated with one of the two disease status groups will need to make negative and the other will have to keep positive. Let's go ahead and start here with the read TSV and DC. I'm gonna do a drop and a on LDA. So again, this will get rid of any row from our data frame that has an NA value in that LDA column. Then I want to modify my LDA column. So I'll do mutate LDA and I'll say, if else, and I'll say class equals equals and I'll make my non-diarrheal control go off to the negative side, right? So if the class is non-diarrheal control, then I want negative one times LDA. Otherwise, I'll keep the plain LDA value, right? And so now what I see is that I misspelled the real. And so let's run this again. Okay, now we get our non-diarrheal control samples have a negative LDA. Our diarrheal control are positive. So they're going off to the right and we can then again ship this off to GD plot where our AES, our X will be LDA, our Y will be OTU. And so we'll come back and substitute OTU in with the actual taxonomy name. But for now, let's go ahead and do geom call. And again, what we see is we've got our OTUs on the Y axis or LDA on the X axis. We don't have our color and we don't have things sorted by LDA. We also would like to eventually bring in the taxonomy information for those labels. Let's start with the color. And so we'll do fill equals class. Good, we've got things colored by class now. To get things in the right order, we'll go ahead and add a line to this mutate where I will do, I'll make OTU to be a factor. So I'll do FCT reorder on OTU and we'll reorder it by LDA. And sure enough, we now have things reordered. So the diarrheal controls are up at the top. The non-diarrheal controls are towards the bottom. That looks good. I'll go ahead and join in the taxonomy data frame with inner join. And I'll take the stuff coming through the pipeline with taxonomy and then I need to do a buy because my columns have different names. So in the pipeline coming through, it's all caps OTU and in the taxonomy, it's lowercase OTU. Add that to the pipeline. Again, that's a strong argument for keeping your columns the same capitalization. So now instead of OTU, what I actually want to use here is taxon because the taxon, the stylized name is what I actually want on that y-axis. So taxon and then I want to do theme classic to get rid of all the grid lines and all that stuff. And then theme, I'll do access.text.y equals element markdown. So element markdown will take those stars in my taxonomy names and make them italicized. And I get that from the ggtext library. ggtext is installed with the rest of the tidyverse. And so there's like way too much stuff here, right? And so you can see maybe the font's too big or I think really ultimately there's way too many taxa on the y-axis. To get rid of some of those OTUs, what I'll do up here in my pipeline, I'll do a filter to keep those LDA values that are greater than four and get rid of some of those ones that have a smaller effect size. Great, so that kind of whittles down the OTUs to a smaller number of features that's a little bit more manageable and more on the scale of what we already saw with our plot where we're showing the relative abundances for the three disease status groups. And we've got a nice styling of our taxa. So let's go ahead and start with changing our labels. So I'll do labs, y equals null, and then x, the styling that they use in the default plots from Lefse is LDA score, and then log space 10, add a plus to the end of that. And then for my x-axis, I'll do scale x continuous. And I'm going to do limits equals C minus six to six. And I want breaks, I'm gonna do, I'll do seek minus six to six by equals two. And with that, it'll be minus six, minus four, minus two, zero, two, four, six. Go ahead and add a plus to that. So that's looking better, better styling. Let's go ahead and change the colors now. And so we'll do scale, fill, manual, and we'll do name equals null. And then our breaks will be non-diarrheal control and diarrheal controls. All right, and then our labels will do healthy and we'll do diarrhea. And I'll do C difficile negative. I know it's another star after difficile. And then to match the colors, we'll do values. So again, the values off to the left are red and those to the right are green. And so I will do non-diarrheal controls will be red and then dark green and we'll add a plus to the end of that. So that matches pretty well. The colors, the colors are bad, right? Should not be using red and green because a large fraction of people are unable to differentiate between these two colors. So let's go ahead and maybe change these back to our stylings that we've been using throughout these episodes. I also need to go ahead and add element markdown for the legend text. So again, instead of red and green, our healthy had been gray and our diarrhea seed of negative had been blue. And I also then need to do legend dot text is element markdown. And so that's pretty good. I think I'll go ahead and put a line break in there in the title and maybe I'll make the size of the plot a little bit shorter. So those rectangles, the bars aren't so tall. And so maybe my height, I'll make four. I'll go ahead and put in a line break in here between diarrhea and seed of negative. So I think this looks pretty good. Definitely much better than the color scheme that you get for the defaults. Again, I'm not totally thrilled with this visual. I would need to repeat this two more times to generate the comparison between the diarrhea seed of negative and the diarrhea seed of positive as well as between the healthy and seed of positive. I already have that in that other plot we generated where we saw the relative abundances as well as the variation in the relative abundances. I feel like as we've been stepping through these episodes we've identified a couple of limitations with different visuals. So you'll recall that like a stack bar chart and pie chart shows you no variation in the data. I mean, it's also got other problems, no variation in the data and no sense of what is significantly different from other things. This shows you significance, right? In a way like the LDA scores kind of like a pedi. It's the effect size, but so it's showing you like what's different, but it's not showing you the average relative abundance. It's not showing or the median relative abundance. It's not showing you the variation, right? Again, if we come back to this version of the visual we see the median, we see the amount of variation. We can see what's significant or not. Perhaps it's a little bit more challenging to interpret the effect size because again, we've got this log scale on the x-axis. But I can kind of, I can at least compare that like for Bactroides, that's like a 10 fold variation relative abundance. Again, that's O2U2, where again, if I come back here to O2U2, that's Bactroides here, I've got a negative four point something LDA score for healthy. It's not really intuitive what that means. So if I really wanted to use LEFSA for some reason, I think what I would do would be to run all three pairs of comparisons through LEFSA and MOTHER. I would then get the LDA scores to figure out which O2Us are associated with which treatment groups, those pairwise comparisons. And then perhaps come back to this plot and perhaps instead of these stars being generated from the pairwise Wilcoxon test, I would then put stars on here and I would say, the stars indicate that the pairwise Wilcoxon was significant and there was an LDA score greater than whatever threshold you use. Here we used four, but perhaps you could go down to two and you could use that to indicate your star. At the same time, I'm not so convinced that I probably would use LEFSA. I don't think that I'm getting that much more information.  Say I had people that went through two different treatments and I had people before and after each treatment and I wanted to know which treatment changed the people's microbiota the most, right? Well, LEFSA would not help me do that because I would not be able to incorporate the paired nature of the data. I could do that type of a paired analysis by again rolling my own statistics through something like a Wilcoxon test, a paired Wilcoxon test, like we saw in the previous episodes using MAP. So again, I'm not totally convinced that these types of LEFSA plots really convey that much information. And I don't know that it's worth having multiple LEFSA plots like this. When again, it's just far more information that you could show in those other plots where we're showing the relative abundance and the variation in the data. Agree with me? Disagree with me? Let me know down below in the notes, in the comments, I'd love to hear what you think. I feel like because the LEFSA R package and I think it's available through CHIME, outputting these types of visuals directly that people are like, ah, there's a file that has a figure. I will use that directly in my paper that that kind of removes a lot of resistance that people face in analyzing their data. So hopefully going through this episode today, you'll see how we can make the plot. If you have a PI that wants you to use a LEFSA plot, how could we make them better by picking better colors perhaps? You can better understand kind of the strengths and weaknesses of these LEFSA plots, but also the other thing that I wanted to drive home in this episode is how cool it is that you can run other programs from within R in RStudio and doing that with mother. You know, I certainly think there is a reason to run everything through a bash script like we've seen in older episodes of Code Club and then bringing that within R, but you know, it's pretty powerful to be able to run commands from the command line from within R using that system function call. Give that a shot. Let me know what you find out. Let me know if you find that useful for your work. Hopefully you see the value in doing that. Keep practicing. Tell your friends about what we're doing here and we'll see you next time for another episode of Code Club.


Welcome to another edition of RCE again. This is Brock Palin You can find all the old shows and find a link to subscribe be their favorite podcatcher or iTunes at www.rce-cast.com I also have here Jeff Squires one of the authors of the newly released MPI open MPI 1.6 So yeah, we just released open MPI 1.6. Very excited about that finally took that one to the Stable series and we're encouraging everybody to upgrade so go look at openmpi.org in the announcements and free code and all those kinds of things but the other thing I'm kind of excited about is Just got a book chapter published about the internals of open MPI in a book called the the architecture of open source applications volume 2 and It's actually all the proceeds go to charity. They go to amnesty international So not getting a dime for it. It was just kind of an open sourcey thing to do to talk about how your software works and Release that out into the world and so go Google that and you'll find it and it's good stuff There's a lot of other applications well-known open source apps in there like got Hatchee and Git and a Hadoop file system and things like that. It's a great Learning resource just to go see how other people other successful open source projects have architected their software. It's pretty cool stuff That is pretty neat. So We're actually doing something a little bit different. We're doing more like a panel discussion. So today we're going to be talking about HPC resource allocations, we're going to stick to like your compute nodes. I want to talk to three different Organizations about the different ways that they allocate out their compute cycles either be it condos or allocations or something in a Metal and so we have with us today. We have representing the University of Michigan We have Andy Caird who I should also point out is my boss We have Brian Gulfos from Ohio's super computing Center and then we have Preston Smith from Purdue. So Andy Why don't you go ahead and introduce yourself? Sure. Hi everybody. My name is Andy Caird as Brock mentioned I'm at the University of Michigan and I am the director of high-performance computing for the College of Engineering here at the University of Michigan and I've been doing the stuff with Brock and a few other folks About eight years now and we've recently in the past couple of years moved from a pretty explicit condo model to a Something closer to an allocation model, although it's not very pure in the sense of what I think Brian will talk about So that's kind of what we're doing and we're looking forward to hear what everybody else has to say Brian Yeah, my name is Brian Gulfos. I work at the Ohio super computer center where I manage our client services I've been doing this job here for About eight or ten months now I've been at the Center for almost seven years before that I was a defense contractor at right pad Air Force Base doing a human interface research for UAV control So it's a little bit different than the HPC stuff, but we do primarily allocations based scheduling and and we're Exploring some other opportunities, so I'm interested in seeing what everybody else is doing in here Which works for them? Okay, and lastly but not least Preston Yeah, I'm my name is Preston Smith I'm the manager of the research support team at Purdue University in the Rosen Center for Advanced Computing The Rosen Center is a unit of what we call ITAP information technology at Purdue at the Central IT Organization at Purdue and we we build and allocate our our HPC resources through a condo or community cluster model Okay, so how about you guys go ahead and give us a little bit Each mentioned a kind of allocation while you have but could you go into a little bit of depth of what that is and kind of description of your resources? Sure, I'll just go in order. I guess again So we have we have a couple of things here in Michigan, but what I'm going to talk about mostly is what we call flux FLUX The name was to indicate sort of the lines of flux to attract people to the system. I'm not sure If that was accurate in the end, but that's what we have And the allocation model was intended to take people who are very used to a condo model So they have this idea of I have this many boxes and they are mine And move them with that same kind of restriction, right? If you buy a computer from HP or Dell or IBM or whomever You get say 12 cores and that's all you get you cannot then Later get more in that same box So our idea was we would give them that constraint at sort of the top end But we'd let them pick arbitrary integers and we would make allocations with that one Limit of cores and then the other limit of course is time So we have to kind of this two-dimensional thing of cores and time And that's what they get charged for kind of whether they use it or not So in a sense, it's more like a hardware purchase in that way If you buy a computer from Dell and you don't ever use it. You don't get money back from Dell, of course So we wanted as huge to that as best we could at least at the beginning We still consider ourselves after two years to be at the beginning So our model pretty much tries to look like hardware that said from the operator standpoint We can oversubscribe it Which we do and as such then the price everybody pays comes down so hopefully we're more competitive in that way because of your subscription and because in our condo model over a Decade or so of operating that we noticed the utilization was fairly low 50% or so so we We thought boy we could oversubscribe by a factor of two and it would work out and if we would pay half as much In reality that doesn't actually work out, but we do oversubscribe by it by a bit 1.5 ish or so 1. It depends. We haven't really figured that out yet, but in any case it's more than a factor of one which is better than piece of hardware and Hopefully we're reflecting that in the cost to everybody who uses it without too much violence being done to their view of what a computer is for them That's that's kind of what we do here So just curious why doesn't an oversubscription factor of two work out? Is there bursty issues that you have to deal with or something else? It is it's the it's the bursty issues Pretty much because of course it if we I think if we had more Computers right so we've only ever reached that point once and we had 2,000 cores We've been expanding it. We're now at 6,000 cores and and I think we'll get to 8,000 pretty soon this summer maybe even 10 this summer And I think with more cores we'd have more jobs starting and ending in any given time and then we could increase the oversubscription and people would See less of a delay Again, we're trying to keep it feeling like your own hardware right so we kind of want it to be responsive you know Within at least Minutes or maybe small numbers of hours, but certainly not large numbers of hours or days for a job to start so But yeah, we tried it up to two and it didn't it didn't work out. Nobody was very happy with that So but we do think that sort of you know 1.7 or 1.8 is probably gonna work We haven't gotten back up there yet with the additional hardware. So right now everybody's pretty happy with it Okay, Brian, could you describe what you guys do? Yeah, we we have fairly Homogeneous setup here. We have a just actually just earlier this year. We brought on our newest cluster hopefully 106 teraflops, you know 832 cores or nodes almost a 10,000 cores and For us we basically we hand out what we call resource units, which are one resource unit is currently 10 CPU hours And we hand out, you know, we get 5,000 just kind of right off the bat for asking Hey, I want to use the system And if you need more than that, we actually require Grant proposals to be submitted and then those are peer reviewed and an allocation committee that is made up of From our user community. Basically, we'll we'll review those for scientific merit You know is the RU request reasonable Are you asking for a reasonable amount of resources to do the work that you want to do and then they kind of build those out And then we we you know scheduling isn't fairly straightforward You just use you burn your your time burn through your RU's so it's all it's all very straightforward in terms of The resource allocation You know, it becomes a little bit more difficult When we start dealing with with special cases Especially large users or issues like that Okay, that's good overview. Let's go with that for the moment Preston. How about you? So at Purdue we operate in a condo model We use community instead of condo, but it's it's essentially the same the same meaning of the of the of the word We every year usually about in springtime will put out a call to our faculty that we're building another cluster We'll work with a number of different vendors to get and get bids on the cluster We do this every year so the faculty can plan no matter no matter when their grants may be coming to or when somebody may be Starting with university they can plan that either their startup money or their grant funds There's always going to be someplace to invest in computing So we we centrally provide all the system administration software installation Consultant support and all that sort of stuff use central data center space central networking and storage So all of the faculty have to do is is pay money in for the for the cost of the node So they know if I buy 116 core nodes that I get that many cores with to choose from for my research group the The slightly odd part of that we're kind of a kind of a hybrid these days with Rather than physically buying nodes for the faculty They're they're buying what appears to be a service so they they couldn't walk into the machine room and look and look at Look at the racks and say that is my node right there But they know they at least have that that much capacity That they that they've invested in Okay, so let me get that straight. I I didn't quite catch all that so You're saying that the faculty play a Flat fee how do they determine how much each faculty members is it weighted or do they get the other? Is it the same fee for everybody or how does that work? I'm sorry I just kind of made it if it's a flat fee per node So it's essentially the cost I know So that this year we know the machine I'm going to refer to for most my questions the machine called Carter which we just brought into production last month It's it's nodes cost thirty three hundred dollars So faculty member could buy in with thirty three hundred dollars get one node and it can then you can then start using the cluster So the all they have to pay is that thirty three hundred dollars they get access to that cluster for its entire five-year life and And then the days go on from there They don't have to pay any of the the the hidden costs for the networking the power of the storage Okay, so they get they get access to one node for the life of that cluster at any given time Is that what you mean correct? So if I buy one node three three hundred dollars I get a 15 a 15 processor Q and I can use that however I need and then you know Going on upwards for the main node as somebody wants to buy in for So I have a question about what do you how do you see what do you do with unused capacity? What if I know never use it to just sit there in the data center? Well, it it would normally but we have we've come up with a way of letting letting all the other faculty that are members of the Cluster use everybody else's nodes when they're idle In early versions of the CUNY cluster program we used to use a preemptible queue That didn't turn out to be very popular. It's kind of nondeterministic if you know if you never really know when your job might get preempted So we came up with what we call a standby queue So I can run as well as large of a job as the cluster will will will let me start But I can only run it for four hours So the faculty know that if their node is being borrowed they'll get what they've paid for within within a couple of hours and That's and the people who are doing the borrowing know that they can count on four-hour chunks in which to get extra computation done So let's just free for for people then It's free for people who are who are partners in the cluster So the the the feet the feet to enter is at least one node So somebody who isn't a partner in the cluster. It doesn't have access Got it. Now do you do any kind of fair share scheduling with that to? Prevent abuse like one guy who buys one node and then continually submits 500 node jobs in the standby queue for the next three years Yeah, currently we do occasionally see some users that do that and I think that's just a virtue By virtue of us of our bad system currently we're using PBS pro on our clusters Which doesn't give us the ability to fair share within one single queue we can do it across the entire cluster But then not necessarily within the queue We are we are in the process of moving to Moab on our Carter cluster our newest one and it doesn't give us that capability so you mentioned there a number of Things that they're really only paying for the nodes and you're picking up a lot of the other costs I'm curious from the other panelists. What costs you have to cover and which costs are kind of offered as a public good? Brian, let's start with you on this one Yeah, well, I think we're probably a little bit different in the sense that we are explicitly funded by Ohio's Board of Regents. So we are a And we always have been you know, we're in our 25th year now a shared resource for the state So there is no no cost to enter for any of our users Basically, they just they come to us and ask for resources and we hand them out. So There's no, you know, there's there's no no sort of cost to pass on any of our users now Some of our users do volunteer to help pay for some licenses Or some storage if they're exceptionally large users, but you know, you could be a tiny user using a tiny amount of resources And you'll pay the same amount of zero dollars as our largest users Okay, Andy, what do you guys do in Michigan? So for we have the mandate in fact, we're I guess the total opposite of what Brian is to me in Ohio We have the mandate of recovering every cost. We have zero extra funding So we we have to cover People pay we have it we have a rate that we set and we have an internal university as everyone does I'm sure it's kind of fake money. You can have an account. I guess it's real at some point, but And you can give us your account number and we will charge that for what you asked for and We are obligated to recover all of the costs for the staff the data center the networking the compute nodes the software We cannot spend we did get some seed capital as it were so we could get going But we are not allowed to spend anything that we cannot then recoup So That's I would not recommend that model by the way to anyone listening I think it's a very difficult matter and it's unlike many other research things where You know when you show up in your office as a researcher you flip the lights on and they come on You don't you don't get a bill for that So it's very hard for people to come to grips with the fact that they have to pay for every single thing Unlike when you go to the grocery store where you know most human beings go to the grocery store And you do pay for everything you buy so it's a different academia in that way is weird But we're trying to I guess here at Michigan look like we are more like a grocery store Where when you walk out with a bag of chips and you know six pack of pop you you have to pay for both those things And that's the way it is on our cluster So just curious you even have to prorate things like power and HVAC and whatnot So if I ran a job on two nodes for two hours You've got some kind of formula for how much that cost and power and air conditioning and things too We yes, yeah, that's exactly right We don't buy the hour, but we do charge you by the minimum our minimum allocatable unit is one core for one month So if you say I would like one core for one month, we will charge you somewhere around $20 for that And then in that $20 is in fact the HVAC and power and that's right Yeah, so it's an interesting Discussion the idea of trying to recoup all your costs and and I will add from our perspective that Even being a state-funded resource is not exactly easy, especially in the current Economic climate, you know our budgets have been flat for for quite a while. And so It's a scrape, you know to get The resources that we need in play for our users and and I think you know That's one of the things that we're looking at is additional ways to get some Money into the center pay for staff time and and capital expenditures and and and things like that and and as a fact As a matter of fact, we we do sell some usage of our time to commercial entities And so we have a a rate card that you know to find a CPU our cost and and we can bill our users for that net presumably covers Hardware, you know capital expenditures operating costs of the hardware and staff time as well So Preston, how do you guys do the cost there at Purdue? So we're actually it sounds like we're kind of right right in the middle in between in between those ends of the spectrum For the faculty all that they're paying is the cost of the node as I mentioned earlier But we do we do fully cost everything else We do the in our in our total cost of ownership Calculation we do include the power of the the data center space the staff time so we know how much the true cost is to the institution and As I heard just a second ago for for external customers like from private industry We do use that as a rate to build them for access to this program But we don't we don't fully take that same amount of Cost from the faculty That's the same furnace that when I say fully costed we do get a lot of Benefits from being being in the you know Broad embrace of the University of Michigan. So I think that if we were to look at a commercial cost It would be hard for us to say this is the true total cost appreciation of buildings those kinds of things We are we don't we don't worry about too much Which we would have to do of course if we were going to do a true total cost of ownership for this stuff But so there are some areas where we have our hands, but the things that we buy it on a short time horizon of years Not decades. We do attribute all the costs somehow and then we charge them to people So Brian why don't you start covering the question of what do you do with the two different types of users? What do you do with the user who has a Consistent need they're running say a hundred cores worth of stuff non-stop versus a user who wants a thousand cores Say every other month That's actually an excellent question Week because we do have both types of those users obviously I What we will sometimes do is we'll set up reservations so Nodes will be held available for a user who will typically have a consistent amount of usage So they continue to get that kind of usage. Well, we you know We're in a situation right now where we are running almost that capacity almost all the time So we're trying to avoid setting up too many reservations on our new cluster You know, we're still running our oral cluster gland and we still have some reservations on that But we could we just really don't want to have a situation where users are Holding nodes not getting charged for their use and pulling them out of the pool for everybody else But we do occasionally set up reservations for some users This actually happens more often for commercial users where they have a Certain timeframe to get some work done And and of course, they're they're paying us whereas our other users are are not directly paying us They're paying us their tax money, I guess, but So so we do we do a little bit of that we do a little bit Q priority sometimes for some folks And Those I think but the two things that we do most often to kind of get away from the normal usage So Preston, how do you handle those two different types of use cases? Right we really don't differentiate in terms of how of how we operate for for either for either use case We just we just give them the ability to to acquire computing and if it's appropriate for them to To use it that way like if they run a lot of jobs and get a lot of value for it Then great if they don't get if they don't run the entire time Then then it just they just don't get it. We just don't get all the all the bang for their buck to speak But it's really up to them So how does that work in with you know researchers have to buy you know whole nodes? Do they have if they really have sporadic use are their machines basically just idle even though they paid for them? Yeah, they work out that that does work out that way But what the nice thing is that most the people that operate in a real bursty mode But they don't need computation very often they're oftentimes the the faculty that may be buying one or two nodes and just using it sometimes Which for for spending $3,000 spread out over five years? Is it isn't really a ton of money the people that are that are putting in large sums of money buying 50 to 100 nodes Though those are never the groups that let them sit idle all the time So for the people that have bursty have have bursty needs the the investment is pretty minimal And it seems to work out fairly well for them in practice Hey, Andy Yeah, so um the the model that we have which is sort of this two-dimensional time in cores model It meets some amount of bursty needs right if you're if you need a thousand cores for a month We can do that. I'll talk about how in a minute if you just use a baseline of 100 cores forever That's very easy for us. Of course. It's predictable If you need something less than that to use a thousand cores for a day, you would end up paying us for a month And that that would be unattractive to you probably It's still cheaper than buying a thousand cores from from a hardware vendor for a day So we have a bit of an advantage there But even so we're looking at other ways to accommodate the very very short very very large jobs Which we have not figured out by the way The our mission or our dream I guess is that if we have 10 people who all want to run big huge jobs over a short period of time We can multiplex them together and take advantage of the hardware without ending up with a ton of hardware sitting around idle Waiting for these big jobs to come I think we're headed in a pretty good direction on that it feels to me like like we might be able to actually do that But like I said, we're still in the early stages of this only a couple years in so we'll see how it goes as we As we add more users and the system gets larger But the general idea is that you you you can pay for whatever you want Up to the size of the system and then our issue is make sure the system is big enough and can sustain itself at that size To accommodate those kinds of requests That makes sense Yeah, yeah, it makes sense. Let me ask a This is not directly related, but it's kind of sort of pseudo related For the guys who said that you do have some idle capacity sometimes Do you ever power down nodes and and there's at least two variables to that one, right? Is it worth it power wise to power them down? Do you actually get any cost savings and number two is it worth it? Mechanically to you know spin down the drives and spin them back up and things like that or does that? Shorten the life of the machine and things like that. So Andy. I'm gonna throw this back to you Sure, we we don't do that at present. We have investigated it. I think we probably will get there Because I do think that power savings for us will be noticeable. We're doing some also some work and some data center stuff Which should reduce our power costs, which then might not make it worthwhile We just leave them out all the time, but We do have just recently we've come into some idle list And we've been getting closer to looking at powering things and we would do it in an automated way We run Moab and torque here and they have a truth add-on product that would that supports doing that And that's what we're looking at using to see if we can do it, but we've not we've not done it yet now Okay, Preston. Do you guys have I I think her call that you said you have idle time too Or do I have that backwards? Well, we we we do have we do have idle time in theory so if If our if a faculty investor that isn't using their time it is in theory available for others to use in practice the between the The faculty using their dedicated cues and the standby access where where I can borrow my neighbor's cue The cluster is used about 75 80% of the time For that extra 20% though we've we've developed a solution where we've created a very very large Condor grid that runs behind all of our clusters So while pbs primarily scheduled to cluster when the nodes aren't being used by pbs Condor is free to schedule on them and that condor resource we make available to anybody on campus at no charge So in practice when you figure in Scavenging the cycle of the condor we get much much much higher utilization Into the 90% So one way or the other that idle capacity gets sucked up somewhere Fascinating Brian, how about you? Yeah, we you know, we don't really have much in the way idle time movement Actually, just check both of the cues right now while we were talking and one of the machines is not nighting 95% of course earn use excuse me 95% of the nodes are in use and The new machine Oakley's 98% and this has been pretty consistent for us for about eight months. So We don't really power things down for a power savings But you know, we're at a situation where you actually hit the cap for power consumption in our facility And we had to size our new machine appropriately to sit inside that power one We had to buy something a little smaller than water that I think our goal would be to never power anything down is to match that, you know the demand to the resources accurately as we could So that's actually our primary goal is to do that and not worry about powering things down So we've heard a couple different things. Um, you know units being a core units being a node when it comes to actual scheduling policy and mixing users across stuff Especially you Brian with the just giving out the CPU hours How do you manage an actual queue within these allocations use fair share policies or max proc or what kinds of tricks do you do? Yeah, we we actually kind of do all of those things We have a hard cap on the number of cores anyone users and be using it at any one time the hard cap and the number Actually the cap happens to be the same on cores for a project. So, you know all of the jobs in particular project could be using I think it's about 2048 cores at one time And we also have caps on the number of jobs per user and jobs per project And that's just to you know avoid any one user sucking up one hundred percent of the resource at any one time We also do some fair share So and I don't remember the exact numbers that we use but we do do some of that stuff too To kind of keep one person from starving too many other people from getting access of course Preston, how about you guys at Purdue? Well at Purdue we allocate it such that each each faculty owner gets gets a dedicated queue set up for them with with a max proc limit Based on the number of cores that they purchase into so if I buy one 16 core node I have a queue with a 16 processor max proc Within within that owner queue. There's really no limit. There's no cap on jobs per user There's not fair share by default. We do have a couple groups that that are interested in doing that Within our standby queue like I mentioned earlier as we're as we're moving our clusters to Moab We are going we are going to set it up so that it is fair shared with them within that standby queue Yeah, so here at Michigan for our allocation based system Everybody's sort of in a way they sell limit right they tell us how many cores they want And that's then how many they can use it any one time And we do do fair share amongst the groups and users so In the event the system is busy it totally fall right at 100% full Which is the only time that stuff actually comes into play is if there are cute jobs to order Which like I mentioned at the beginning our goal is to not ever have Very many or any jobs in the queue we want them to flow through pretty quickly So hopefully stuff doesn't ever make any difference But within one project of course it might you might have a two-core allocation you asked for And you know 50 grad students and that would be then very busy But then we do apply fair share within that so people can have a fighting chance of Of getting their stuff to run at least no one person can dominate it for a long time So let's let's go through that same round again, but there was something Preston early on you mentioned that you've kind of gone to a servicing like a faculty can't find and find a node with Their specific name on even though that's what they're that they paid for a node Does that mean if they have 16 cores that they can have if they submitted 16 one-core jobs They could be running on 16 different nodes Yeah, that's correct. They'll get at least a number of of cores that they've that they've bought Yeah, so we allocate based on the cores, but we but we sell based on nodes But what about something like memory like what if they request they just submit 16 one core job But each one of those jobs request the entire memory of a single node Then in in practice that'll work out that the bat system will only let them get one job through at a time So Andy, how do you guys handle that kind of problem? Yeah, we sort of do it. It's it sounds like in practice the same way that the Preston is doing it Every the actual unit that you get is one core and it's associated memory. So in our system, that's four gigs of RAM Um, so if you say you've got a 16 core allocation and you submitted a one-core job I can't do that math made. Let's make it easier. So you've got a two-core allocation You submitted a one-core job with asking for eight gigs of RAM You only that one would start the second job would wait for that one to finish So we do count resources both in processors and memory in that way They're not tied to each other explicitly Where and you can only run four gigs of RAM with one core But the sum the sum of them all has to add up to what you have been allocated And then Brian, how do you guys handle this and how do you charge your CPU hours based on you know? excessive resource use other than CPUs Well, we so we only explicitly charge for CPU usage So if someone is requesting one core on a machine, but they want all you know 48 gigs of RAM on that machine Currently they're only going to be charged for that one core However, you know, we've got the ability now with Oakley to use containers Not where I'm right at six I believe So we're investigating. I think we're probably going to start charging basically a proportion of the node You know, so if you're asking for half of the RAM on the node to be charged for six cores Even if you're only using one and we'll probably do something similar with scratch-based in the nodes as well All right, so just out of curiosity just to make sure I understood what you are saying you guys actually do In real time allocate more than one job per node if if required So like it let's say I've got a 16 core machine you couldn't schedule two Eight way MPI jobs onto those onto that machine. Is that correct? So we schedule Serial jobs and we could what so I should perhaps clarify our terminology We consider any job that stays on one node a serial job whether it's one core or you know all 12 on Oakley So we'll allow serial jobs to co-exist on a node So somebody could be using six cores and another person be using the other six cores Parallel jobs we require Entire node usage. So if you want to run across two Two nodes we consider that a parallel job you would charge for use of 24 cores even if you're only using one core per net Gotcha. Now you guys the other guys Andy Preston you guys do something similar We are purely core allocation. So um, yeah, we could end up if we had We have 12 core nodes right now. So if we had 12 12 way MPI jobs it is conceivable that One task from each MPI job would be on one different node and then be then 12 different MPI Jobs one test for me to one on a node Gotcha.  Does it does try but it's a good question, you know, and I don't know if it's explicit It doesn't generally work out that that happens I don't think the scheduler is that Aware of it, but it does try to pack things as tightly as it can in that sense So we don't we don't see that in practice very often And we do allow people if they want to make cost them some queue time But they're allowed to request if they want I want all 12 of my cores on one node So if the human interaction by the user can influence that Yeah, and then it then a pretty we operate the we operate ours the same way it's by the core The scheduler does try to pack them, but if the user wants to spread them out, they're free to do so Okay, Andy, let's start with you Why don't you give us three things that you like about your allocation model and three things that you don't like about it? The three three things that we like about it so the first thing is we do get pretty good efficiency with respect to hardware usage machine room usage Staff usage for operations, especially So we think that that's a quite a nice thing for us. That's one good thing the second good thing is we one of our original goals years ago two and a half years three years ago and we started thinking about this was We thought the barrier to entry was high for people who may not be Computationalists they might be though So we thought boy, you know asking somebody to give us a couple thousand dollars and make a three-year commitment Just so they could see Is a lot So we thought boy, it'd be nice if we had a way that you could come in and try for just a little while So one of our goals was in fact to lower the barrier to entry So, you know, if you have a couple hundred dollars say you had a hundred dollars You could get five cores for a month You can run an MPI job with that you could you could get some modest work done And it would cost you a hundred dollars so that we thought was a nice. That was a nice way to start for people So we think that's that's quite a nice model It's very nice feature of our model I guess See The third the third good thing boy Brock is there a third good thing about our model Great So that was I mean that was I guess there are only two good things about our model And I think that they really are they the efficiency of it is we think is quite high And we think that we've enabled researchers here at Michigan to To come in And give it a try without a big commitment or big expenditure I don't have to actually own hardware that I've worked on machinery in the rack space And because of that we were able to provide a fairly wishy-configured resource, right? It's all in finnaband we have quite a bit of scratch base using luster over in finnaband So it's things that you could not even with a medium-sized cluster. You may not get for your own You have to be at some scale To to achieve those things at a reasonable cost, so I guess that would be the third good thing about our model is the We're able to provide a Fairly rich feature set at what is essentially incremental cost now for people which is which is nice Okay, Andy. Let me let me interrupt you. Let's go get three from everybody and then we'll come back and do three bad from everybody So Brian, give me give me three good things about your model So I think you know, I think from our perspective as a shared resource center One of the best things about doing the way we do is that we keep our utilization very high. We don't have a lot of Idol notes And and when we do it's the case that anybody can submit a job and then we'll run very quickly and so Looking at it the kind of user community as a whole utilization high wait times are low Kind of a second thing that I like about our system is that requests for resources Are justified a reviewed Be that your review process. I think that that's a nice way to ensure that That what we're we're handing out resources appropriately to to what their needs are rather than what they think their needs are And the third thing that I think is good about this is there's no real barrier to entry Small users, you know, maybe don't have any money I know we've talked about small enough money But maybe somebody who doesn't have any money can still access these HPC resources that you don't have to worry about Purchasing a portion of cluster or purchasing their own cluster You know, maybe they only need the resource for a short period of time and they can easily access it with our allocation model Okay, cool and Preston So what one of the the best things about our model is I think a couple of them center around the cost Since we since we do build a community cluster every year It's a predictable thing for the for the faculty To be able to get computing no matter when their grants may be arriving or when a new faculty member may be starting and then also because we do this every year it It realizes some pretty tremendous cost savings to the entire institution over the years are We've negotiated with the vendors to get good prices on the clusters But then we're also able to use those same configurations and prices for servers in every area of the university So we can save a million the dollars of in server acquisitions over the last five years For the for the faculty member the barrier entry is is fairly low It's not zero but at the cost of only a single node at about two or three thousand dollars It's it's pretty minimal. So just about anybody can get access to the computing Okay, and let me flip this question now, you know, what are three bad things about your model? Let's go in the same order again Andy Sure, so now that I've had some time to think I'm probably be better at the bad things But but I don't think that's a reflection of my opinion of our model quite quite fond of it Actually, but the three bad things are what one is We have spent a lot of time here at U of M talking to our internal audit group and we've not yet figured out a way I'll be getting closer to accepting grant funded hardware with the restrictions that come with that. So You know, if you're listeners, you're familiar with the A21 rules We we hear to those from quite closely here at Michigan and that prohibits sharing of hardware If it was purchased on grant money with anybody other than the grant. I think there's some flexibility in interpretation Nationwide on that, but the interpretation here is quite strict So we because we have a totally shared model what they said for utilization reasons and efficiency reasons We don't we can't really take a grant funded hardware into our model So if your grant funding is hardware specific and not just plain flexible money The model doesn't work and we have condo clusters to address that So there's that issue the the second thing I think is our granularity is a core for a month There's been a lot of interest in people either getting direct bill pay for what you use Or we're getting a smaller granularity core for a day for example That's not to be fairly complicated for us, but we are working on that as well People are gonna see the same cost. I'm afraid and it might be less attractive at that point But but the granularity is weird for us in that way And the last part of the model which is truly a model thing. It's a financial thing Because we are obligated to recover every cost We are somewhat The things that we cannot charge for a tribute direct costs to we cannot not put into a rate We cannot not we keep their costs. So those are things that we don't do At least not as part of them this very narrowly scoped model of the hardware things like consulting for example If we don't outside of it different funding streams that kind of stuff So that makes it a little bit trickier for us in places We do address that by sort of having a stone super approach to it where we asked everybody to throw in a consultant And then pretty soon we have a bunch of consultants we can kind of work with so that does work out for us But it makes a little bit challenging because then it's not centrally wrong. It's it's very much, you know contributor based So that's those are kind of the three. I think it shortcomings of it So, okay, probably some other ones as well. I'm not thinking of so I don't you know, it's not perfect, but But it's not good Brian, I think you know one of the big challenges we have is Convincing people who do have resources and funds the value in contributing to a shared resource like oh, I see you know, I Think condo model is much easier to sell to somebody then economies of scale Which we can provide and and the ability to to kind of maintain the shared resource You know if somebody's buying their own cluster in their lab. They have to maintain it They have to pay for it They may not have the expertise to run the cluster as well as we can and that's a hard That's a hard sell It's a hard way to convince people that to kick in money to help us buy bigger bigger clusters So that's a challenge for us and one of the reasons that we're exploring the idea of perhaps Getting into the condo space For some individuals, you know kind of second problem some individuals weights may be long Depending on the resources that you're requesting If you if you need You know one of the larger memory nodes or GPU access, you know, it's We have like eight nodes with a hundred ninety two gigs of RAM I believe that's both of the forty eight on most of our notes So those are going to be in higher demand and the weights to get to those resources are longer The weights to get you know one in every ten notes of the GPU So the weights to get to the GPUs may be a little bit longer Than if you had your own kind of dedicated resource where you asked for that stuff up front and it was always available and Yeah, kind of big I guess the big white elephant in the room is we haven't talked about disk usage Long-term storage that's something we don't account for at all in our allocation model We tend to be very generous in giving out this space, but you know, it's we found that our users Can gobble up the space as fast as we put it out there and That's that's becoming a real challenge for us and in the usage is different than CPU CPU is you know, but It's very easy to to reallocate a CPU to somebody else when it's not being used But storage is is kind of perpetually in use. It's a different resource entirely. So Disk storage is just a big problem for us Indeed okay Preston, how about you? Hi, I find it interesting that you mentioned the storage because that that was actually one of the areas I wanted to mention the challenge for us Where we're where our problem lies is we don't currently have a provision in our model to do condo storage Which is something that our users keep coming to us with as I have X amount of dollars I would how many terabytes of disk can I get for that and as of yet We haven't got that got the perfect way to do that. So that's definitely a challenge that we're working towards solving We don't currently do the the number of cores times unit of time Model yet, even though we do have a we have in the last couple of months had several years of users asking for it So we do see that this is the direction that people are going to want to go So there will have to probably be some future developments and making that part of the model and then finally I think that one of the one of the weak points in our model is where it scales to to Very very large-scale users. We do have many research groups that buy hundreds of nodes on each cluster And that's great for them when they can when they can use That their entire share of the cluster But if they want to run something on twice as many nodes as they own which may be most of the cluster There's not a reliable way to do that yet We've we've got to do some more exploring whether we're going to do reservations have dedicated times where they were Somebody can do a whole cluster run and today that's really been kind of an ad hoc type of process So we definitely need to come up with some more structured ways of doing that in the future Okay, everybody. Well, thank you very much for your time I think a lot of the people who are looking at you know starting an allocation thing or Combining a couple of research groups together and getting some economies of scale will find this very useful So again Andy Preston and Brian. Thank you very much for your time and we will have this show up soon Yeah, thanks guys, I think we could have gone on for another hour here. This is fascinating stuff Well, if there's if there's response from the from your listener group that there's more interest I think I'd be at least in for doing another hour of it Yeah, I would do but I think per your suggestion over the instant messenger. We should do it over beer next time Well, that makes everything better doesn't it? Yes, it sure does. Okay. Thanks guys