After Friday's win, the Beaver's soccer team was in uncharted territory, playing in the second round of the NCAA Tournament for the first time in program history, in their way the defending national champions from central Missouri. Turf clear for the most part, but still a snowy afternoon, scoreless in the first half, the Jennies attacking, Jada Scott passes one to Makayla Toth. She chips it up and over Anna Fobi and they take the lead. Second half, Beaver's trying to equalize it, Rachel Norton dribbling down, passes it up to Sarah Wendt, she beats the keeper, but the ref's flag is up off sides, so wipe it off. Bemidji still down one, Courtney Killian in the 54th minute, dashes past the defense, over to Maria Wilder, who boots through it in. Bemidji unable to come up with a goal, they fall two to nothing, 10 shots on goal for the Beaver's, but just never able to break through all afternoon. But I was proud of our kids, just same as the whole season, just, you know, they put everything out in the field and as the game progressed, you know, we put enormous pressure on them and, you know, I think if we could have fallen that first goal, we would have fallen the second and it just wasn't our day to day.


Thank you. Thank you very much, folks. Mr. President, how are you? Nice to see you. Nice to see you. I think some wire service photographers are here. Hello. Nice to see you. Nice to see you. Thank you. Thank you. Congratulations. President, uh, February the 6th, 1985, President's meeting with Cadet Wing at West Point. First Vietnamese lady to go into the West Point. How'd the thing inside go? I think all of you know why we're here. It's just a real chance that you're right. To say thanks for what is up to now in the President's service, but in a few minutes we'll be in your previous service. But we're also being willing to take on these new chores and assignments. I'm deeply grateful that the country should be. So now, you raise your right hand and repeat after me, but I I strongly swear I will support and defend the Constitution of the United States against all enemies far and against all enemies, far and against all enemies, far and against all enemies, far and I'll bear a true faith of allegiance to the Saints. I'll bear a true faith of allegiance to the Saints. I'll take this obligation free. Without any mental reservation and that I will well and faithfully discharge the duties of the office on which I am about to enter. The duties of the office which I am about to enter. So help me God. So help me God. Congratulations. Thank you. Thank you very much. Thank you. Okay. I'm done. Thank you. Hi, John. Thanks. Thank you. Thank you very much. Take one, or carryutians, believe in the family. Yes. All right. Yes. Thank you. No problem. All right. So take one, or carryutians, believe in the family. Yes. Alright. Yeah. Bring everybody. All right. Bring everybody. Yeah, bring everybody. No problem. You're right here. All right. You're right. Okay. Thank you. Thank you. It's just a great job. We move along with this director. It's a nice, happy staff. It's a good place to be. Stage manager. I have six more years now. I have six more years now. I'm going to take a look at what you've got. I'm going to take a look at what you've got. Thank you very much. Thank you very much. Thank you. Thank you. Thank you.


Pretty strange out there I'm sorry, I lost track Something amazing happened Well, I thought it was something amazing What's up? Let me help you What are you doing, huh? You holding out on me? You drinking? You ruining your hard work? Smiling around Huh, what you say? Nothing I've not been over on purpose Why you do that?  I'm sorry It's late, I know But some things happen on the way back Do some stuff You can go through them when you're... You okay?  What you say? I asked if it was jukebox He said he was leaving the house real early No, no, no, before that You asked me You said somebody was coming around Yeah And I asked who Must have been no one I'm asking if I leave Where I got to go Why I stay Ain't I scared Ain't it a sanitary church kid? What you tellin'?   Black dumpster trucks and social workers Church folk and police storm in the shade Tomorrow morning Someone was murdered, remember? I've been too When was that? All that screwing them out Turned them into the police, didn't it? Drugs And whatever The point I'm saying Is where these people going Tried to Do you know this new mayor Tried to pass some bill To give folk a bus ticket out of the city? I told you, don't care What about us? No, I don't Well Where are we going to go? Have them smokes They smokes Can't believe we finished our day yesterday What?  You mean this?  What's it gonna do about it?   Guess! I started the morning The way I normally do, right?   Now she got a big, big fine And what? So what?


Gleben Gloutengloben. And we're live. Hello everybody. It's Monica for your data science chat this Saturday. Oh, I already see one person. So as I'm waiting for people to join, that's what I usually do because I can't figure out how to do a little, you know, we're waiting or, you know, the chat's going to start soon or whatever, you know, because all I can do is data stuff. I can't do anything else. All right. And that is reinforced by my previous live streams. I've been trying to watch them. When I watch them, I realize there are people in the chat. And I couldn't see it. And I think the reason is I'm using restream. And restream, first of all, I think it's good software. So I'm not, you know, saying anything bad about it. Probably I'm not using it right. But one of the things is that I'm low budget. I'm a low budget streamer. Like I don't have any moderators. So if I had moderators, they could be actually watching the YouTube and the LinkedIn stream, because I'm doing them to both platforms and tell me if people are chatting there. And I'm not seeing it. But I don't have any of those. But I figured out I have my other monitor over here. So if you see me turn my head, it's because I'm trying to see if anybody's actually talking to me. And I'm not talking to them because it's not coming through on restream. Oh, I just realized I have to mute myself here or I'll be all confused. All right. So let's see here. So well, I guess we haven't even started the stream. So it's okay if I'm doing like technical difficulties, please stand by kind of stuff. Which is exactly what I'm doing is technical difficulties, please stand by kind of stuff because I'm just trying to get this technology working without any moderators. So I now restream tells me that there's somebody here so welcome. And if you try to chat, it should see how I look a little different today. That's because there's this place called there's this chat overlay, right? So I'm using that to try and make it so that if you're listening to this and you're watching this and you try to put something in the chat and it doesn't show up over here on the chat overlay. Well, hello, Mutua Robert. Thank you. See, now we have some accountability here. He said hello and it showed up on the chat overlay. Hopefully you can see that. So that's why I'm using the chat overlay is so thank you for chatting, by the way, because you allowed me to test this. So because and go ahead and say hi. One of the things I was trying to do with these live streams is to make it so that people showed up just to meet each other. Like even if you're like, I don't care about G-Power or I don't even do sample science calculations, I could make it so that people who are like data science people would just show up and just meet each other. Because what I started realizing, especially during the pandemic, is when I would go to like our things or whatever, people are just not that good at chatting. And that's because like the presenters were not like me. Like I'm like the party data science girl. Like I want everybody to be talking. And you know, I'm an educator, so I'm cool with questions and people being uncomfortable.  You know, like I don't know how to make that arrow in R. You know, just say anything. Like me, I have so much trouble with them. What is it? GitHub? You know, putting anything on it and stuff. Oh, hello. Hello. Thank you for showing up, Meyer. And thank you for showing up, Alexandra. I hope I'm pronouncing your names right. You can tell me if I'm not and I'll learn it because I'm good at that.  Some people call me mine. They say mine. Help me out with this. Help me with that. You know, so you probably hear me say that. I talk about people talking like that. But I'm really glad this chat is working and thank you for showing up. I have some nifty stuff in the store for you today. Part of the reason why it's nifty is because it's free. G-Power is a free software and that's my favorite price. If you hang around me long enough, you'll learn that. And I'm actually getting better with the software. So let's see how I do. All right. So it's one o'clock now, Eastern Time. So I'm going to officially open the stream. Thank you for coming today if you're here. If you're not here yet and you're listening to the recording, thank you for listening. What I am going to be talking about today, and this is on the, just the inspiration of one of my connections named Rima who had a competing meeting. So she's not here. That's fine. But it wasn't it wasn't just Rima, but also other people have asked me about sample size calculation or power calculation. And just to be clear for everybody, what that means is when you start to do a study, and especially if you're going to do like a clinical trial where you're comparing one group to another group, let's just pretend. One of the first things I used to do as a research secretary was I would work in cardiovascular epidemiology and we were comparing like blood pressure between groups and we would give people blood pressure medication trying to lower it and we wanted to compare it. We do these randomized trials. And often we had data from not clinical trials, like we had, we would had observational data like what happens if you give M. lotapine to somebody with high blood pressure, you know what happens, right? So we have some hints about what we would expect if you actually did a randomized clinical trial. But the problem is we needed to decide how many people we needed to recruit in each group to see if we could answer our question. Well, what was our question? Sometimes our question was is A better than B? Okay, so A may be the new drug and B is the current drug that everybody uses. Or sometimes we would say is A at least as good as B. So in that case, A would be like maybe like a natural version and B would be the drug that everybody's used. So it depends on sort of how you're asking the question, but the bottom line is this. Truth is out there, okay? A and B are either equivalent, okay? Or one is bigger than the other or the other one's bigger than the other. Like there's only three things that it can be, right? Now, we all know that things can be like different, but if they're a little different, they're basically the same. Hence, you know, the null hypothesis and all that. So the trick is we don't know if they're essentially equivalent or if it's this going on. We just don't know. But we know if this is going on, we don't care. That's equivalent to us. So we just want to make it so we get enough people in each group so that if it's like this, we know. And if it's like this, we know, okay? Now, if you don't get enough people, it always... Oh, hi, Zahira. If you don't get enough people, unless you're looking at something that's so big, you can just tell before you do the clinical trial. If you don't get enough people, everybody looks the same. So like if I say, you know, does amlodipine work as well as hydrochlorothiazide, if I just get 10 patients on either of them, it'll probably look like they're equal. Like I won't be able to tell.  Why don't you just have five zillion patients? Well, we all know that would cost too much. And also it's bad to give everybody all that blood pressure drugs. So when you're doing a sample size calculation, we're trying to find us that sweet spot. The number of people you're going to have to put in each group, the least amount of people to put in each group to see if it's really different or it doesn't matter how many people you got, it's the same. Right? Like so let's think of things that are the same, that we know are the same. You know, anything, I don't know if anything comes to mind, but like people who are, like if you use an anxiety screen, a lot of the anxiety screens come up with the same answer. So if you use an anxiety screen, two of them on the same person will probably be correlated, probably be the same thing. So if you wanted to see if those two are different things, it doesn't matter how many data points you had, they're not different. It'll never be different. Okay, it will be different. It will be different once you get five zillion because that's when statistics don't work anymore. I mean, even you see me demonstrate with the behavioral risk factor surveillance survey data, that's like 400,000 people in it. Like the mean age between men and women is statistically significantly different because there's just too much data. Right? We don't really have that problem in clinical studies where we get too many participants. That's never the problem. The problem is we get too few. We're always trying to get the least amount necessary. We don't want to hurt people if we're giving them drugs and we don't know if they work. And also we just want to put all people in as expensive and everything. So that's the politics behind sample size and power calculations. So what people will do is come to the statistician and I'm trained as an epidemiologist but we're taught power calculation. But then I got, I opened a side shop of sample size and power count. Oh, I'm sorry, I'm going to have to cancel this person. I have this zoom bombing problem here. As you can see, I caught them quick. Oh, here's another one. Let's see. Oh, I think, I think I did it. Okay. So I think I canceled that person. But I'm sorry about that. Like, again, low budget, no matters. Oh, let me just look in the chat before I move on to just the other chats to see if it's anybody besides my zoom bomber here. Nope, it looks like I'm catching everybody. Okay. So let's see here. So the I opened my sort of like side business of calculating sample size, mainly because when you go to a statistician and you ask them, it's often a very painful conversation. And you might wonder why. And if I show you an actual use case like I'm going to do in this live stream, you'll understand why. Because it is a very sort of sensitive thing that you have to sort of discuss. And a lot of people who are attracted to statistics are not going to like that kind of a discussion, right? So often, you know, when my friends would go to the statistician the same, Monica, I'm going to go get our power calculation. I'm like, all right, sure, you don't want me to do it. And they're like, no, we have a free statistician work. Then they come to us, we need 2000 people. I'm like, okay, come over to my, I'll talk to you, I'll talk to you. Or the statistician will say they need four people. I'm like, okay, that's not right either. So I'm going to just sort of show you the reason why I'm walking you through a use case is it's just a great example of what you go through when trying to design a study. And you'll see how in this case example, we're not just doing power calculations, we're actually picking the dependent variable, we're actually like kind of making study design decisions. And it's really important, because at the end of the day, that's what evidence-based medicine is, is doing the statistics and doing the tasks, a priori planning all that, and then answering it. And if you don't do all that, and care about the measurement, which so many people like just, they're like, who cares? Let's just say the clock.  But before I get into the use case, I just want to give you a quick overview of what G-power is. Now, I don't know if Daniel's in the chat here. So Daniel's my colleague who is the SAS guru. And he would be here, if he were here, he would tell you, proc power in SAS will give you anything you want in this software. I'm going to show you G-power. And he's right. Proc power, when you declare it in SAS, you put all these arguments in it.  You know, when you have a big equation with a bunch of variables in it, if you put all the variables in except for one, it'll calculate the other one that it can solve it for. So in proc power, you have a way of telling it the arguments that you have the answers to, and then it's going to tell you. And you'll see G-power is kind of doing the same thing. We're just using menus and stuff. Oh, I have a question, so I'm going to answer it here. So would the way sampling, such as simple, random, and stratified effect determining the sampling size in any way, bingo, you get an A plus? Exactly, exactly, that would. And actually, I didn't mention it. We were doing, I'm going to get into the case study, but we were doing a clinical study in an oral health clinic. Now, that's basically simple random sampling. We were going to do, but in that blood pressure studies we used to do, we would do like a blocked randomization, which is kind of like stratified sampling, because we wanted to get like the same amount of men and women in each group.  We didn't have to care about that. But yeah, the BRFSS is a multi-stage sample thing. So basically, you have to have a PhD in statistics, be working on their team that sets up their sampling frames, because they have to care about that. So very good question. You're right. But we were doing something very simple. So first I'm going to just show you, let me, how do I share my screen? Let's see, if I remember, Chrome tab here. First I'm going to share with you this web page. And let me see how this is looking at. Did it share here? Maybe I've got a little delay and I can't see it. But in any case, let me look at my re-stream. Everything is so slow. Well, it looks like it's sharing on my re-stream. So I'll continue. Okay, see this page. It's in English for me, but if you go there, you'll see I link to it in the description, you know, in our agenda. This is native in German. Why? Because this is actually a German university. Okay. And it doesn't, like, see this manual and everything. And then you go down here. This is where you download it. And you just download it and install it. And it does seem sort of sketchy. It is not. It has been the subject of peer reviewed articles. Here's all the documentation of what it's using.  Okay. And then if you download it and you install it and you can just run it, I ran mine and I just ran it and I didn't do anything next. So we're, I'm going to share that window with you, which is, here it is. Okay. So when you run G Power, this is what you're going to see. Is this this thing here? Okay. And what will happen is that you can don't don't really look at this up here. You can see that there are these two tabs here. I don't really look at this tab. I usually leave this one out. But you start with this test family. Now, if you're familiar with the statistics, you kind of know what these tests are. Like these, the sky square is going to be a categorical, like how many people, like proportions, how many people get diagnosed during the trial, for example. But we were talking about, like I was talking about blood pressure, we would use a T test for that. And you'll see that if I change this, it'll change what this form is. So this is basically a way of sort of taking proc power and turning it into a menu driven thing. And then, so if you choose T test, there's only so many different scenarios for your study design you would have. And so you would pick the scenario here. And then you'll see here, like this is tails, you're looking at tails one or two, you know, the effect size, you have to put in here the alpha, remember, 0.05. And then this is like, I usually use 80% for this. But then there are all these, like you might, one of the things here is we usually set alpha at 0.05 and we'll set power at 80. I don't know why this default is 95. I almost always use a two tail thing in healthcare. But what always is a problem is this effect size, is knowing what the effect size is. And so what's great about this G power, it'll help you calculate effect size if you have some estimates from the literature. All right, so let me go back here and see. Okay, so what I just went over was just what is this software that we're going to use for this use case. And just let me know if you have any questions. Okay, now I'm going to get into the use case and then come back to the software. Because this is the part that statisticians often don't do well with when working with the principal investigators. They don't do well at this communication stage of trying to figure out what exactly is going on in this study and what are we trying to do. Because you really have to understand the study design to be able to pick the right measurements and to make the right choices. You will see as I do this. So I'm going to go share my blog post that I made because it just helps me walk through and make sure I don't miss any topics. And there's a link to it. So let's see here. So this is the link to download it. All right, so the thing we were studying was gingivitis. And I don't know if you've ever had gingivitis. I have. It's where your gums swell up and then they bleed. And which reminds me I got to brush my teeth again. But anyway, so gingivitis is not good.  It's pretty easy to solve. Like there's a whole lot of literature that says chlorhexidine, which we have reviewed in the CX, that chlorhexidine mouthwash will, you know, it's got alcohol and stuff in it. It will just kill all that bacteria that's causing inflammation. And so it's great. But there's two downsides, right? One is that it has alcohol in it. And if you're in the Middle East and there's people don't drink alcohol, they really don't want to use alcohol mouthwash. Not everybody, but a lot of people are like, yeah, I don't want to do that. And I kind of don't blame them. I understand. And also even I in the US, I've hung around a lot of people who are in recovery from alcohol. They don't even want, they don't want it. Like if they taste that, it's going to push them the wrong direction. So, so that's an important one. And then the second one is chlorhexidine, unfortunately stains the teeth. So I, one of my collaborators is an innovator, and she's just so amazingly intelligent. So she mixed up this potion of natural remedy based on stuff she saw from the literature. And she called it, we called it NSM. So NSM is the alternative mouthwash. So notice already, we're going to do a clinical trial of gingivitis patients where we're comparing usual care CX to NSM. And think about it, we're just looking for equivalence, right? And like if NSM does as least as well as CX, it's good enough, we can prescribe it instead of CX. Okay. So already it's sort of complicated. Okay. So then let, we, you know, this, the study was more complicated than like this is. I'm just focusing on the parts that were important for the power calculation. Okay. So we had several visits, but at visit one, which is baseline, what we decided is we would measure the outcome that we chose that we were trying to impact with the CX or the NSM. So remember when I was talking about blood pressure studies, if you, you know, we would look at systolic blood pressure or diastolic blood pressure as the outcome. Well, here we had to choose an outcome. What is this outcome we're going to do? And that was part of this power calculation to figure out which outcome we exactly wanted. But whichever one, we were going to measure it at visit one. And then we were going to measure it again at visit three. And in between, there was going to be treatment. And what was treatment, it was going to be either they were going to use chlorhexidine mouthwash or they're going to use the NSM. Now I love working with this PI. She's so brilliant. She's super nice or super close. And why is that important? It's because she just really cooperated with everything I asked her to do to help me create like the best sample size for her. I love it when people are like that. She's like so smart. But anyway, so in oral health, there's this sort of standard measurement of inflammation in your mouth. And it's called bleeding on probing or BOP for short. So if you think of a dental probe and you probe the dentist probes your gum and some blood comes out, it's kind of bad, right? So what happens is bleeding on probing is technically calculated by like they'll probe these sites and technically they'll probe six sites on each tooth, but sometimes people have lost teeth. So it's not it's different for each person how many sites they end up probing. But that is the denominator. And then the numerator is how many bleeds. So if you get a proportion of 1.0 or 100% then they're all bleeding. This is very bad. And it's going to be 100% whether you only have a few teeth or you have a full mouth of 20 teeth, right? And so that's what's kind of nice about bleeding on probing is it's like you can use it to compare people. So somebody has like a BOP of like you can use you can use a percentage but I use like a proportion for this.  You know and just to be clear most people have healthy gums and they don't they have a BOP of less than 25%. It's gingivitis is like more than 25%. Okay and so we were thinking of using bleeding on probing. I just wanted to check the chat here. We're thinking of using bleeding on probing as an outcome because that's used a lot in the literature but I was a little nervous about it and here's why. I was afraid that their gingivitis patients might have a small range, right? Like there's really bad gingivitis patients might only be 0.4 or 40% you know because you have to be 25% to even be gingivitis and of course this was going to be on gingivitis people, right? So I just said you know can we do a pilot study because I want to see what kind of gingivitis you're getting in your clinic and I also want to see you know maybe there's another measurement we can do like a way to classify them. I came up with it on my own. So just to reiterate the BOP that we were thinking of using it had a range of 0 to 1.0 because I was using the proportion and I was worried about the range. It was the minimum was going to be 0.25 because that's definitely gingivitis but I was afraid I was just not going to get a good range and if I didn't get that you know if people are too healthy they can't get better no matter whether you give them chlorhexidine or NSF. So I was just worried about that. So I also came up with this experimental measure which is more of a rating scale that the clinician would do a range of 1 to 25, okay? So what then my my colleague did the PI is that she did this piloting and she she just went to her clinic and she said she measured 25 people of gingivitis and 26 people without it. She just measured them in terms of BOP and in terms of the new measurement I created and that way she was able to give me like the mean and the standard deviation for each group. Like the mean BOP for people without gingivitis like is it high or low you know is it close to 25 or 0.25 and and how bad are these gingivitis people and standard deviation right that's a measure of variance you know range standard deviation and variance I could see for myself right and so we now could use g-par to make our estimates. So I always have this joke when people come to me and they say Monica will you make me a correlation I say yeah yeah but you know what you have to bring your own estimates you know and the reason why I say bring your own estimates is I need something from the literature in order to be able to know what to put in g-power for example okay and um oh there's another thing that um I want to show you and that is the notes I cast because one of the things I said about this presentation is I was going to show you the curation and that's almost like more important than anything because that's what allows me to do this presentation today because I think it was years ago that we did so I don't even remember okay so I'm I didn't make this available to you but I'm going to show you um a word document I kept just so um so you understand like why it's so cool a little bit over here why it's such a cool thing why g-power is so cool okay I literally made this this document okay I've edited a little to just uh did act you know redact like confidentiality so I said here PI and I needed to decide between BOP and another measure to power the study figure us out PI did a pilot study where she collected it and 25 people have been invited which 18 had a BOP of 0.25 or greater or were in of the moderate severe diagnosis we are recruiting for a study okay that's what was going on 26 people were healthy we put this on a spreadsheet I counted the mean total score and the mean BOP for both 18 with moderate to severe and the healthy ones to decide between BOP and another measure I decided to do a power calculation for both note another major has a range of one so this was the power calculation I did for the other measure so you'll notice there's this main window which I was just showing you but there's this little thing on the side okay and I'll explain what that is this little thing on the side pops out under certain circumstances and this is where I entered this data here and I actually put it on some slides it's a little hard to see but this was data I got from the pilot study and this was for the other measure okay and then this was for BOP and again here I am entering the data from our pilot study and uh and then you know I'm making some notes like because I'm changing some of the data I'm playing with it and then I wrote this we finally made the following decisions we will power this on BOP because just remember I can still use my other measurement in the study we're just not powering on it okay so we're powering this on BOP we will power it on a change from at least 0.45 at baseline to 0.25 at follow-up we think this is safe because the estimate from the pilot study was 0.65 at baseline here it says alpha 0.05 and power 80 total of 11 these groups we will get 13 in each group okay now you'll see I say on my blog post that I don't like to get less than 30 in a group okay so of course I always have the argument with this customer all the oral health customers I have an argument but they always want to have like 15 people in a group and I'm like okay try it but remember what I said if you don't get enough people if there's a difference you won't be able to see it and if there isn't a difference you won't know because you know you didn't have enough people so it's like kind of are you sure you know you want to do it but anyway so I told her 30 in each group I told her I'd savor the trouble and just start with 30 each group but then all right so then another thing occurred so this is how we decided to power it on BOP okay but then another thing occurred and that was I just sort of thinking about what are we actually asking okay so I'm going to go back to our the blog post because I want to talk about you know what like this just kind of explains what I just showed you and this just shows you what we entered so what I was asking was what we were asking was um does does nsm work as well as chlorhaxidine and the reason why chlorhaxidine works is because if you do a paired t-test on people who got it at visit one and then or who had gingivitis at visit one and then um got chlorhaxidine and then you they don't have gingivitis at visit three and you do that t-test it's always the paired t-test is always greatly statistically significant because it works correct so what I this so technically if you want to know if nsm works better than chlorhaxidine or worse statistically significantly you're looking at t-testing the difference that difference between groups but I don't really care I just wanted to see if the nsm at least worked as well as chlorhaxidine does it reduce gingivitis at the same rate like in other words is the experience in the chlorhaxidine groups the same as the experience in the nsm group because if it's the same then they're the same in a practical clinical sense so what I decided all I needed to do and and I think it worked out I can't remember the study all I needed to do was to make sure that you could do a paired t-test in the nsm group and if nothing was going on you would know it and and that I think we could do you remember my no I think we could do that with 13 people in each group I think if nsm was doing nothing we could see it with 13 people because we know chlorhaxidine with 13 people it would clean up their mouths because it just we have all this literature so if an nsm if it's not really cleaning up their mouth forget it right like you can tell so that that was like all of this politics and talking about it and all my background like don't you think I've read so many so many articles about about BOP and stuff why because I just love this PI we've been friends for like almost 10 years I think now we've been so close well why I just study what my friends study because that's what's cool about being a methodologist you can be friend anybody just hang out with them learn about them and then you know you've got a friend and and then you can do magic with them because uh no one knows how to do statistics or do science or any of that right so um so once we got there I was like you know I'm telling you the end before the beginning I you know I told her 30 in each group okay that probably would have been too many in this case and then she said okay no please do a calculation for me and I said okay pilot study and we did that and then we chose BOP and then I said okay and then she's like okay power calculation for BOP now you notice in that word document I said what we decided it was like 13 but how do we get there right so that's what I'm going to show you now is I'm I when I realized that I had the stuff we could enter into G power I started trying to enter it right so let's go back to G power let's see okay so and let me see here I still had I still have my word document open because that kind of helps me so you saw how I took a screenshot of what I ended up doing which is you know kind of a good thing to do because um because then you remember like what's settings you chose okay so one of the things I did so I'm looking at my screenshot here mean so what was I doing I was doing um matched pairs right remember I was just trying to see how many did I need to show a paired t test was significant and we are a paired t test is just asking is the difference between time one and time two statistically significantly different from zero you know so is it big enough we wanted a two-tail test because you really use those most of the time oh and operate compute required sample size given alpha power and effect size remember I said we decided on alpha I decided on the power was 80 percent because this is pretty standard but I did not really know what the effect size should be um but what's cool okay now here's the hot stuff this determine here you can click that right it even makes a little sound so I guess I designated one of the groups that's group one and you know what I'm going to do here is actually enter let me see I'm going to enter the final um um numbers I used to convince myself I wanted 13 so 0.45 was the mean mean BOP in one of the groups and the mean in the other group was 0.25 okay that's that's our imagination here's our standard deviation and this was taken directly from the pilot study okay 0.2 to 0.04 and so the first group was the gingivitis group so basically in this pilot study we had healthy people and gingivitis people we were using the healthy to estimate what the population looks like after treated and so that's mean group two and SD group two we're using the gingivitis people to estimate what they look like at visit one which is 0.45 whatever this is a little question mark FX size DZ at all DZ so you have calculate here or calculate and transfer to main window you do calculate it'll just show up here but if you do this calculate and transfer to main window it'll show up over here so I'm going to do that see that it showed up here and here okay now that's how I filled that in now I'm going to do calculate here isn't that pretty but this is what I see out here is this 11 and that's what I'm looking for is that number okay so you probably already said well what if I'm not so sure about my FX size which is actually actually how we were throughout this process because remember I'm showing you the end that this is what we decide on but how did we get there like how did we decide on this what you know obviously there's so many alternatives so that's the next thing I'm going to show you okay let me go back to is is how we kept track of all the different alternatives so that we could actually make a you know logical choice okay so I included a link to get hub where you can download the spreadsheet I'm about to show you but well actually I'm going to show you a blank version but then I'm going to show you a completed version because I made a blank version just to demonstrate just so you understood what was going on but when you go and you download it'll be completed okay so let me just share the blank version with you for a second here um okay so this is actually um let me see if I yeah you can see all the all the tabs on it okay you can see that there's different tabs here and I'm under sample size estimates so sort of at the point where the PI had gotten me these estimates and I knew I could estimate like like remember how I was filling things into estimate FX size I think remember how I was filling things into estimate FX size in the um in in G power I just said well what if I just fill in the different because there's not many effect sizes you can have because it's BOP it's bounded so I was like well you know like this is a big one like you know 0.9 90 percent right but most of these people are not bad gingivitis right you know like like the average gingivitis was around here so how much can you make it less but anyway I made up this sort of like homework for myself that what I wanted to do was put in the alpha and the beta which or the power and then calculate how many in each group and then total and because remember I'm looking for you know a comparison of the experience ratio that I put these assumptions up here okay um now I'm going to show you let me go back to I'm going to show you the filled inversion okay oh this starts you'll see I summarize the study design info in the first tab just so you can remember it and then here okay then here you will see this is what happened so basically I plugged in this information right and you know what if the effect size is 0.1 now remember what is effect size it's it's the end difference between the groups right it's basically the the difference between visit one and visit three like how much their gingivitis change right so if you were one of those 0.45 people and it went down to 0.35 that's an effect size of 0.1 but you could have been a 0.9 person goes down to 0.8 so I I didn't really know what to expect in terms of people going up and down but I just said okay let's just pretend that I don't care because guess what we can't get a hundred people we can't get 200 people in the study so we can't do that so let's just see what we can do right because remember the bigger effects you need to be your people see how if you do things like this you start seeing there's some boundaries right like you start seeing that maybe if you got 50 people in each group see this 45 you know like if you got 50 people each year like almost 100 like 100 total of this this 50 people each group you'd know if these big differences were there and chlorhexidine makes big differences you know and um I mean chlorhexidine cures it so you should be able to tell but then you know it's like how many do you need well I saw this 30 and each 34 and I was telling there you know you really want to be sure but I think in the end we decided on 13 because it was just easy to recruit them and if at the end of the day we looked at it and we saw a lot of variants and stuff we could go back to IRP and just say we need to recruit 13 or a few more we could get up to 30 each group so this is the interpersonal side of the power calculation is be able to negotiate and be able to like just say okay what what if we don't get enough people right because think about this was an innovation we had no idea what to expect from this NSM and also you know after 13 people maybe we'd see that there was nothing going on with NSM but I was a little worried so I did two different simulations the first simulation I did was this one okay it's kind of silly but in this situation I said they were sick CX got healthy and NS didn't right so CX v1 the mean was 90 I'm just doing this and I'm saying I didn't do it on the I did on the percentage scale I should have done them for course to go but I said like like this is their mean BOP at visit here's a visit to or I said visit to it should have been visit three C is a mess and then here's the difference right in this chlorhexidine group and here would be the difference in the NS and here's the difference between the difference would be 26 and here's another scenario sick neither got healthy then the difference between difference might be very small 10 not so sick and got healthy you know both of them so I was just trying to see what kind of differences and you can do this you can do these simulations make up fake people make up fake groups but it's I'm the only one I know who's ever really done that and I'm telling you it's so helpful because if you're working with a clinician they can tell you what fake people look like they can say oh no nobody has this or yes everybody has that you know and I'm not a clinician I don't see patients so clinicians are my patients so then um then I made this one I said fcx reduces BOP by this much 60 percent and NS reduces BOP by this much here's the effect and here's the effect size because notice the effect size is technically the percentage you know which I've been sort of ignoring in this discussion until now the percentage change and here's the effect size so this is a new list of effects sizes that I was plugging in and here's the n and h group but don't don't even look at this what are we seeing we're seeing third 29 45 so you're starting to see where we are we're like if you want to be sure sure sure sure sure sure sure sure then you'll get 50 in each year if you want to be pretty sure you get 30 but either way if nsm is awesome you might be wasting your money maybe you should start with 13 maybe maybe you should just get more later you know if that doesn't work out so see it's like probably the least mathematical and most sort of emotional uh thing in statistics is doing um these power calculations now I'm showing you like one use case where we went from what do we want for a dependent variable to okay uh how many people do we need each group and you know and and so now what's awesome is I don't remember doing this since it's on this but I know if they brought it back to me I'd know exactly what I was going to do I was going to do this pair of t-tails and try to show that the difference between each group was either the same or different you know like we try to figure out what was going on clinically with this nsm and so um to something I didn't tell you and I didn't put in the blog post is this pi is very sophisticated and so she had a lab and she had already and she has a bunch of students and so oh no not this again sorry um I didn't mean to show it I meant to block it someday I'll get better at this stuff let's see if I blocked it yeah I blocked it I don't know how is it that these zoom bobbers can find me but data scientists can't find me I guess I don't know maybe they think I'm sexy let me just go and make sure that nobody's chatting and I'm missing it in linkedin nope it looks like nobody's chatting let me see if nobody's chatting here um nope it's like I'm not missing anything um all right so uh I don't know how many of you are really still with me because I don't trust this software but if you are still with me and you have a question you know please try to put in the chat and I'll see if it comes up uh and try to answer it um but in any case I what I would say is that um let me let me look back at my um in my example and see uh yeah you know let me just put this out there for you um so um well one of the most important things I want to emphasize in this live stream is that you want to document all of that stuff like you saw all that documentation I pulled that out because somebody you know contacted me and asked me a question about this but most of the time when I have been helping um PIs but someone else did their power calculation and now it's time for us to like they finish the study and it's time for us to write it up I actually need to go and find out what were they thinking when they did this you know what was their um you know like I like to put in the period article I said we set the alpha 0.05 we said you know I would totally explain this I mean I probably skipped that we considered other measure but I would totally explain how we arrived I mean I I would explain you know what I filled in for our final power calculation to get 13 I would explain um that that's like how um how I got to 13 and I part of the reason I was working with this PI is she used to go to a school in Boston a college in Boston and when she was going to that college I knew her and they had a statistician there and he would make these calculations and bring them to her and I'd say well what were his inputs like what did he what did you put in the calculation and he would never be able to tell us and what I noticed was because I hung out with her for a while and hung out with her friends and stuff is that um it didn't really matter what the question was the answer was always 14 literally it was always 14 so I don't know not know what was going on with this guy um okay so this documentation stuff is reporting so you can write it up later or pass it along to somebody else so they can write it up otherwise you can't write it up then the next one is like I used to go the University of South Florida College of Public Health um if anybody's thinking of going there I would strongly recommend against it I did not there's some individuals there who are really awesome like what I'm going to talk about but their program was bad like they I wanted to become a data scientist and they essentially kicked me out so that's why I don't have a PhD but in any case the people I like there one of them is Dr. Eliang Zhu who's still there like you'll find out that I don't really listen to most people because I think they're full of BS but Dr. Zhu is so intelligent I can't even tell you and he's such a great teacher he like a lot of the stuff I teach you guys about statistics in a practical applied sense I literally learned from him I took just about every class he offered I tried to take a computational statistics class he said Monica don't take it you're you're not you don't like math that's how good of a professor he is so if you can take a course from him do it but don't go to the University of South Florida if you can avoid it but anyway one day there was a student presenting in class her power calculation she came up with like 14 like that guy and he said to her do you really believe like these 10 people adequately represent this whole population because think about it that's what you're doing when you're sampling is your is you're saying you know these people um I don't know I'm getting distracted what you're saying is these people I'm sampling represent a background population now if you think about my study I was doing what were we doing we were attacking this clinical population that's pretty homogeneous so I feel so bad about 13 people but I've been like I just met with somebody today and I read a study I hate to say it was in Saudi Arabia where they literally got like 100 people to fill out a survey on I don't know some sort of like I don't know use of the drug I guess I don't remember what it was but it was like they sent it to people on WhatsApp and on Twitter who is that who is that represent you know like like what population and so that's exactly the right question to ask is do you really believe these fill in the blank however many people adequately represent this whole population make sure you have a population in mind you know that's otherwise it's not good oh hi John Charles I'm glad you came to my uh live stream do you have a question I can answer a question um because I'm sort of coming to the end of what I wanted to teach you which was that um you can learn to use g-power and each time like actually I only use g-power now for my power calculations I could use our their great packages I could use sass I use sass I like g-power because it's so easy to take a screenshot of what you finally decide on you saw how I documented with all those screenshots also using this menu is a good way if you're like me and you don't do these very often to just make sure that you are um you know not doing some stupid thing you know because those words spelling it out for me you know otherwise I have to look up proc power and make sure I'm entering the right arguments and stuff and um and sure enough I'll probably get that wrong um so that so that's like using g-power is great but you have to know what's going on behind the scenes and you have to find some way of documenting all your queries that spreadsheet is a really good example and actually I made a bunch of different tables on there but I I want to I guess give a shout out to the people at um ccbr at the University of Minnesota they're the ones in the 90s when we would do grants back in the day there was the like this is how old I am there was a time when you could choose different alphas and you could choose different powers and so we would have different scenarios of alphas and powers and they put in a table when submitting a grant and they would use that matrix to say you know these this numbers here this is why we're choosing 50 in each group or whatever because it hits many scenarios and I'm like so clever it's just so good to do simulations you can just make make up the data in your simulation but it's hard to do a simulation if you don't have any real data to guide you so it's so important to do pilot studies and they don't really have to go through the IRB it depends on what you're doing like in my case with my um PI these were all her patients you know she's director of the clinic and she just had to write down their um their values so it's like not you know if you're in the U.S. HIPAA has a clause on it which is research like preparatory for research that's what that is is preparatory for research because it's like she just gave me means and um standard deviations or or values or whatever like I there was no real like human research data but that is so good because when you are like a biostatistician or acting like one on tv like I am you know you really want to like like for example cytokines what are cytokines cytokines are chemical messengers like interleukin you know if you hear like aisle six aisle eight aisle two aisle four you know those are um cytokines so what are your cytokine levels well I don't know I I think cytokine research is stupid and I don't do it but all my friends do it so I have to do it with them and one of my friends was doing cytokine research and she got a cytokine assay to measure it and she got these numbers and I was like wow look at these numbers they're over the place and then I went into literature and I found like I I don't know which cytokine was let's just say it's aisle seven because that's my favorite um so let's say like in one um article it would be like the mean aisle seven was like 500 in another article the mean aisle seven was like 5000 in another article the mean aisle seven was like 50 000 I was like okay let's find articles where they use the same assay we find the same assay the same thing like I I'm like how are we doing this how are you getting stuff on different orders of magnitude now I've seen that with platelets like you can have no platelets in your blood which is a very bad situation you have zillions of platelets in your blood so platelets has a really big range and you can have a lot of wonkiness in a distribution but that's not normal usually ranges look like blood pressure ranges or BMI ranges you know if you're looking at a continuous variable they're usually like bounded like BOP BOP is such a nice uh uh outcome right even it's like when you think of grades you know grades are like BOP right percentages they're they're an outcome and um and then the trick is like if you're trying to do an educational study and increase a score you want to get a lot of people stupid a baseline right so you have room to make them go up I always say that that you don't want a ceiling effect if you get all these A plus students in your educational study it doesn't matter how good your education is you won't see anything because they're already they already know it or whatever so that's why like your measurement is so important in a study and a lot of biostatisticians are men and men tend to get out of doing data entry like they don't have to do it or they don't want to do it or somehow or another most data are probably entered by women because most of the people in the healthcare system at the lowest levels are women like 80 percent of the lower level healthcare jobs are filled with women in the U.S. those are the jobs that are doing data entry in healthcare and I'm one of them I'm a woman I always joke I say I'm a brown woman whose name ends in A you know like all of us are and we do all the standard okay so what is data entry it's measurement it's measurement of business data and we know if you do a lot of data entry you will know that there is a different amount of measurement error for each variable your entry so if somebody says tells you there how much they weigh that's going to have a lot more error than if they tell you what their date of birth is okay and these all these things are never taken into account when I read books statistical books that are written by men they leave just totally forget about differential measurement error and so why did I even spend any effort on choosing BOP or another measure because I just wanted to make sure that we were getting something that wasn't full of errors in our group in our population you know and and these kind of sensitivities to like population measurements you know study design like the exact you know doing like the parity test versus actually like I had had to power the study for a difference in differences there would have needed to be a lot more but that's not we were asking we're asking is this NSM as good as chlorhexidine you know is it just good enough right so there's really no reason to like overpower the study and if we didn't think we got enough you know we can get more like one of the things that I noticed I was the only one but I used to study Alzheimer's disease we we would try to look for things that improve cognition so that's a continuous variable going up and everybody had the same hypothesis like giving people vitamin D would improve their cognition so they power their studies and then they do it they do their studies you know placebo versus vitamin D and it never worked so they kept saying well we need bigger studies we need bigger and bigger and bigger and bigger studies and they kept doing bigger and bigger studies but it never worked right and so that is a great example of how if something doesn't work it doesn't work right it doesn't matter how much sample you get and then you end up with sort of the big data problem where if I was able to get like a hundred million people with Alzheimer's or whatever in the study taking vitamin D versus placebo yes it would be statistically significantly different but clinically it'd be meaningless right so the the difference between statistical significance and clinical significance that's often what I see that sort of gets lost if you have statisticians only involved like if you have a statistician and a clinician and no epidemiologist you end up with all kinds of goofy things and how I know is I'm often the person that they bring the data to and say can you finish the statistics I'm like well how did you power the study why is it you only got 10 people but you're not going to be I'm not going to be able to know if the answer is yes or no with 10 people like I'll just tell you that right now you know sometimes you know from having a big standard deviation like we didn't have such a big standard deviation among the gingivitis people in our pilot study but if we had you need more people you know g-power would have said it you know they would it would have taken it to account but um but yeah so that's that's the thing if you're in a population where the changes you think are going to be small you're going to need more people to show it um and that's why pilot studies are really on the on the sample or a population that you're sampling is really important for your power calculation because then you know you're doing the right thing for the people you're going to study and you know we did this we applied for grants you know if you're applying for money each participant costs money but it's kind of like half a car buying half a car here's a thousand dollars buy half a car right you can't buy half a car you know and and if you bought half a car you can't run it right so you don't want to underpower your study actually that's what pilot study means is a deliberately underpowered study because you're just trying to work something out the measurements or something so if you ever read something as a pilot study don't be upset that they don't have enough data there it was a pilot study was deliberately underpowered all right well we're towards the end of our hour together our data science chat i'm glad thank you for showing up if you showed up um i was trying to do a little better job of monitoring the chat of course we got our zoom bombers um and hopefully um that this overlay helped i got a few questions but not as many as i'd hope i'm just trying to do a better job of advertising so in case anybody has questions about sample size calculation g power and stuff they can come to me and also if you have any specific questions about g power and sample size calculation which i didn't cover because i didn't really cover much um you know let me know and and then i can always hold another live stream focusing on your specific question so um so if you showed up thank you for showing up and if you're watching this as a recording thank you for watching um i really appreciated i i'm trying to improve my subscriber list get more subscribers on youtube so if you're watching this on linkedin you know uh i'd appreciate it if you could go over to youtube and subscribe and um and and also if you're watching this on youtube you can go to linkedin and connect with me on linkedin just let me know that you're interested in my live streams in that way i'll make sure to send you an invite each time i do it well thank you very much and i hope you have a wonderful weekend


This is Dave Vellante at wikibon.org and this is theCUBE and we're back here at 590 Madison Avenue. And we've been talking today about how function, particularly storage function for the last 15 years has moved out of the server into the array, for good reason. To share, to replicate, to improve disaster recovery. But with Flash, we're starting to see function move back. As we've talked about wikibon and SiliconANGLE for quite some time, you really can't control fast data from slow storage. We're here with Colin Parris, who's the general manager of IBM's power systems business. We're going to talk about these and some other items, Colin, welcome to theCUBE. Glad to be here, Dave. Yeah, so we're hearing today about the big investments that IBM's making in Flash. And as I was saying in my introduction, we're starting to see servers and storage come together a little bit. So I want to talk about that. But first, give us an update on the power of business. I mean, it's a big business for IBM, driving a lot of transactions. What's going on in that business today? Ah, the power business is doing great. I mean, we're now at the point in which we've introduced the power seven plus, the newest generation of power, using some powerful new processors. We brought that out starting in October. We got the low end systems out in April and February. And so now we have the whole line of power in the marketplace. So it's all ready to go. Great against the competition like Sun and HP. And we've been doing fantastically well. Yeah, so a lot of databases running on power. Huge amount of databases. And Flash is the perfect complement to database. What are you seeing in terms of performance and just architectural changes that portend greater things for business value down the road? Well, I think two things matter to us here, right? So the first thing is the capability to do these online transaction processing. The OLTP, given the bandwidth that we see and the latency of Flash, we can do a lot of these huge transactions very, very quickly. So that's the first thing. The analytics that comes out of this as well is another astounding thing. What also matters to power quite a bit is the fact that we have a high level utilization. You know, normal Intel boxes run at 30%, 20% utilization. You can use power up to most clients, 60 to 90% utilization. That's great for us. Because with Flash right now, we can get the IO that allows a sustained utilization of all computing resources. So you buy less boxes and you buy power boxes. You don't want them running on these other machines that could only go up to 40%. So for us, it's the perfect combination. We get the much lower latency and we get to utilize these processes a lot more. So where are you seeing power uptake in the marketplace? Well, in terms of Flash, the two areas that come to mind right away, usually when you deal with things like any type of online trading, any type of transaction processing reservations, we see it right there. We see it in analytics, right? We've talked about people who are looking to find, for instance, intelligence agencies, once they get the data that quickly says where they use the analytics, they pull the data in, the data says, this is what's happening, now they want to run a large amount of computation on that data. We can run the computation very quickly and return a result. So it's not just getting the data off the database and doing the analysis, it's running the computation on that as equally as fast. So we're seeing a lot of people using the analytics for that reason as well. So if I think about just databases today, there's a lot of databases out there, they're relatively small and you try to minimize calls to the database because you don't want to go to the spinning disk if you don't have to and applications are somewhat limited by the amount of data that they can bring in and generally database systems are isolated from the rest of the systems in the organization. It sounds like given the comments you're making about analytics, that you see that changing, those two worlds coming together where transactional databases and analytic systems actually come together so that real-time decisions can be made. Are you seeing that today or is that more futuristic? No, we actually are seeing that today. The entire thing is to get the data as close as possible to the transactions. So you see it already in this in-memory view of the world. Now the interesting thing about power is that power, I think many people know it because of Watson, right? So we have the big, jeopardy machine. What most people know is that that was the largest in-memory database because what we had to do was we had to ensure that all of the data that was used in a jeopardy competition was on the Watson systems. It could not in any way connect to the internet. Largest in-memory database, you're pulling the data closer because you needed that response time. You needed that close speed. Flash begins to get you even closer, right? Normal disk, we're talking in terms of milliseconds. This is microseconds. Then you combine that capability with the in-memory capability. Now you have something truly unique. Everything is getting much faster, much more able to use the capabilities we have in terms of utilization. So we heard Steve Mills to say today that the discussion around in-memory has been around since memory has been around, but what hasn't been around is an extension of memory that's persistent. And we've sort of dreamed about this day for a number of years. How do you see that changing? Well, there's an old saying that we use at Wikibon. The best I.O. is no I.O. How do you see flash, that persistent storage medium, affecting memory as an extension? Do you see it going to the other side, what I would call the right side of the channel? Or do you see it as largely something that is still outside the channel and is really a storage medium? No, I quickly see it coming in. I think by and by, just the type of problems that we're going to address will require being there. If I go back 20 years ago, the discussion was all about the transaction, right? What am I doing to the transaction? Now the discussion is all about the question. How am I processing the question? So it was getting information was what you wanted out of a transaction making sure it was done. Now it's getting insights out of a situation. And the way to get those insights is to bring these things closer, to have all of your memory capabilities as close as possible. You see it even now in Hadoop systems in which you're trying to bring memory closer to the processing to do things with it, right? Now when I have flash capability and everything and in terms of the size is closer, there's some very different things I can do. And that is where I think transformation and industries begin when you can bring these things together. I'm delighted to start now because I think the people who are on it as early as possible get to transform industries. Yeah, now one of the things that IBM from my perspective has not done well is leveraged its expertise between specifically the storage group and the systems group. Now the organizational changes under Steve Mills and Rod, I think a lot of times companies, the organization is the biggest challenge. You've made those moves. Are you starting to work more closely with the storage group? Can we expect that you'll actually develop this end-to-end system? Definitely, because that's what you're seeing even now in this announcement, you notice I'm here, I'm running the Power Group. Ambudge is here, we're on storage. Pitchiano is here, we're on software. All of these pieces are coming together. And it's coming together for two reasons. One is clients demand it. So and we are clearly responsive that our client experience. The second reason it's happening is because they are now interdependencies that become very interesting. The SBC program that storage has, which gives you something known as easy tear. Now we have something that actually does easy optimization on Power as well. It begins to align the VM so that it can be as close to the cache as possible. So as you're aligning things to be close to the cache and you're aligning data to be close to memory and processors, wouldn't it make sense when those two things come together and that's exactly what's happening? You see some of the technical specialists we have here, the IBM fellows, they are working even closer together because now the fellow specialization is not about going deep, it's about going wide. So you'll notice here as well, you've got four or five fellows from storage, from Power, from software in terms of IM coming together to have these discussions. So we see this thing coming together on a fantastic rate. We've already done some things internally as well that make this happen. Also the other part that's interesting is in the go-to-market, we've begun to change things. Compensation schemes and ways that we actually allow the sellers to actually get paid for selling different things. All of that is beginning to come together as well. So I think you'll see some fairly profound changes.  Talk about that a little bit. I think it clearly has been. If you look at some of the really cutting edge things we do, it has all the pieces in there. One thing I should say though, part of us playing out is also it playing out to the clients because when you go into some of the larger clients, they're organized in silos as well. It isn't coming together there. Now that we see in certain areas like analytics and cloud drawing people together in the organizations and sometimes in our clients, it makes it easier for us to come together because before if you went to the wrong person to do a sale, everything fell apart. So the industry itself is thinking about these things differently. If you notice, I mean in power, we also have something known as System Eye. So the legacy eye systems have always been about integration. We actually use that to leverage things like Pio. Pio has storage tied into it. So a lot of the products that are coming out now also are pulling us together. So it's both the industry doing it, the clients doing it, products doing it, our line men doing it. I think it's coming together. It will have to happen. So you're seeing the clients actually break down those organizations. That's happened over what the last 12, 24 months. Is that right or is it just starting to happen? I think in the last two years, I've seen it begin to take a different turn because issues like security that are showing up, issues like analytics and cloud force you to think together. You can't do it alone. You can't do a cloud in silos or analytics or security in silos. It just doesn't work. Okay, the lines as I say are blurring. As we've said many times, you can't run fast data from slow storage. And these worlds are coming together. Colin Parris, thanks very much for coming on theCUBE. It's a pleasure meeting you. Dave, it's a pleasure. Thank you very much. All right, keep it right there. We're right back with our next guest. This is Dave Vellante. This is theCUBE live from Madison Avenue.