Alright, there's been a lot of talk about gratitude here at the Rewired Soul, so I got Rod and Vince here, and we're going to talk a little bit more about it. What is up, everybody? This is Chris from the Rewired Soul, where we talk about the problem, but focus on the solution, and I'm here with two good buddies of mine going ahead and introduce yourself real quick. I'm Rod. And I'm Vince. We're a self-deprecating romantic pop rock duo, RKVC. Yes. Ooh, and that's a leaf blower. We're a band, we're songwriters, and we suck at everything else, including taking care of ourselves. Yeah, so yeah, we're going to be talking about gratitude, and by the way, stay tuned until the end of this video, because I will be doing a giveaway. So, real quick, like, whenever Rod and I hang out, and Vince, and like, Rod always tells the story about how we met. So let him know real quick how we met. I guess the TLDR is when the whole Shane Dawson, Jake Paul thing was happening. Katie Morton was featured in those videos. A lot of you saw those know that. Actually, if you're watching this, you probably know what Katie Morton is. Yeah. And Katie is an acquaintance of ours. We see her at conventions and stuff. Something got suggested on our YouTube suggestions and things. I was like, is Katie Morton a good therapist? Like, mother... Yeah, she is, so I clicked on it and watched. It was this guy. That was in the morning, and then at nighttime, we were on our way to like, a vid summit after party that Patreon was having. And I saw him in the hallway and was like, it's gonna sound totally creepy, but are you Chris? Yeah, and I was like, oh my God, I got noticed. Yeah, and then we just started talking for like, ever. And then these guys came out to Vegas. What was it, last month? Yeah, we got to hang out and stuff. But anyways, yeah, we were sitting down and talking about gratitude. And like, we've been trying to meet up and like, do, you know, collab. I'm like, that is a great, great, great topic. So tell me, tell me again, about like, I know you got, you start writing stuff down. Vince, you getting in on it a little bit? I'm starting, I'm not starting. I did have my other things though, and I could talk about those, those ways that I express gratitude. Yes. Things I talk to myself about. Yeah, let's, let's do that. Like, because there's a billion different ways to do it. And something I tried to teach all of you was like, there's different ways to do these things. Like, I'm not a big journaler. Like, I don't have like a journal. Like, I have like 15 notebooks that I write stuff in, but then I use apps and stuff like that.  The short version of how it started was once again, Katie Morton. We were at a Patreon convention and I had asked a question at the end of her speech about like, how do I properly internalize anger? And also, how do I be more positive and not, like, not be as angry with the, not to call any specific YouTubers. But a lot of our YouTuber friends do not like their life right now and are blaming like everybody else. And I was like, I don't want to be that. So how do I not be that? So Katie was like, hey, don't internalize anger. There's no correct way of doing that. And that's a whole other thing. But the second thing that she said was, A, let your friends know that you're trying to be positive. So if it seems like you're ignoring the complaints, it's not them. It's you. Yeah, boundaries. And then the second, the other part of that was B, like she was like every morning wake up and genuinely be grateful for something. She's like, you could write it down.  So I went through this like process of like what works for me. So I knew previously from other habits, I used to run every day, which I need to get back into. Yeah, me too. Social media keeps me accountable, particularly Granny Pottymouth. So like if I post on Twitter, every day I'm going to be grateful for something. If I miss one day. Yeah. Granny lets you know. Yeah, you better. And the rest of the internet as well. Especially Granny. So I was like for the first 30 days, I'm going to post hashtag RK is grateful for. And it's something I'm genuinely grateful for. And not just making up or whatever. I'm like, this is something worth waking up for. Yeah. Sometimes it could be like a big thing, like we accomplished something huge or something in our careers went really well. Or Katie said, don't be afraid to be grateful for small things. You know, like I can see today. Or I, you know, I woke up without some sort of weird joint pain that's inexplicable in my 30s. Or not as much joint pain. I don't know, like not sarcastic and not making it a chore, but like genuinely being grateful for it. So you're specifically using like social media making it public? That was the first 30 days for me to get into the habit of it. Because I think after like a couple of few weeks, it's just like involuntary. Like it just kind of happens, you know? Yeah. So after that 30 days, I was like, I don't want to like, or bother everybody on the internet with it unless you want to follow this. And so at that point I started using like notes on my phone to keep track. For me, that didn't register as well. I didn't retain it as well. I'm not saying that that won't work for other people. Yeah. So at the start of the new year, I literally just took a actual notebook. The Patreon gave me. And it's a blank notebook. And then just start like 1.1. You know, like I am grateful for. And then right now. And I think the important takeaway from that too is to be mindful of what's working for you and what isn't. Like some people like try and try and try. Like I'm huge on meditation and stuff like that. But if like formal meditation, like sitting down, focusing on it but isn't working, figure out a different form, you know? So like if you start doing gratitude lists in your phone and it's not registering as much, try something else. What about you, Vince? What you doing? It's a perfect transition, this guy. Really, it all revolves around I wear my emotions and I project anger a lot. So I had to learn what things that triggered that and start to develop these ways to where when I'm in those moods I can like, it'll shoot me back to this one thing that I've continued to do every single day. So when I get up in the morning, that's what kind of determines what mood I'll be in. So when I get in the shower, when I feel the water hitting me and like running down my body, I say out loud and I say to myself like, let all my anger wash away, let all whatever bad feelings wash away. And then while I'm, when I get to a peaceful point, that's when I start saying like I'm grateful for being in California, I'm grateful for my friends, I'm grateful for this, I'm grateful for that. So that later when I'm in like an angry fit, hopefully it shoots me back to the shower but saying those things, you know? Those mornings when we run out of hot water I know to like leave them alone. Like, oh, today was a day. So it's like two things. It's one thing is like doing it something repetitiously, so that later when I find myself in these ungrateful or like mad or angry fits that instead of projecting out bad things I can like shoot myself back to that mantra and like continue to say like I'm grateful for this like wash away this anger try to turn the switch, you know? Because it's hard, it's tough. Yeah, none of this stuff is like and it's a habit that we develop. And so two things from that. For example, I was just using the example of meditation. Not a lot of people are into that like formal like focus on the breath. But like that, I do that in the shower too. It's like sensation mindfulness, right? You're being aware of what the water feels like.  Yeah, so like that and that's all mindfulness is being aware of the present moment. But like I can definitely relate because one of the reasons I wrote Rewire Your Anger was because I can get angry too and gratitude is like this antidote for anger. So like less than six years ago, like when I got sober I had no driver's license.  And like so now like because you know all the things that have come my way and the work I've put in, I have a car and everything. So I'll be like stuck in traffic right?  You know? I'm grateful that I at least have a car that was able to bring me to Los Angeles to hang out with my friends and do videos like this. So like gratitude is a great way to get rid of anger as well. Yeah.  Yeah. And as long as you can stay in that positive thought cycle I mean you'll just go back down into a negative but you at least you can figure out what the trigger is to get you back into a positive thought cycle. Yeah and like to dive deeper into that like with anger or even like depression and like negative thinking like I don't know if you guys ever get into like those negative thoughts like oh my career isn't going anywhere or my life sucks or whatever it is. Do you find that gratitude can help you bounce out of that like when you're like when you're just like sitting in that pity party like well nothing's working out. That does happen for me I'm a little bit of an odd duck like it's very preventative for me. That's what I found recently hopefully like a year from now I'll be saying the same thing but like I find myself getting to that cycle less that negative cycle less so like we were talking about this when we were visiting in Vegas that once you know back a few months ago the Aquaman movie had its premiere and we were there and normally when us peasants go to these premieres they just kind of put us to the side and like you get to be here in Instagram a little bit but this time they walked us across the or it wasn't a red carpet it was a blue carpet cause Aquaman they marched us across and then they put us like right where like all the interviews were happening with Jason Momoa and Dolph Lundgren and Nicole Kidman stuff and so we were standing there and granted it was kind of a long night but long was like a couple hours and I don't whenever I tell the story it sounds like very like you know mightier than that or whatever but it's not meant to be that way. It's just some observation. Yeah. I would say there I was not the first time but the first time in a long time I was genuinely like just in awe of everything so many good things were coming at me that perspective that I was thoroughly enjoying myself I was like I'm at a movie premiere for something that like a bunch of my friends are trying to get into that me four years ago watched TV and was like how do I be there you know and to just be there and see a movie that's going to it ended up being like the best like selling movie I don't want to say best movie but best selling movie all year and it was it was an incredible I don't know I could name a thousand things I was grateful for that moment but everybody not everybody a lot of people around like my immediate circumstance like my feet hurt it's cold outside blah blah he came to a point after a couple hours of that that like I started just to feel sorry for everybody else is like they're not getting what like they're not seeing yeah I'm seeing perception is always fascinating to me like it's something I talk about on my channel all the time two people can be in the exact same situation one person's hating it one person's loving it and it's like how is that and it's the way that we work on our mind and how we process things and all that there's a meme that always comes to mind I've never I've never forgotten it that I saw like years ago of like two people on a bus like driving around a mountain and one of them like looking into this beautiful valley the other one's like looking at the other side it's just like a rock wall they're like this trip is awful and the other one's just like enjoying all he has to do is turn around yeah and like this is why like I try to you know implement like scientific studies maybe I'll link one in the description but like practicing these things actually help the video I actually did with my mom over on her channel I'll link that because she links some studies on that so let me ask you this like being in you know lovely Los Angeles and like to kind of piggyback off that story about the red carpet do you guys notice that like do you when you look and observe people like do you think a lack of gratitude is leading to some of their I don't know not being fully satisfied with where they're at or what they do have oh yeah absolutely and like in like what way like what situations a little bit in the Hollywood not to name names we don't spill tea here we can start that channel a little bit in the Hollywood world like we have like a toe in in the film world and then our main thing is music but like in the YouTube world specifically once again without calling out names there are a lot of people that are super negative and not that I'm trying to villainize them like sometimes it's very like justified you know frustration yeah but it doesn't help to just be angry staying in the problem not getting in the solution yeah yeah so like you want to be a friend you know you're like there listening and you're trying to give constructive criticism of what's happening or whatever but like you could tell that whatever decisions that person makes after I can't just like it's awesome it's going to be affected by the thoughts of you know that negativity is going to branch out to every decision you make it and if you stay in that then it's not going to work out but there was a number of occasions but I'm specifically thinking of like one conversation we have some friends that are on YouTube that like were like well YouTube screwing us over yeah and so I was like okay well you know maybe it's kind of like when you lose your job or you know that situation when you're at like a fork on the road we're like well I can you know I give choices now it might not be voluntary but like maybe this is the opportunity if you were ever afraid to do something on your channel to break it if you know if YouTube broke it or whatever maybe this is your chance to try that thing and they're like well I don't want to like shake things up too much because I want to break it and like okay then I guess stay the same like hold your ground and come back around but we have to change something because YouTube broke it like I don't know what to tell you other than like just stop I'm not sure like yeah and I yeah that's a that's a whole other video about like stubbornness and like you know that definition of insanity keep doing the same thing expecting different results but yeah my my girlfriend is a great job keeping me grateful too because sometimes I get caught up in like the numbers right or if it's a video especially like what I notice about myself is like it's something I'm really proud of and you guys probably feel this way too as musicians you know creators like you put so much in you just want people to see it and my brain can say only a hundred people or only 500 people saw this then my girlfriend you know that that perception thing she's like Chris 500 people might have gotten a message that helped them out and I'm like damn you're right you guys ever like point of view perspective and so we haven't talked about it on our channel actually at all I don't think but one of the things that YouTube did for us is a kind of like replaced touring because we toured our asses off when we were in our 20s but our 30s were like you know what this is a bit much but I think last year we did like half a million views on our channel which for most of our YouTube friends are like they literally do that and like before we had lunch I need to become one of those YouTube friends but for us once again being grateful we're like one of our friends who's like OG YouTuber like retired from YouTube like great was like but what you guys do is great like a hundred or like a thousand people watching your videos like imagine all those people in a club at one of your gigs that's a lot of freaking people you know it's like you're right like even if we have like only less than a hundred people watching a video like yeah that's a good thing for us in a club yeah so half a million eyes on our stuff we couldn't have toured every crappy bar in the country and gotten that many eyes on this yeah you know yeah absolutely and yeah like I think we're gonna wrap this thing up so let me and by the way we're doing more videos together so same to you um we're about to do a video over on their channel so that'll be down in the link in the description they have some uh some music that we're gonna dive in some lyrics mental health related but you gotta go check it out to find out but head over to instagram at the rewired soul I am giving away three now one not two but three gratitude journals to celebrate some gratitude so make sure you're following me over on instagram so you can enter in that giveaway and where can these beautiful rewired soldiers find you guys um for our social side it's uh oh wait I got the backwards for our music side it's a spotify apple music all that stuff um our social side is our youtube is probably our biggest thing and then um all the stuff like our kvc views our kvc insta our kvc tweets yeah we have a lot of weird usernames but if you search rkvc yeah many of those platforms you'll find us yeah and they'll be linked down below too all right anyways thanks everybody thank you thank you guys and let's turn this off so we can record some more videos yeah


Hi everyone, Basel Chapman sitting here for Tom O'Brien. I'm the host of the Tiger Technicians Hour, 10 a.m. to 11 a.m., Mark Adays, that's Eastern Time of course, and the author of the opening call, Dating Newsletter. Now let's look at some of the patterns that I work on all the time. Let me just show you this quickly. In the Chapman wave, we try to identify the lowest low bar, we count each excessively higher peak, alphabetize them sequentially, A, B, C, D, Ds where other things can happen, that fourth highest peak, but it can still go to E, F and G, there's never an H. The other thing we look at here is the patterns, only three patterns, straight up, straight down, that's a straight line, cup formation, it could be a V-shaped formation, but it's a cup formation basically, and an arch formation, a mix of one and two, or one and three. This is one and three where the H goes to the left side low, if it goes there after just one peak, like a peak A or a peak B, and then fails, it usually goes below it, and you've got to be careful, and on the right side, straight up means if it makes this cup formation, takes out that left side low, how it does that is really important. So that's the core to the Chapman wave, I use many other things as well. And so what we're looking at here is the Dow trading at 233. You see this, let me just widen this quickly. First of all, I'll show you the pattern in the large form, because this is what the Dow has been the leader, and we've been liking it for subscribers to open and call. I've liked it for a long time, we're actually still long from the low of March 23rd of 2020. But within that context, we've taken some profits, and we also have had short-term longs and shorts, but now what we're looking at, we have a rather aggressive long position, and what we're looking at in terms of the pattern is you see this cup formation, and you see this little H pattern here that just took out the left side low of 28,715. On the 13th of October, we went to 28,660 within 50 points. And the MACD, the Moving Average Convergence Divergence, was moving higher. All of you have it on your, if you do any trading through online, I'm sure your trading platform has the MACD, the Moving Average Convergence Divergence. And then there's the slow stochastic, I use the slow, not the fast one. The slow one gives you a longer time period, and it's holding well, but it has gone under 80%. I like it over 80%. And the blue line you see here in the steady shot is the on-balance volume. It's not that strong. And the relative strength, the RSI is actually holding very nicely. But the key action that we look at here is when the nine goes over the 14, you can see what happened here. This green, actually talking about a green line, you see this green line over the black line, and it's still holding extremely well. And that says that even with pullbacks, you'd have to get a move down to under 31,000 in the Dow to see that nine-period move underneath the 14-period move, which would be very negative. So, so far, the Dow is the leader. Dow 30 is not the Dow industrials. I don't know why they call it the industrials. For ages, we've been talking about it as really the Dow 30. Then there's another technique that I'll look at here. Here it is. See how I joined the upper edges of the wicks in the candles? So, 36,952 on the week of the 5th of January. That was the high. Keeps coming down, makes lower highs. And I have a pattern that I take one trend line, and then I see if I can make a parallel line, a very thin parallel line that says this should be the resistance.  When the price goes above it, or in this case, it was the propellent zone. And when the price goes and tests it and then successfully pushes away, it's like a magnet when it attracts the price, and then it reverses and propels the price. Well, the Dow is in the inside track as we're closing out with 45 or so minutes to go for this week, Friday. And we'll see where this closes. But it went into touch that edge in a peak B. All the others have failed at a peak B minus. B minus, peak A failed right there. 36,824 back in the early part of 2022. So this is a really important moment for me because in this particular pattern, if there is a close, a decisive close above this arch formation in this weekly chart, and that says if the Dow can get, I'll give you the exact price right now. Used to have a 34,561, the high of the 19th of August, the week of the 19th, if it can start to close for two out of three weeks above that, we have got a trend change in the weekly chart that is very significant. Why? Because the magnitude should get even stronger. The stochastic, which is only at 43%, that's not very good. And the on balance volume still needs a lot of work. But that nine period for the first time in a while will cross positive. And if it can hold positive, that's going to be a really strong support. I said, talking about these moving average, I like to do things live so that you can see this. I did it this morning, I'll do it again now. But I didn't show you something that I've just done. I wonder if I've got it here, yes. So I was asked about the YM, this is the 10-minute E-mini chart. I actually have, this is still a leg A, it broke above the 50% level from the low of the day. What I wanted to show you was ESZ22, ESZ22, there it is. Look at the strong, powerful move. So there's a pattern called the Eiffel Tower. It happens very often after the Fed makes an announcement. Or there's just something that jobs report, whatever it is. And the market sharply spikes to the upside or the downside, but in this case, the upside, and then comes back down. Well, what happens sometimes is it goes straight up, this went to an A and then pulled back and then went to a B and then failed. But I typed in Eiffel Tower A pattern. That means that A says you can come down and be careful of the key support levels. Well, what do we do? We made over a period of time, we've made, look at this. He has a two-minute chart.  I talk about bar symmetry. Look at the bar symmetry here, and look at the bar symmetry I did in another timeframe where from that low to that low, look at the arch formation. It just missed matching exactly. I actually, I did it a little differently. Let's see if I can do this again. I did it in a different timeframe. I just don't wanna go there right now to find it. So let's just go right there. And what I do is you don't have to have these rectangles. You can just have a straight line or you can just do it. You can count the bars. I just do this when I demonstrated live, you can see it and look at that bar symmetry. Is that not unbelievable from the low that was made in this two-minute chart at 12.06 this afternoon, at 37.16, we read it, made an arch formation, came down and at 13.10, that's one 10 this afternoon, it goes to where? 37.17, slightly high. And then what does it do? Two bars early, it makes a low. And what is that low? Right there at 37.15.25. And now it's gone. Peak ABCD, there should have been an instant restart but there isn't. So I continue the count EFG slash C and here you've got D and we're in leg E and I had drawn this, I don't know where I did it, it doesn't matter. All right, I'll do this and then we've got a lecture discussed, Basil Chapman sitting in for the one and only Tom O'Brien and I'll be back in a moment, the dollars at 255.


Good afternoon everyone can everyone here everyone the back. All right, cool. Let's just do a quick level set I want to get an idea of what everyone's skill set is so who's familiar with pandas just quick. Oh, wow everybody Okay, so I can't learn How about machine learning fundamentals like gradient descent error metrics things like all right cool All right, so good shape. All right So yes, I will be talking to you take about rapid prototyping and data science with big data and Python So a little bit about me and the company I work for I am a real live data that sorry senior data scientist I'm an instructor at Metis as well. So we have a couple of different offerings. All we do is data science training So we offer a sorry full-time boot camp. It's a three month boot camp, and that's what I teach We have evening courses for people who want to learn a specific skill or just want to dabble aren't ready for the Full thing we have online courses same kind of setup, and we do corporate training as well. So number of offerings All right, so a couple key assumptions about this talk again ideally you have some sort of machine learning experience and Familiar familiarity specifically with gradient descent if you don't that's okay You'll still get a lot out of this talk, but there might be some Aspects of this pipeline that I talked about that might seem like magic, but don't worry. It's just math All right, so I am going to assume that you're familiar with sci-kit and pandas So when I get to my code, you'll see that this looks exactly like the stuff that you've seen and modeled with and We're going to assume that all prototyping from beginning end is done with this thing called the pi data stack So this is your typical num pi pandas sci-pi sci-kit learn All right, so I want to parse this title up a little bit Let's talk about this thing called rapid prototyping. So basically the process always starts the same way I have some sort of problem. I'm trying to solve I generate some ideas and then you know, we're all here We all need to build some sort of prototype some sort of software to answer that question well, this can be very difficult in data science because You know, we don't always know the right solution. So it's sort of this iterative process So we build our first prototype we create some sort of metrics or constraints to test it with And then we analyze those metrics and say did we answer the problem or did we not? And then usually that leads to some sort of refinement. So again rarely do we get that on the first go This leads to new ideas. We build another prototype and we do this as rapidly as we can We don't want anything to blog down our process All right, so this idea of data science. How do we apply it to this rapid prototyping scheme? First question is always, you know, what is the problem in this case? It might be we are some sort of online advertiser and up to this point We've been showing everyone the same ad well We have this hypothesis that if we can segment markets and we show them a targeted ad we'll get a better response rate So what we do is we collect some data. Maybe what age group are these people? What are their interests? Do they cars or they into I don't know outdoor adventures and then location? So geographical location. Are you in the Midwest? Are you in Ohio? Are you in California? And then we use that to try and separate these different groups So you can do that number of different ways actually you can use a supervised approach like classification or you can use an unsupervised like clustering So pretend that we had some sort of oracle that let us know that there were actually two groups And we tried one of these out say classification and we we got two groups So we're on the right track, but we had misclassified one of these red X's Well, we would reiterate and hopefully by making some adjustments capture the correct groups or at least a better version So again, it's iterative process rapid prototyping with regards to data science All right, so now let's talk about the slip return big data All right, so a couple common definitions Gartner basically if I distill this down for you They're talking about high volume and or high velocity and or a high variety information So large coming in fast and then either coming in from disparate data sources or some sort of nested data Wikipedia a little bit more simply just says data. That's so large or complex that traditional techniques completely break down So I'm gonna talk about these guys, which are synonyms here large data and then a solution I'm gonna show you at the end is gonna be able to Allow you to handle high velocity data. What am I not gonna talk about this idea of complex data? That's a completely separate issue. I'm not gonna discuss that today So just to wrap that up talk about data. That's large in size. Yes Coming in fast not complex All right, so we have this pyramid here, right, you know data at different scales So the question is fundamentally, where do we draw the line? What is big data? It seems like this changes every day. So do we draw it here when we get into mega or gigabytes and above? Do we draw it somewhere here here or beyond? So, why don't we take a quick poll? Who thinks it's gigabytes and beyond raise your hand? One two, okay two souls. How about what's that? I ask you though. Is that true? So gigabytes plus we had two brave souls. How about terabytes and above? more people and then petabytes and above What if I told you you're all right and wrong It's a trick question. Okay, so when it comes down to let me define it for you It's data that is too big to fit in RAM. That's fundamentally what big data is Oops wrong way, so let me give you an example So it could be at the kilobyte scale if you're using Arduino this particular Arduino, you know It's only got 256 kilobytes of memory My laptop here 16 gigs. I'm at the gigabyte scale. That is a big data problem So it really all depends But at this point you're probably asking yourself. Why does it have to fit in RAM in the first place? And he actually anybody now Before I get there It's half the equation All right, so let's talk about it So if I was looking purely at speed if you look at your memory hierarchy here You know I go from hard drive or solid state. I move up a little bit in speed I've got RAM cache CPU registers if I wanted purely speed. Where would I be? Yes, so that's not the answer So if we take a look at this in a different way and we look at capacity You know how much data can I hold if you look at the hard drive and solid state? This is where you're going to have maximum capacity, right? RAM a little bit less. So maybe on the gigabyte scale typically Cache you're looking at maybe 12 megabytes and then CPU registers like half a megabyte typically from an order of magnitude perspective But again if you guys are right if I look at speed It's the complete opposite relationship CPU registers wicked fast and then I go down the pyramid here But I also pick up cost. So I can't just have an array of CPU registers. That's not going to be cost effective So to capture this in a really simplistic diagram what I can say is cache and registers have great speed awful capacity On disk awesome capacity, but no speed So RAM this nice balance of capacity and speed and that's why this is the sweet spot. This is where we want to be All right, so a couple common solutions on how to address this problem of big data Number one we can just probably the most obvious We can just sample our data if I can't ingest the whole thing. Let me ingest a subset a couple issues though, right? You know if I'm going to sample, how do I know that my sample is representative? Give me a giving example here. So we've got some sort of cubic function, right? This is reality, but if we take a sample I just randomly sampled maybe it's somewhere in the middle could be somewhere else It looks linear. So if I build a model on that, I'm going to be way off What happens is my data continues to grow well, you know, maybe early on when I don't have a lot of data Okay, you know it might pick up the trends It might do a good job capturing the signal But over time you can see that sample because my memory is not increased my RAM is not increasing I'm capturing a smaller and smaller percentage of the overall data. So am I really capturing trends or or Signal in that data probably not and it gets worse over time And then three how do you capture? How do you quickly capture trends that change over time? So sort of a simplistic view of this would be you know pretend that we're pushing out some sort of product It's new to the market generates a lot of interest. We see a spike in sales Maybe it's been out for a while it sort of tails off and then maybe we introduce a new feature or a new model And it jumps up again and actually will actually a better example is think of stock prices So this happens very very quickly. We have massive changes very quickly. So, you know, how do we capture that with sampling? Probably not going to do that effectively So summary of the sampling here, you know, you can use sampling to get data to fit into RAM The pros are I can use my pie data stack. So thumbs up there I don't have to recode anything. So yes, I've got this rapid prototyping happening But everything comes with a but right cons. You have to make some extremely strong assumptions that really usually don't hold one that your Your sample is representative of the population To you're somehow capturing signal in this mountain of data using a tiny little piece of it three Your population Is not changing over time. So think stock prices again Another very common approach is to completely move away from the pie data stack or at least portions of it and use a patchy spark So a patchy spark and easily be its own talk I'm not going to go into a lot of detail for our purposes All you need to know is that it takes the data That's too big to fit into RAM and breaks it into chunks So each one of those chunks can be fed into memory Processed then pushed out and probably process the next one and so on so we get to use all the data. That's great Summary, so what are the pros here? You can continue using Python Spark is built in Scala. So it's not native to Python But we can still use Python via the PySpark API We can use all the data for modeling. So this addresses some of the issues that we saw with sampling. So that's awesome. That's great What are the cons? You have to say goodbye to scikit-learn You have to completely change your machine learning library So if you've done a lot of work Maybe you've put in a month or even a couple of weeks of code and work and you've spun up a number of Jupyter notebooks And you feel comfortable with your models Too bad you had to start all over you have to switch from say scikit-learn to spark MLib And it's not a one-for-one translation So there's a lot of scikit-learn algorithms not included in MLib MLib is pretty feature rich But not to the extent that scikit-learn is and Of course, you know you have this learning curve So if you're in the middle of your project, you're coming up to a deadline I need to push a model out into production. Oh wait time out guys I need to revamp for the next two months and redo everything. I just did not so great to tell your manager, right? All right, so a big problem here it addresses all the issues that we saw with sampling, but it kills our rapid prototyping process All right, so here's the big question, right? What if there was a way to take our Pi data stack and Somehow get the same capabilities we saw with Apache spark. That'd be awesome. Who wouldn't be for that? Guess what? There's a way. That's why I'm here today So enter desk in combination with scikit-learn So these are both Python tools are both native to Python They both work really nicely together and they provide some really cool capabilities Only some of which I'm going to talk about today But if you have never seen desk before if it's new to you You should definitely go home immediately and read about it because it's a really powerful tool All right, so I'm going to give you a high-level overview of what's going to happen So imagine this rectangular box here is the Pi data stack So again, I'm going to start out with data. It's too big for RAM I need to split it up into these chunks So each one of those chunks can fit in a RAM and then I'm going to push it into a model So what's happening on the left hand side here all of this is being controlled by desk So desk is going to split the data into chunks It's going to control the flow of these chunks in and out of our modeling piece And of course the modeling all the machine learning heavy lifting is being done by scikit-learn So I can actually put these two pieces together and they can work as one All right, so a little bit more detail about what's happening. It's gave you a high-level overview Let's go down a level So we split our data just like we did before We instantiate our model, you know, this is going to be some sort of Stochastic gradient descent regress or a classifier more on that later I'm going to ask is going to push the first chunk into the model The model is going to update and see how we're going to be a better model So I'm feeding it training data And if we were to zoom in on this particular piece, we can see what's happening So if I have training data Because I'm doing this stochastically what I'm doing is I'm feeding in one observation at a time So the whole point of this if you're familiar with gradient descent instead of doing batch Where I feed in the whole partition and then update the weights once I can do this very quickly I can update the weights after every observation so I converge much more quickly onto what the weights actually should be So I pushed the first observation into the model model updates by using the gradient It adjusts the weights and then it gets pushed back out and then the second observation comes into the model We make updates because of our gradient and then it pushes back out And we continue that process until we've exhausted all of the observations Now I should probably mention that this is called if you're familiar with online learning or stochastic gradient descent This is a single pass this only happens one time You could definitely have this happen multiple times What you would want to do is just shuffle all these you wouldn't want to send them in in the exact same way that you Did before otherwise you run into some some localization issues All right, so we've read through chunk number one We've updated our model desk pulls that chunk out feeds in chunk number two the exact same process happens So our model updates and then chunk number two is taken back Chunk number three goes in the model updates and so on All right, so the big question how do we actually implement this So you've seen theoretically how it works So here we go For the pandas users out there the beautiful thing is that desk is actually built on top of pandas So all the calls are virtually exactly the same all I do is I call desk dot data frame and then you know here I'm reading in with HDF 5 You don't have to use HDF 5 I happen to like it because of its on-the-fly Compression capabilities, you know specifically with Blask if you're familiar with with compression But you can use CSVs you can use parquet file formats sequel table or be calls Like I mentioned earlier, you know here we've got on the top we have desk on the bottom We have scikit-learn, so we've got SGD classifier SGD regressor. These are going to be what are called online learning algorithms The model is a number of parameters There's way more than what I have listed here But these are some of the the key ones that you should be aware of of course We have to provide some sort of loss function some sort of way for the model to know is it moving in the right direction? You know, what is it that we're chasing? Penalty is regularization whether you want it or not in which flavor the learning rates This is think of step size for those of you who are familiar with gradient descent Power T. I should probably explain this to you. This might be new to some of you think of power T as How you learn for example if power T was one this would be this would work great on an iid data set In other words if you were playing a game like checkers where the rules never change This works really well. You'd want a power T of one so very quickly you can figure out the rules and then I can essentially stop learning Power T of zero would be I'm playing against an adversary. That's constantly changing the rules So I have to be very adaptive to what what's happening in the world power T of point five is a great balance because you don't want to be overly eager from a Modeling perspective and chase noise, but you still want to be adaptive to changes and trends All right, so there's a number of loss functions I haven't listed them all here for you, but some of the key ones say you want to do classification You can do on the fly linear SVM. You can do logistic regression So you can use hinge squared hinge log loss Of course if you want to do regression there's good old OLS. You can use Hubert loss This is a sort of a robust version of mean squared error It's much better about not chasing outliers And then of course Epsilon and sense if you want to support vector machine regression So like I said, you know reference the docs for there's even more functions in this But these are the key ones you should be aware of Some regularization options. Obviously if you don't want regularization, you can just set that to none L1 if you want Typically it leads to sparse models L2 or elastic net elastic net is just some combination of L1 and L2 some proportionality Which I can set with this L1 ratio And then alphas my regularization strength. All right, so we've talked about a lot of Concepts at this point, but let's motivate with a real example. I'll show you real code and output So has anybody worked with the Higgs data set before just out of curiosity Nobody Okay Who's familiar with the UCI machine learning repo? Okay, just a few hands Well, this is an excellent resource if you want to find some curated data sets kind of like Kaggle But just you know, I want to try out some classification algorithms. I want to try some clustering There's a whole host of of data sets for for the taking that you can work on So I chose this one called Higgs. It's pretty famous a little bit about the details So it's got 28 features the first 21 are just sensor measurements And then the remaining seven are just Think feature engineering derivative features that are helped are designed to help you discriminate between the two classes And this is a binary classification problem just saying okay, you know zero would be the sensors are just picking up noise One is something interesting actually happened So for the purposes of this talk just to keep it brief I'm not going to go through the entire data science pipeline show you EDA and splitting into training and test sets just know that the the data has been split already and The training set has 8.8 million examples the test that has 2.2 million just for reference All right, so let's go ahead and get our data So we import desk data frame and then we just pull out our training test set You may have noticed here that I have the exact same file Higgs data. H5 What's cool about HDF 5 you haven't used it before it kind of you call it kind of like using a dictionary Where you provide a key and I can have data sets all within one master data set. It's really cool All right, so I need a couple other key libraries So I am because I'm using gradient descent. I need to scale my features. I need to be able to do that on the fly I import standard scaler. So this is mean of zero unit standard deviation. I of course Pull in my SGD classifier and then I'm going to be looking at the log loss on the test set more on that to come But I need to import that as well All right, so I need to instantiate my standard scalar object I create an empty list for log loss again This is just going to tell me at the end of a training and when I send in a chunk I update my model. I'm going to see how that model predicts on the test set over time just to see what's happening So we'll actually see a nice smooth curve counter zero and Total partitions This just says okay when when you tell desk, okay I've got this data set or multiple data sets that are too big to fit in and to ram it'll split it up It'll parse it for you into what are called partitions or chunks This is just capturing this trained at n partitions is just desk telling you there's 10 or there's 15 or there's nine So essentially I'm going to use this to just keep track of you know What partition number is it and to show me my progress over time? All right, so we have to instantiate our model for this particular one This is logistic regression. You can tell by the loss being log. I am using elastic net It's my regularization with an L1 ratio perfect split An alpha of point oh one, you know, you can you can tweak all these things But I set the random state to 42 anybody know why I set the random state Looking at you Ali That that's actually a good answer That's I should probably leave it on that one But okay Right Yeah, reproducibility so you know if I run this on my machine and then Ali runs it on his machine We'll get the exact same answer so you can imagine you know there might be a little bit of shuffling It happens, so we'll get very similar answers, but they might be just a little bit different if we don't set the seed I like that though All right, so let's actually go through Running through the different partitions now. So simple for loop using range on the number of partitions All right So this data set has the features and the target all as part of one data set I need to split those for those of you who are familiar with scikit-learn, which is pretty much everybody And then I need to standardize so I need to use this partial fit function. Does anyone use this before? No, so it's like fit except that I can do it iteratively So partial fit not all of the algorithms use partial fits a small subset, but you can with Standardization so I fit on the training set a little aside here So a lot of people make this mistake where you take the data set at the beginning and you standardize all the features And then you split into training tests. Don't do that. That's awful practice. What happens is you have information leakage You really should split first Train on or fit on your training set and then use that to transform your test set. That's the proper way to do it So here I'm fitting on the training set and Then let me explain this. There's a little bit going on here. So notice that I Have to provide, you know, what are the what's the ground truth? You know, what is the actual targets and then because I'm doing this on the fly I have to tell the algorithm up front how many classes do I have so you can imagine if I didn't do this See my first partition had two of the classes But they were really three and then it makes you know it updates and then the next partition comes in all of a sudden it has three it's gonna break So you just have to let it know up front how many how many classes So I'm fitting a model on the transform the standardized training set I'm also providing at the ground truth and letting it know how many classes to look for so I've made an update on the first partition Then what I'm gonna do here is I'm just gonna use predict Prabha on the Transform test set to give me back probabilities. So logistic regression is gonna say there's say a 70% chance that this is class One versus 30% chance. It's class zero So that's an action. It's actually a pretty cool thing a big differentiator between logistic regression and SVMs SVMs, you just get class back. You don't get probabilities. So something to keep in mind Is anyone use log loss before I guess maybe I shouldn't gloss over that So, okay, so maybe I explain log loss to you a little bit So you're all familiar with accuracy, you know, like did I get the class right or wrong? This log loss is a better measurement. Your algorithm has to output probabilities, but essentially what it says is how Your algorithm is gonna say the probability of this class. It's gonna look at is it correct Is it, you know skewing in the right direction and how confident is your model? And it's going to penalize it depending on how it does. So if it's very confidence the wrong class it's gonna get hammered and Then I just append that to my log loss list So of course, you know while I'm while I'm training this I want to see some printout tell me how I'm doing on the test log loss And then what partition number out of the total are we at right now and then of course I increment my counter All right, so here are the results. This is what it's gonna spit out. So this is epic one I'm not gonna explain all of these terms to you, but I'll hit on most of them This t value over here is the number of partition or sorry the number of observations in this partition So remember the training set at 8.8 million observations here. It's brought in a million This is the average log loss on the on the training set here You can see the training time was about half a second and The test log loss. This is this crucial to keep in mind is about 1.75 which is pretty awful So it's the first partition not surprising that it hasn't done a very good job yet Now I bring in the next partition So the first one is the readout we just saw the bottom ones what we want to focus on here You can see that the test log loss is dropped from 1.75 to 1.14 and you can see I have partition 1 of 8 So I'm moving in the right direction So I keep doing this so now I'm on the third partition and I've dropped below 1 on the test log loss So my model is getting better and better over time So I'll skip to the punch line here. I won't take you through each one So we get to the 8th partition here and you can see note the t value here So the t value is 800,000 So you can see how desks split that up The test log loss point seven so we dropped a whole point now. We're getting into a much better range and This whole thing the whole training time took just over a minute On an 8.8 million examples not too shabby What's that like the the data set so this data sets not particularly big it's about a two gigabyte data set But this whole process can scale to whatever you need. This is on 28 features, right? Yeah, 8.8 million observations So no data science presentation would be good without at least one graph So on the bottom here, let me orient you the x-axis here is the partition number So when the model read in that data it trained and then how did it do from a test set perspective looking at log loss You can see no surprise at the beginning. It does a pretty bad job over time Everything's moving in the right direction and we actually start to sort of converge asymptotically towards the 8th partition already All right, so let's recap what we've talked about here So we split or in other words chunk the data so we can consume all of it. We did that with desk We built a model by ingesting one observation at a time so stochastically with scikit-learn and then Really the crucial piece from a rapid prototyping perspective all the code even if we didn't start out using this Maybe we tried random for us and a couple other methods We can we don't need to completely change all of the coding that we've done We essentially just swap out certain pieces. We're still in the pie data stack So just minor tweaks and we did that with a combination of desk and scikit-learn So in other words if I combine these two Rapid prototyping continue check. We're in good shape All right, so I've gone through a lot of material here today I just want to step back and just say okay. Where have we come from and where are we now? So we started out with this idea that we want to follow this rapid prototyping process with data science because Solving problems and data science is hard. It's almost impossible to get it right the first go It's a series of trial and error, you know There are stock algorithms stock approaches that we take and sometimes those work and sometimes they don't but it's this rapid iterative process that gets us towards an answer as We saw you know from a scaling perspective Usually we start with something small maybe a small data set a couple stock algorithms build on that But ultimately we want to adjust larger and larger volumes of data and we can get bogged down So if I'm working on my laptop I get to a point where I've maxed out my RAM now What do I do? Do I pivot do I use a subset? So too common what I'm saying are suboptimal solutions is Sampling we saw the issues associated with that and then you could pivot to a patchy spark, which is an amazing framework So if you're not familiar with it and you you start scaling into data It's a great framework, but again, you know if we make the assumption that we're trying to stay within the pie data stack That pivoting is going to cause massive problems So both come with baggage And then really that here's the big key takeaway So there's this new library desk in combination with an already familiar scikit-learn gives us really powerful tools to find better solutions faster and Allows us to tackle problems that before we're just completely impossible So if you want to connect with me if you have questions You know you can find me on LinkedIn you can find me on Twitter at David's again To email me directly. I've got a bunch of stuff up on github bunch of repos I'm always adding to this So if you're interested in some code that you can rip right from today's talk check out out of core computation Just know it's a work in progress. I have a lot of other things in the work So I'm going to keep adding to it and I do have a blog I blog about all things data science machine learning If you're interested in things like online learning deep learning How to ace a data science interview check out my blog and Big shout out to Metis who gives me time to work on projects like this Supports me coming out to speak to all of you and of course Thank you all of you for being here at the end of the conference on a Sunday getting out in the afternoon and That's about it for me. Why don't we open it up for questions? dimensionality reduction So dimensionality reduction is a great tool the one problem you'll run into is if your data set to begin with It's too big to get in memory. How do you process it to even do that? So that's one issue There are other things you can do too like say you have a sparse data set maybe you're doing NLP right and you have you know Bag of words model or something where you have some columns with some ones, but it's mostly zeros There are some approaches you can use You can use like CSR as an algorithm to compress all that information So you basically just say like I'm just going to memorize where the ones are and I don't care about the zeros So I can compress a big data set into a small one and sometimes you can use that to actually get your data set in the memory But as far as like once the data is in memory If you use some sort of data compression There are there are advantages to doing that usually processing time Sometimes removing noise because you know, you don't want your model chasing noise. So it's it can be a really good approach Yeah, so actually the the underlying theme between both of them is this idea of gradient descent So as you know in a neural net you push your data forward just like we saw in desk You know, we push the data in and then the model updates you have back propagation Which is just a fancy term for gradient descent. It could be stochastic or mini batch, but fundamentally It's just gradient descent So if anyone's sort of new to machine learning the best algorithm You can ever learn is gradient descent because it runs pretty much every algorithm Except for like the analytic version of linear regression Did I answer your question? It is the exact same process. Yeah, there are now there are different flavors of gradient descent Like you can have momentum and some other things, but from a principal perspective, it's the exact same thing. Yeah Yes, I'm glad you asked that question because there's a whole other side of desk I didn't touch on just for time constraints, but desk allows you to move into distributed environments It scales to hundreds of machines with thousands of cores It has this idea of a task scheduler under the hood it uses DAGs if you're familiar with DAGs So really what does it takes the same code that I create on my laptop here It allows me to scale when I'm ready to a cluster and I could take that exact same code with all my code base the Pi data stack and now I can do things that Apache Spark does Yeah, so everything's kept on disk it partitions it on disk and then it pulls into memory what it needs And then it pushes that out and then brings in the next partition So you can think of what's happening with these partitions that partition is really just a panda's data frame The desk implementation uses the vast majority of Pandas calls that you're used to like group buys and such there are some operations that are just truly hard to parallelize Those are going to be the operations that you're going to see desk doesn't have those capabilities for obvious reasons But so basically anything you do with pandas for the most part you can do with desk about Well, so the main thing you have to remember is that desk and Apache Spark were designed for two different scenarios So Apache Spark really really shines when you have massive data sets when you get into like Pedabyte level even like terabyte level like it's going to really shine just by the way it breaks down tasks It breaks them into like these big tasks whereas desk breaks everything down into like really really little tiny Tasks and it's all built natively in Python So first of all if you're not a Python developer, you don't need desk, right? But yeah, I mean There are trade-offs, of course depending on your use case Desk may not be the right thing, but if you're a Python developer and you're familiar with scikit-learn and pandas Unless you're dealing with massive data sets consider this first Yeah, so yeah, that's an area. I didn't talk about so there is there are ways to optimize Like for example, like if you use HDFS Hadoop typically what it does is it breaks it down into like 128 megabyte chunks That may or may not be useful for you desk I think I don't quote me on this But I think it breaks it down like a hundred megabyte chunks somewhere along those lines You can set that you can play with those parameters. You can repartition. You can do all these things. That's where I Don't have any good rules of thumb. That's sort of a trial-and-error process. Maybe somebody knows You know how to but that starts to move into more like the data engineering realm, which I'm already kind of pushing into the gray area here I mean the only limitation is the capacity of your disk and how quickly you need the processing to happen So again, you can move to a distributed environment to fix the the capacity issue But if you're using CPUs, you know, maybe you want to think about GPUs or something other questions Can I use SAS as a data set? Yeah, you mean Yeah, I mean how as long as you push the data out in some sort of format that desk can handle whether it's a CSV or a SQL table or something along those lines. I don't know if there's a direct connection There may be I'm not up on Docker swarm. So I don't know that I can Is it just are you just capturing like are you pushing model? What are you pushing to the different nodes? Tell about the data Yeah, I mean you could that's one way to do I don't know how that performs against a desk which does it can do the exact same thing Yeah, yeah, I don't know. I haven't run a performance test on that. That's it's an interesting concept Okay, shoot So you're talking about like one hot encoding So you're talking about going from say like a long format to a wide format using desk and on disk. Um, I Am trying I haven't done it yet. I'm trying to think if there's any reason why you wouldn't be able to do that I Don't think there'd be any reason why you couldn't do it. The only thing is you it might be a slow process No, it's a great question. These are all great questions It kind of depends how you read it in so like I used HDFI which has its own chunking operation under the hood If you read in like a CSV or you read in Parquet file or something you can actually set like the number of chunks you want or I think there might be some other parameters where you set the size that you want to pull in But yeah, you're starting to stretch into the limits of my knowledge for sure Yes Yeah, you can you there's a repartition function and you can do that It's an I believe it's an expensive operation, but it's it's definitely a capability Thanks, everybody


Hey everybody, this is Brian and this is a viewer requested video, how to set up a virtual machine for recording video tutorials. Now I use a virtual box. If you don't know what virtualization is, it's putting a computer inside of your computer. It's actually like running an entire computer as a software application, much like you would like Chrome or Microsoft Word or you know, Outlook, whatever. Basically it is that simple. You're running an entire operating system inside of your machine. There are other virtual hypervisors out there and when you hear the word hypervisor, really it's just software that runs the operating systems. For example, Proxmox and VMware. They can get quite pricey though, especially if you get into VMware. Proxmox is free, virtual box does exactly what I need it to do. I don't need like a whole network or anything like that. I just need to run one virtual machine and I need to be able to share a clipboard between my main computer and the virtual machine itself. Before we dive in, you should have a definite reason for doing this. And for me, it's very simple. I do development tutorials and I specialize in some technologies, for example, Qt and some of those settings wouldn't really mesh well with my desktop computer and I don't want to start playing around with my desktop and potentially breaking it. Now I had an old virtual machine, but the new version of the software I work with came out and it requires a newer version of Linux. So I've completely deleted my old virtual machine and we're going to start from scratch. I'm going to do this step by step. The very first step is of course you need to download things. So at a minimum, you should download three things, virtual box. So just go to virtualbox.org, click the download button, run the executable and follow the directions on the screen. It's going to be slightly different for every single operating system. So for example, on my Linux machine, I have a dev file which I can just run this package, but off Windows it would be an EXE Mac, of course it would be a DMG and you just run it for your operating system. I'm not going to cover the individual steps because it is ridiculously simple and it just walks you right through it. Anybody can do this. Also you're going to want OBS Studio. I'm going to assume if you're watching this video, you already know how to use OBS Studio and I'm using OBS Studio to record this video. Again we're not going to do a step by step exhaustive, elaborative tutorial on OBS, but just a couple little gotchas and tweaks that you need to know to record a virtual box. And lastly you're going to need, well, the operating system that you want to run as a virtual machine. This is going to be an ISO file and I've already got this downloaded. You can see I've got Linux Mint, Cinnamon, 64-bit.ISO. Key takeaway here is it should be an ISO file. Once you have those three things, basically go ahead and fire up virtual box and you need to be able to get to this screen right here where it says welcome to virtual box. Now we need to configure the host and for Windows and Mac you can probably safely skip this step, but just in case you have a problem pay attention, Linux you're definitely going to need to do this step. And what we're talking about is configuring your group memberships so you can use the shared clipboard feature. Now what is a shared clipboard? So we have a host which is your current machine and I've got this text editor on the host. We have a guest which we have not yet created that we want to copy and paste into. This is extremely cool. It's called a shared clipboard. So you can copy and paste from the host to the guest and from the guest to the host and back and forth. It makes it very cool. So what I typically do is I have this text editor off the edge of the screen on another monitor or just off to the side. I'm actually only recording a small segment of a very large monitor. I'll talk about that a little bit later in OBS Studio. But in order for any of this to work you have to have the guest additions installed which we'll cover in the next segment. But also for Linux you need to be a part of the VBox users group. So go to users and groups, select your name, click on groups and scroll all the way down to the bottom and make sure you are in the VBox users group, hit OK and then go ahead and reboot your machine or log in, log out for that to take full effect. If you don't do that, if you skip that step, you're going to try things and then nothing will work as far as shared clipboard. It's extremely frustrating and their documentation is a little lacking. Again, Windows and Mac, you may find this just works without any configuration but Linux, that's a step you're going to want to take. Let's go ahead and configure our guest operating system. To start off you need to actually click New or go to Machine New. It doesn't matter which way you get there as long as you get this screen. Now this is the operating system we're going to create. You notice how it says Windows and this might be a little confusing if you're on Windows, but yes, you can install Windows and Windows. You can install Linux, whatever you want. So the first thing you need is the type. I'm going to be installing Linux Mint, so I'm going to go down to Linux and I want a bone to 64 because under the hood, Linux Mint is actually a bone to 90 to give it a name. Once you have the name, the operating system and the specific version it next and you're given this very confusing screen. If you don't understand computers, this just looks nuts. You don't know what you're supposed to do here. It says the recommended memory size is 1024.  You want way more than that. So what is memory? Well, it determines basically how fast your computer is going to go in very, very simplistic terms. Memory is where programs run along with your CPU. The bigger this number, the faster it's going to go. So you're automatically inclined to just max it out. Don't do that. See how it turns red right here? This is the amount of physical RAM you have in your box and your number is going to be different. I have 32 gigs in this box. You don't want to do that much because you have things like, well, virtual box and Chrome and calculators and notepads and things like that running that also use RAM. So you want to find a nice balance between what you need versus what you have. So how do you know what you need? Well, consult the documentation of the operating system you're installing. Linux can run with next to nothing. That's why it's saying the recommended is 1024. If I did like Windows or something like that, it'd be a different number. If you want a custom number, simply take whatever number, multiply it by 1024 and you get that number. For example, eight gigs would be 8192. 20 would be 20480. I'm going to go with eight gigs, which is 8192. So I'm just going to type it in. And that gives me plenty of room to grow. The great thing about memory in virtual environments is you can always adjust this number later. So I'm going to hit next. We want to create a virtual hard disk. If you're not a computer savvy person, a hard disk is where it gets installed too. This is going to be a giant file out on your hardware. Says the recommended size of the hard disk is 10 gig. Now notice that is very low. 10 gigs is basically for the operating system and a few minor applications. Let's go ahead and create a disk and you're given this confusing screen. I'm going to leave it as VDI or virtual box disk image. If you plan on migrating to like a VMware or Proxmox or something else, you might need a different format, consult their documentation, but I'm only going to use virtual box. So I'm going to leave it as a VDI. And then of course, another confusing screen, dynamic versus fixed. This takes a little bit of explaining, but basically you are going to make a file and put the operating system in that file. You need to tell this how you want that file to be created. If you do fixed and you said you wanted a 10 or a 20 gig file, it's going to make that file immediately that size. That's why it's called fixed. It'll take longer to create it, but it's faster once you actually get it up and going. I'm going to do dynamically allocated. So what this means is it will create a file of zero bytes, meaning nothing in it. And then as I add things into it, the file will grow. It's a little bit slower, but it saves me some disk space. For example, if I say I want a 20 gig disk, it will allow this to grow up to 20 gigs and not further. Now you can adjust this later on. However, not every operating system is going to allow you to readjust the hard disk. Again, consult the documentation of your operating system. I know I say that a lot, but every operating system is slightly different. So I've got it dynamic, just in case, double check dynamic. Now how much space do you need? 10 gigs is recommended for my specific setup. I'm actually going to bump that up to, let's just do 25 gigs. Notice how you can put this into the terabytes, which is massive, you don't really need that. I'm going to be installing a lot of development tools, so I need that little extra space, but 25 gigs should be enough. Again, I can always attach more storage to it later. And let's go ahead and hit create. Ta-da, your virtual box is powered off, but it is ready to rock and roll. The next step is to install the guest operating system. So let's go ahead and right click this, go to settings. And these are the settings for your virtual machine. I'm not going to step through all of these, they're pretty simple, but just some main takeaways here. If you use disk encryption, it's going to slow this down drastically. And for your system, I wouldn't really tweak around with any of this too much. These are those magical steps that virtual box takes when you select your operating system and your version, it figures out what you need automatically. Display, you might be inclined to try to video capture the display, but I've not had much luck with this. It actually slows it way, way down. So I'll use an external video capture, for example, OBS studio. Storage, this is the primary step we need to look at here. Notice this little DVD icon is empty. If we try to start this virtual machine without some sort of bootable media, it's just going to basically say operating system not found and it'll do absolutely nothing. So we want to go and select an ISO file. I've already done it, but I'm going to go ahead and hit choose and select the ISO you downloaded. For me, it's Linux Mint 20 Cinnamon 64 bit. For you, it may be something totally different, but you want that ISO file you downloaded in the very first section. Hit open, and it's going to give you the location and you can just hit okay. Once you get to this point, you notice how it's powered off. We need to actually start this. And it's going to pop open the window and it's going to start that virtual machine. And this is where everything's going to be totally different depending on what you're doing. For me, I get the Linux Mint installer, you're going to see like maybe a Microsoft logo or whatever you're doing here. And it's going to just start the installation process. You get this menu across the top and this is not the guest operating system. This is the hypervisor itself. So for example, we've got our disk in there and we have different inputs. We have different views, things like that. This is actually incredibly cool. One note though, if you do full screen, you might get a little confused about what you're actually doing. So I try to avoid full screen and you have your different machine settings and you can hard power this thing off. So be a little bit careful. If you click into the virtual machine, you may actually capture the mouse.  If that's the case, hold down the right control and home and it'll pop up a menu and then you can move the mouse around freely. Notice how I have two mice. This one that I'm moving is the host, this one is the guest, but when you click in, magically they become one and that's why it gets a little confusing. If you find you go to move the mouse off the screen and you don't get it back on the host, then you got to do that little control home trick that I showed you and then it works. All right, you'll get some annoying popups. You can go ahead and close these out. This is going to be different for everyone. I'm installing Linux Mint, so this is what I see. You may see like the Microsoft installation wizard or whatever you're working on. I'm going to install Linux Mint. Feel free to continue watching this section or jump ahead. Just going to double click the install icon because I'm running as a live desk here. I'm going to choose English because that's my native language and English again. I'm going to install the multimedia codecs. This part's a little bit scary, erase disk. You may panic and go, I don't want to erase my hard drive. It's not actually your hard drive. It's the virtual disk. So we're going to hit install. It's going to tell you where you're going to do it. Just hit continue. Select the appropriate time zone. I'm in Detroit and let's go ahead and give it a name. It's going to name it appropriately. You can change the name of everything you want. I've always in the past tried to make this look like my desktop, but it's confused people. So I'm just going to leave it as virtual box. And let's go ahead and give it a password. This is where I would caution you. Do not use the same password you use for your host system because if you're recording and you enter your password, everybody in the entire universe now knows your password. Oh, that is frustrating. So I'm going to just make a password. And this is where you got to be a little bit careful and make sure they match. Once you have a good strong password and they match, go ahead and select require my password to log in. Remember what your password is or you're going to have to pretty much reinstall your whole OS. And the rest of it is coffee time. You just sit back and watch progress bar. Okay, so just about every operating system has one thing in common. When you're done, they're going to want you to restart. Before we do that, there's a couple of key things. If yours is telling you to remove the installation media, you're going to want to go to devices, optical devices and remove. If I try to do that, it's going to tell me, hey, unable to eject because Linux is going to want to do that for me automatically. So just know if it's saying remove the media, go to devices, optical and then remove. Also, you notice how this is very, very small. It's got this small kind of screen. I want to go to view and I want to go to virtual screen one and I want to put this, well, at the size I want. And I tend to record at 1280 by 720. So I'm going to go ahead and resize that. Now it takes up most of the space. However, it's got this annoying menu bar up here. So we're going to go ahead and we're going to get rid of that. So I'm going to say menu bar and uncheck show. Now it takes the entire recording area. If you ever want that menu bar back, you can simply right control and home and it pops open this menu, you go to view and then go back to menu bar and show menu bar. So I'm going to go ahead and hide that again. And I'm going to go ahead and restart. All right, once you have got your operating system installed, you should have a beautiful virtual machine staring back at you, but we're not done yet. We need to install the guest add-ons. So go ahead and log in using the password you used during the setup process. If you forgot your password, I'm sorry, you probably need to now reinstall this. So make sure you wrote that password down. Now every operating system is going to be a little bit different. So just kind of ignore a bunch of these pop-ups that just pop up randomly. All right, once we've gotten to this point, this is a full operating system. And this is actually really, really cool that you can use this technology. Now, if you skip the last section, you notice that there's no menu bar and you're probably going, where did that go? So it is on your keyboard, the right control button and home at the same time pops open to this menu. You can go to view and menu bar and show menu, but you can access all of those same things through this little context menu that pops up. It just may be a little cumbersome to navigate through. So for example, you may be sitting here trying to find your devices, but you're looking for this insert guest addition CD image. And I'm going to actually show it on the menu. All right, let's go to devices and it's down at the bottom, insert guest addition CD image. The first time you click this, it may pop up and say, it's not found, it'll want you to download it. Just hit yes and it'll download and do everything for you. For example, could not find, what do you want to do? Let's go ahead and download it, download again. And then we're going to go ahead and click insert. And what this does is takes that ISO file, it just downloaded and inserts it into the operating system and your operating system should automatically detect it. For example, mine says this is intended to be automatically started, would you like to run it? Some operating systems may just run it for you. I'm going to click run and you'll need whatever password you put in. And this is going to be different for every operating system, but basically you want to follow the installation process. Now I think Windows and Mac, it pops up a nice beautiful graphic interface where you can just click next, next, next. Linux of course, you've got to do it off the command line, which is really not a big deal. The major takeaway here is you want to get it to the point where it's installed. And you'll know when you're done, because it'll tell you. I'm just going to let this run in the background. If you're on a different operating system, feel free to jump to the next section. Okay, as promised, once it's done, you'll know it tells you right on the screen, we are done. Most other operating systems, if you have a graphical prompt is just going to say you're complete and you can just close it out. I'm going to do that. And then you need to unmount this desk. Now, some operating systems, you can go to devices, optical, and then remove, I'm on Linux. So I'm just going to right click and we're going to go ahead and eject this. Okay, the final step in the configuration of the guest here is we're going to go to devices and shared clipboard. Notice this is disabled. Basically what we're doing is we want to be able to share the clipboard between your virtual machine and your computer. So for example, I have some notes off the screen. I will read these notes and once in a while, I want to copy and paste something into the VM or from the VM back into my notes. So we're going to go to devices, shared clipboard, and you can do from your computer to the virtual machine or from the virtual machine to your computer. I like doing bi-directional. And then we're just going to open up a text editor and test this out. So I'm going to say guest and I'm going to copy and I'm going to bring my text editor here and we're going to paste. And then just to show you that we're not playing around, I'm going to copy the host, go in here and boom, we have bi-directional copy and paste. This is extremely cool if you have notes off the screen. So for example, if I were doing a full tutorial, I could just grab something off the screen and say, ta-da. And to the end user, it just looks like magic, but really all we're doing is bi-directional copy and pasting. Spoiler alert, if you haven't rebooted your machine after installing the guest add-ons, you're going to want to do that or it just won't work. And again, Linux users, I did put a note in there, you're going to be a part of that virtual box group or this simply won't work. The guest add-ons are mandatory for this to work. So if you skip that step, it also won't work. It's kind of like a three-legged table, you need all three of those steps in there. And finally to wrap this up, here's some last minute recording tips. So I am not recording the entire screen. It may look like it as I'm recording, but I'm not. I'm only recording a chunk of it. This is just a window inside of my screen. I'm going to put this at the top left. So it's all the way up in the top left corner of my computer and then we're going to configure the menus, the status bar and OBS. So the first thing's first, view menu is your best friend, go to status bar, you can hide that, go to menu bar, go ahead and hide that. You probably didn't see the status bar disappear because it's off the edge of where I'm recording. And now I have this nice seamless window because it's in top left, I don't see it. Now you are probably picking up this top bar here right here where it says running Oracle VM virtual box. How do you get rid of that? Highly dependent on your recording software, I'm going to assume you are using OBS Studio. So I've got a scene set up in OBS Studio called 1280 by 720 screen, which is recording my computer screen and picking up my Yeti microphone. Here you can see I'm actually talking in real time as I'm recording this. So right click on your computer screen, go to properties and instead of doing the whole screen you're going to crop this. This is a little bit of trial and error and what I did is I screenshot my entire screen imported into an imaging editing program and figured out 1280 by 720, how much of my screen was left over. So for example, let's say it looks something like this. I would take this edge here and I would determine how much of the screen was left over and I would take this top and determine how big that was. When I say this top, I mean this guy right here. Measuring that that was about 30 pixels. So when I crop top and I move this up to the top it disappears out of the recording area. Now cropping right and cropping bottom, it was about 640 pixels to the edge and then about 330 to the bottom. Now I get this nice seamless recording. You cannot even tell that this is not even my main desktop. I hope you enjoyed this video. You can find the source code out on github.com. If you need additional help, myself and thousands of other developers are hanging out in the Voidromes Facebook group. This is a large group with lots of developers and we talk about everything technology related, not just the technology that you just watched. And if you want official training, I do develop courses out on udemy.com. This is official classroom style training. If you go out there and the course you're looking for is just simply not there, drop me a note. I'm either working on it or I will actually develop it. I will put a link down below for all three of those. And as always, help me help you. Smash that like and subscribe button. The more popular these videos become, the more I'll create and publish out on YouTube. Thank you for watching.


What's going on everyone Sam here late in sports cards ripping one box of 2022 panini national treasures FIFA road to the world cup soccer hobby for Rick E. So good luck here, Rick. Let's see what we got in here, man. These have been so much fun to rip very exciting All right pickle pickle Rick. Unfortunately, he's gonna have to get out of the way I will this will be in the background there for you guys But I need my trash can next to me. See sealed Yeah Definitely, I think come in a couple of years to especially some of the big rookies that are in this You know, especially if it's like Ronaldo or Messi's last world cup like stuff like this is gonna be a big year for product for sure I have to I'd have to imagine alright We've got Farron Torres 53 out of 99 for Chile or for Spain. Why did I say Chile for Spain? I don't know why I said Chile my apologies there. Not Chile I mean, I'm just gonna grab some thick top or thick sleeves really quick Rick any one second So I've got an accident right next base card. We've got nice two out of ten for Belgium How about Charles? They cattle are you? Charles they cattle are you Nice one there for Belgium that next one's nasty Charles they cattle are you nice hit for nice hit for you there Rick for Belgium? DMC we're streaming over on YouTube as well, but we have a website lane sports cards calm Yeah, I'll drop you the the link to the website This is where you you purchase and then once you purchase I have the orders in front of me and then I just rip them as they come in from like the earliest up to the most recent Appreciate it Fuller. We've got nice one here. That's Edison Kovani for Uruguay One out of 49 on the definitive ink on card Gorgeous signature on that one too. He's got a little 21 in there and everything Grats on that one there Rick Edison Kovani or Edinson Kovani Yeah, I do apologize if I say any of the names wrong Edinson Kovani All right, I'm just gonna flip this around We've got nice going to Portugal three out of 25 Ruben Diaz Ruben Diaz nice one there Not associated not too bad mark probably like an hour right now. I would think Ruben Diaz congrats on that one again Rick fresh this again We've got for England and that's gonna be numbered nine out of 25 We've got Ali walk-ins on the National Treasures biography materials Congrats on that one again Rick nice Ali walk-ins player warm biography me I love the biography materials set. I love those and century materials that they do and Over in the other end and the empty football as well All right up next we've got going back to the US of a that is gonna be Giovanni reina Giovanni reina that's 18 out of 25 on that one again not associated absolutely Absolutely, huh?  And I'm like absolutely not we've got nice one coming up here This is gonna be Going to a Belgium. We got Christian Bantaki or Bantiki 3 out of 15 on the definitive ink on card again Congrats on that one Rick Love the on card stuff man Gorgeous gorgeous signature on that really did a good job using the autograph window as well That'll work right there Christian Bantaki or Bantiki For Belgium Next up we've got nice one there as well. It's gonna be Dennis Pratt or Dennis Pratt or Pratt 10 out of 29 or 49 going to Belgium. Do you imagine this is a Belgium purse our Pick your team out of in the third card right there Dennis Pratt 10 out of 49 that is the national treasures signatures Forget what the exact it's not the lone star because that would be one and one or one but The star signatures Up next That's gonna be numbered 16 out of 25 legend legendary signatures for the Netherlands Edwin van der Sar Legendary goalkeeper Otto there nice one there I've loved the legendary signatures are awesome as well Edwin van der Sar 16 out of 25 on that one for the Netherlands And last card of the break ended it off with a good one 35 out of 99 for Portugal Luis Figo There you go. Congrats Rick man. That is awesome. What a card Again, that is the timeless talent signatures That those are all gonna be on card and the I guess like the true technically would be numbered to 99 But that is solid congrats on the Luis Figo Get this one in a mag for a man That will do it for your box there Rick empty as you can see there and Lid is empty Thanks again for jumping in man really appreciate it. Your second break is coming up in very shortly. So yeah ready appreciate it man