There's something to be said about being prepared at a moment's notice. Not prepared in the sense that you know where all your gear is or how your equipment works. But being prepared in the sense that in a split second, you can make the right decision when lives are hanging in the balance. Retired United States Army Colonel David Hackworth once said, if you find yourself in a fair fight, you didn't plan your mission properly. You know, readiness is a great question. And I think in regards to this, General MacArthur said it best, that on the fields of friendly strife are sown the seeds that on some days and some other fields are sown the seeds of victory. By definition, Red Flag Alaska is a joint coalition, air combat training exercise that corresponds to and tests the capabilities of all its cooperating and participating units. But really at its core, Red Flag is designed and created to test its most junior participants. In life, we often quantify our success by wins and losses. Red Flag Alaska is no different. You either win or you lose. It's how you look at those wins and losses that makes the difference. As an aggressor pilot, it's a different perspective to take because when bluer is winning, we're losing. But ultimately in the grand scheme of things, everybody's winning. When we are all dying and bluer is getting it, that makes me motivated and humbled because I know that the United States and its allies are the most capable of they've ever met. Red Flag Alaska isn't only about the United States pilots receiving training. Over the years, the growing integration of foreign militaries has helped the United States and partner nations share tactics and techniques and better prepare each other in the event of a real world scenario. As America is the beginning, by training in language, culture, and living environment, we can understand each other's common sense. At the same time, we can understand the difference between Japan and the United States. We also have a very big interest in the relationship between other countries. Back home, we do not have other assets like the F-16, the tankers, the F-18s, the seat players rather, readily available. So here we have all these assets provided to us to integrate together. So this allows us to learn more about how they operate and how we can get their strengths and our strengths put together to achieve mission success. The relationships and wealth of knowledge that is shared during Red Flag Alaska is simply immeasurable. But the teamwork from every member that makes each sortie happen is just as impressive. The logistics and the maintenance piece that goes into putting up Red Flag Bowl is something that is an extraordinarily complex feat to achieve. All these different elements that can synchronize all their efforts to get these aircraft airborne, typically within a one- to two-minute window is a testament to hard work and the skill that is resident here when you look out on the ramp and see all the maintainers and all those professionals out there getting those aircraft ready to fly. While the majority of us spend our time focused on jets and the air aspect of Red Flag, there's another part of the exercise that is just as important and plays a huge role in the Red Flag Alaska mission. Red Flag Alaska has evolved from primarily a fighter jet exercise into a full-fledged military exercise integrating ground forces from all services as well as partner nations. When Major General O'Neill took command of U.S. Army Alaska just about a year ago, one of his first charges to us was every training event you do needs to have air ground integration and strive for joint interoperability. And with Isles and Air Force Base in Red Flag Alaska right here, that is really when we started to lean into what was in the realm of possible. A lot of times we learn tough lessons in an operationally deployed environment because we don't do this in training enough, but I do think that there is some multi-domain battle potential here for us to really start to drive solutions and not just how we fight and introduce different services and countries capabilities, but also how we interoperate with each other in that environment. With the growing integration of ground forces comes an elite group of military forces. We don't always see them, but rest assured, special operation forces are always there and always ready. Any place, any time, anywhere. Red Flag is a unique opportunity for us because we're based out of Florida, so what it does is it gives us a chance to train in a challenging environment like Alaska. One of the other big takeaways is obviously you have all this Air Force integration that you see around. It's a chance for us to talk to multiple platforms and basically just liais with our Air Force counterparts so that way we can bring together a good fusion between the soft element on the ground and the air elements above, which is really what it's all about. You know, getting really good effects on targets, allowing special operators on the ground to leverage aviation platforms that we normally don't get to work with on a daily basis. When it really comes down to it, the evolution of Red Flag has taught us one thing. No one person, unit, or service is better than the other. It takes a total force effort to train the way we fight. So if you ever hear anyone say that they're better than anyone else, do us a favor and remind them that we all make the mission happen. No matter your job or your service, we all play a role in ensuring the United States and our partner nations never find ourselves in a fair fight.


What's happening guys today? I have a very special video for you. I got somebody on the channel who worked in retail At Victoria's secret specifically and she was able to transition over into digital marketing in about three months or so So really impressive how fast she was able to transition over I think you're gonna get a lot out of this one. You're really gonna enjoy it So gently tap that like button and let's jump right into it. All right, Ariel. Thank you so much for coming on the channel I appreciate it. Oh, thank you so much for inviting me awesome so let's go ahead and kind of talk about your Background just a little bit like your professional background. Maybe just kind of introduce yourself you can share whatever you're comfortable with and Then maybe just talk about how you discovered digital marketing. Yeah, sure I started working at Victoria's Secret a while ago It was actually one of my first jobs. It was something that I was doing in a transition of Like having a first job while I'm in college I have an associates degree I'm a liberal arts associates degree not really much good for anything because I didn't know what I wanted to do But basically I was working to kind of kill the time until I figured out what I wanted to do I wound up staying there because I got promoted, you know at the time. It was really good for me I was a sales girl on the floor And basically that's really important to this because in Victoria's Secret We have different sections of the store. So there's cash wrap, which is basically like register and back stock We used to have beauty girls Which were the people that just sold like the perfume and when we did have makeup They were them if your local Victoria's Secret store has a pink They were Victoria's Secret girls and they were pink girls. So I was a Victoria's Secret girl I was the person that as soon as you came in the store I was that annoying person that was like, hey, how can I help you? What are you looking for today? It was a very demanding job because unlike most retailers where you kind of just walk in Walk out with whatever you want. If you need help, we'll be like, yeah, sure We had to sell we had to Go after the customer be very personable and get you to buy what we needed you to buy So in that sense, it was very demanding But I liked it because it got me to come out of my shell. So it wasn't that bad at first And then I got promoted like I said as a bra specialist, which yes, that's a thing Basically, I am Supposed to be the most knowledgeable on how to fit somebody What the best price or somebody? Um And I guess I got trained on that it was long hours of training and it's kind of like a low-level supervisor for the sales girls and I would transition from victory secret to pink multiple times because that's different types of selling but You know, um, so I was there for I'm gonna say like five years and that whole time It was good and it was bad because I was getting a lot of experience it was um bad though because I didn't have a life basically I had two days off a week um But in retail it really depends on what day you get And I did have a Saturday I have I had Saturdays off, but that Saturday was either me thinking Am I gonna rest after being exhausted from closing from the night before at 2 a.m I'm gonna see my family. Am I gonna see my friends like it was it was really really bad and I was like at a breaking point where I couldn't do it. I was physically exhausted I didn't have any more hobbies that I kept up with because I had no time And I was moving away from my friends because I never saw them everyone had basically like a regular nine to five off on the weekends kind of thing and that's not true for everybody, but for my life I was That was true of everybody and I didn't see anybody. I remember it got to the point where I I'm not a crier at work But I remember I started crying at work one day because I couldn't do it anymore. It was it was that bad and um, I decided that I have to I have to find something else like I have to do something for myself or was it's just Gonna lead me down a path of like real misery But I think what's interesting is that a lot of people don't leave their jobs there at even though they're miserable at them It's because they're afraid and I feel like that's what I had to face is that I was afraid that I wasn't going to be able to do anything else like I didn't go back to college because I didn't know what I wanted to do So I was like, what am I going to do then go to another retail place to do this do that?  So I always liked marketing in a general sense Like I loved the idea of marketing adversizing is very into that stuff But I really actually never thought of pursuing it as a career. I was like, well, you gotta go back to school for that And I at first was wrangling with the thought.  They can't afford to Send me to school fully With the salary that I was making I certainly couldn't do it and I had friends who had gone to four year colleges and they were still paying off their loans and have a lot of debt and I didn't want to be in that situation until I was like, absolutely sure. This is what I wanted to do so I started researching Into marketing and I took like tiny little courses that they sell online like for 20 dollars from like you to me or Corsaya And I was, you know, just doing that But it wasn't leading to anything tangible. I'm like, okay. It got some knowledge but like Where is this going? And basically the more and more I looked google knows that you're looking for I got some suggestions on youtube and it was for Seth Jared's digital marketing course And the first thing I watched was an interview That he had with somebody else who joined his course I didn't believe it at first. I'm gonna tell you I was not a believer. I was like This is not real. This guy did not pay this much money for this course And now he has like a whole job making six figures in a year. I was like, this is total BS A lot of people think that a lot of people think that when they watch I can't blame them. But I mean when you see like dozens and dozens of interviews What do you think they're all paid actors come on now That's the thing is that there were so many And I kept watching them and then I would look on their LinkedIn The actual people in the interviews and I was like, okay, they're being serious and So what I did do was I got his free mini course, which was like the introduction to his course It took me I think like two weeks to make a decision I was actually going to Miami to meet up with some friends for like a break And I was like, I'm gonna go and I'm gonna come back with a decision and He charged about like $800 for his course. I think which In the grand scheme of things $800 $40,000 for a one-year marketing out of school. I don't even know if I want to go to so It is a lot of money But I feel like I had to invest otherwise I was gonna go nowhere so I sent him a message um like an email and I said hey like so Is this like a career like is it really marketing like is it creative because I'm a very creative person and you know Like is this something that somebody like me would like to do and he Answered me back right away and he's like, I think SEO would be perfect for you He's like, but I don't know you gotta give it a shot. So I did and It took me one month to finish the whole course That is not true of everybody. Everyone has to go at their own pace. I personally was just I was like on a mission to like Change my life, you know, so I really rationed my time after work if I could do it after work or before work I would go to my local Barnes and Noble and I would study for like two hours a day every day Monday to Friday And that was my life for a month and I was like I Going to be done by the beginning of May because I started at the beginning of April And I did it was hard because I feel like there's some points where You get caught up in the tricky parts and you're like, I can't do this It's too hard and I was like no like this is these are the times that you push through it. So I did and Think about a week after I finished the course and I did my resume the way Seth instructed Um, I started sending out applications and within the first week. I started getting interview responses, which was great. Um, that as most people know You really hope and pray that like the first one is going to be the one that just gets you You're going to be like the exception. You're going to get that job right away. Um, it was not the case Um, I went through a lot of interviews. Um, and sometimes to the point where I they would be piled On top of each other in like one day and I would be so eager to take these interviews that I would literally like Stick them in between lunch hours and then like kind of make them through like, oh, I can do this right before I leave for work or right after like It's a little crazy. I think I went a little overboard, but I was so eager to just get in there and um, I remember The job that I'm currently at now was actually the job that I thought I had no shot at getting Like I said, this is definitely not the case for everybody But it was for me. They were offering Way above what was I guess kind of normal Um for an entry level SEO position and I just applied because it said entry level I said, I'm definitely not going to get this but like I just did it anyway. And I got a response back that was basically like, hey Um, we saw you applied we're interested But um, just so you know, this is a very client heavy facing job. Are you comfortable with that? I was like, of course I am but I was like, I'm not gonna get the job anyway. But yeah, sure. Whatever. Let's do an interview I did the interview and I got invited back for a second interview They told me all about the job and I thought it was great. I loved the people there Literally the next day I got my first, um, acceptance letter and I almost started crying because I was like, no way like they are crazy. They are not looking for me Like, you know, what I actually did do was I responded back and I said, hey, you know, um Can I think about it? And I know that sounds really crazy, but I was like, I didn't want to say yes right away I wanted to make sure that this was the job for me because I remember someone Seth's course said Just because you get a first response back doesn't mean it has to be that job that you want, you know, like somebody From the team who's my current boss now who's amazing reached out to me. He said, hey Do you want to do a zoom call? We can talk it out. See if you have any questions that might make you Help me make a decision. I said, that would be great. So I discussed the role with him I am entry level and I want to make sure that this position is for that And, um, basically he was like, no, we know exactly where you're at. Um And he was very honest with me. He said we had people that were Way more experienced than you that did apply for this job, but They did tell me that they picked me because They liked my personality and that I was a culture fit for the company I was like, well, that's great. And I accepted right then and there and Now I'm in my position. I've been here for Three months It's great. It's really great and I definitely see why they hired me because to work at this job specifically you have to be very friendly with your clients very personal And they are very big on company culture here We all work from home. We can't be together and it's hard to stay connected, you know, and for me I feel like company culture is such a big thing because if I can't feel connected I don't know why I would want to work there for me. That's motivating other than, you know, the paycheck, but Um, that's it really got me and I'm very very happy where I am now and I feel really blessed that like I was able to Come here. Okay. So at your current job right now, I know you said like work-life balance is the most important thing And that's awesome. That's great that you, you know, have your priorities straight Would you say like digital marketing in general if you want to have really good work-life balance? It's a good path to go down. Oh, absolutely. Yeah, I feel like it's like, you know, some people say like It depends on like the the exact job you're going into in digital marketing They say like a job like mine. That's very client-facing. It can be more restrictive My company told me as long as I'm scheduling things around my clients. I'm just doing what I gotta do I have all the flexibility in the world and realistically a lot of companies don't schedule meetings outside three o'clock. So As I can be out by three if I want, you know, or if I have something to go do, I'll go do it I feel like it is a really flexible job. If that is something that you value Um, I would just say that always be aware that because you are working from home and it is flexible You have to keep yourself in line. So let's go ahead and uh, whatever you're comfortable sharing I know everyone probably, you know, this is the question everyone wants to know But uh entry-level job and digital marketing What would you expect to be paid or if if you're comfortable you can tell me kind of what the offer they gave you was Yes, so what from my understanding from Seth and all his students and from what I saw in the applications What they were initially offering Generally between 35 to 45,000 to start that was pretty much the medium I was offered 60,000 to start gosh I remember I asked Seth I was like is this normal and he's like it's a bit higher than usual, but um It's not like wildly crazy. Um, but I think it was like I said they were looking for an already experienced candidate and what they got with me was you know I told them I have to be treated as an entry-level person. I need the training. I need the time um To grow and I feel like what they did with me was they just invested in me to basically kind of make me Like their creature essentially so that way I can grow with the company really absorb everything and then just be the best that I could be um But I feel like, you know, that's Those are personal moves that companies make so another thing I quickly wanted to ask you about is um Just personality wise Who like what type of people tend to do best in digital marketing and maybe is there like certain personalities? To do better in the different lanes in digital marketing So for instance, like creative people analytical introverts extroverts, etc It's definitely a mix And it's definitely something that you have to kind of discover on your own, especially if you're joining a company Like I said for me Um, my job, it's very client-facing. I have to be comfortable with talking to people Um, understand that there can be like Confrontations and stuff like that. It hasn't happened to me yet But like I have to learn how to deal with that in a very personable way for an introvert I don't think that's something that they would want to deal with Not a lot of SEO jobs require this to be very client-facing. There's people that are on the technical side They don't have to talk to anybody but their bosses or whatever and they just go about their day. That's great um, ppc same thing Behind the scenes they go about their day, but I know people in ppc in my job They have to be okay with being client-facing too. So they kind of have to like bring that out It's definitely for all different lanes of people and I think that's why this career is so great because it mixes all those kinds of people I know a bunch of people in my company that are very Opposite me very introverted And they do fantastic. They're amazing people. Um, and I feel like introverted people when they're around people that they like They come out of their shell more and I feel like it challenges you to be More than you expected like for me. I'm a very creative person analytical like People are like, you don't like analytical things and yet you're working with google analytics every day and I'm like, well that's what came with the job and I challenge myself every day to Be that person to be that person's like, yes, I'm doing it. I'm doing it better every day May not know it as fast or as well as the guy next to me But I'm gonna do the best that I can do and come out of my shell and just Do it actually I did want to mention Seth does have a free master class So if you're kind of a little bit confused about the different types of digital marketing like seo pay-per-click Search engine marketing all this sort of thing All of that will be explained in Seth's master class. So I highly recommend checking that out I'll put it down in the description as well as the pinned comment below One last thing I wanted to ask you and that is if somebody's kind of on the fence About buying Seth's course Investing in Seth's course And going through it. What would you say to that person? I would say Do your research Because I feel like it's a it's a personal choice and for me I really had to do my research like I watched so many interviews I read so many outside resources and I had to really think about like is this going to be valuable to me Is it going to give me the kind of life that I want? And sometimes it's just a leap of faith, but definitely just do your research and If you're really feeling like you want to do it, I say do it I know you do have to drop some money, but It's definitely worth it as long as you stick with it He has a community of people on facebook that talk all the time asking so many questions Supporting each other because it can like I said, it can be very difficult, especially in the job process where The job finding process where you're sending out all these applications And you're getting rejections like I got so many rejections and there was a point where I was like, you know what? I don't think I can do this. I think I'm doing something wrong I remember I went in the facebook chat and I was like, you know, am I doing something wrong? And everyone was like, no, this is the process watch give it some time You'll find something and I did And I feel like that that's really what motivates people is other people that are in the same boat supporting each other So I feel like with that You'll never be alone In this journey Get out there and like connect and I definitely just think it's totally worth it Like everyone that has taken this and seen it through have all been incredibly successful Yeah Yeah, absolutely. Some people have even told me they've just gotten jobs from the networking through the facebook group essentially, right? Yeah, that's how Yeah, it's it's like literally thousands of people who are you know interested in digital marketing or already in digital marketing So yeah, it can be super super useful. Well Hey, thank you so much for coming on the channel and sharing your story I think it's going to inspire a lot of people out there Really cool story. Congrats again on your success as well Yeah, so thanks for coming on. I really appreciate it Thank you. I'm really happy to share my story and I hope someone out there Maybe makes a decision from watching this because it's totally worth it


So now we're going to put a pause here on copy on right table. I don't want to like overload it because the first Jupyter notebook that we're going to work with is going to be a copy on right table. So I just kind of want to cover the storage type, kind of what it provides, and going with the different query types you can run, and now we're going to switch over to the actual hands-on lab. So if you go back to the lab that when pointed out, you're going to see there's an introduction to part two and hoodie. If you go to the workshop ID as check-in, this is some logistics that we're going to work on now because we need to assign IDs to individuals. So when you copy the notebook, you have your own ID and they're not concurrent writes happening to the same notebook, which is why we're going to have a suffix of the ID to prevent the concurrent writes.  I actually have 50, so self-assign your number. I'm actually 50 because that's what I worked with. Self-assign, I can make this bigger. Your ID because we're going to do some changes. We're going to need that for later on in the workshop in about two minutes. So I'll give you guys like a minute or two to self-assign your IDs. And remember your IDs. Don't be troublemakers. I'm presuming everyone has goodwill here. Yeah. First ten out of community. Yeah, first ten out of community. Have goodwill and do goodwill to others. So if you have questions around hoodie or presto, now is a good time to ask them because we're on some downtime while people are self-assigning their IDs. I don't have Slack open, but let me check Slack. Cool. I don't see any questions yet. Okay, I think everyone is pretty much good to go. I don't know why I put a D there. Okay, cool. So the next part of the workshop. Where is my thing? Oh, yeah, that's right. It doesn't create. It opens. Okay, yeah. We're going to access the Jupyter notebook. So similar to what Wen did earlier, depending on your birth date, please pick a specific EC2, I guess, instance, right? Yeah. So I'm 11, my birthday is December 12th, so I'll go with number two. And then the password here is summer sun. So just copy the password and then click on the EC2 instance. Actually, let me just open up a new file or a new tab. There you go. And then let me know if I need to make this bigger. Make it bigger. Ashley, let me self-assign to 50. I'll start with a new one. I'll use two. So we have a lot of things left. Put this just in case. Okay. So if you also go in the workshop, if you go to, so we are in the Jupyter notebook and now we're going to copy, we're going to show you what to do in this step here. So the first thing I want you to do is that you see that there is a lab template, Houdi COW. Let me make this a little bigger. IPYNB. The first thing I want you to do is I want you to duplicate it. But this looks a little different from what I'm used to seeing. This was different, I think. Hold on. Can right click it and then let the duplicate go. Okay. Let me go back then. So the file. Yeah, let me try that. And then duplicate it. Oh, yeah, there it is. Okay. Yeah. So we're going to go ahead and duplicate this. So you're going to right click on lab 1 where it has this COW suffix and you're going to click duplicate. Okay, now when you duplicate it, you're going to get a copy one at the suffix.  So, delete copy one and put your ID as the suffix right before the extension. Why is it not letting me re-copy? Let me duplicate again. Rename 51. Okay, there it goes. All right. So you see I have 51 here. I want everyone to rename their file or their lab notebook to their extensions. I'll put a few minutes, one minute or so there. Oh, okay. Rohan says make bigger. Rohan, do you want me to make it bigger more? Is this good? Can everyone see this okay? Okay, cool. Let me make sure I'm on the right notebook. Yeah, I'm on 51. Okay, so the first box, what we're going to do is we're going to get our environmental variables and keys set up over here, and where it says ID equals 3 or whatever it is, just change it to your ID that you self-assigned. And then we're going to concatenate that ID to specific parts in this workshop. Yes, looks good. Okay, great. And then after you have your ID, we have our access keys and seeker keys. When I believe this is the instance that we're running, right? So this is not your instance. This is what we're already running. And then we're just getting our arguments. And this is all like environmental variable and key stop. So we're going to go ahead and run that. And then after you run, you shouldn't see anything. The next thing I want you to do is I want you, there's actually nothing to do here. We have the app name. This is the application for the spark that we're creating. And then you can see that we concatenated your ID at the end. So there's nothing you need to do here. Actually, you can just probably run it. You can create your spark instance. And I'll pause here. Did everyone, did it load okay? Did we get any errors? Just want to make sure everything is, if you get an error, please flag on Slack or flag on here and then we can help you troubleshoot through it. So once we create our spark instance, now we're going to create our table name. The table name that we're creating is just the COW table, which we talked about earlier. And then it's just, we have a suffix of the ID. And then we have all the places, all basically the, where are the data stored in S3? These are like the locations. So we're just setting the variables to the locations. And then from here, we set the hoodie, the hoodie S3 path. So all these are just basically the locations of where things are the data, where the, where it's located. See if there's any questions. Okay. So you're going to go ahead and click on that box and you're going to click on run, which is the little triangle button there. From here, we're going to create our schema. So that we're dealing with a CSV file. And because of that, we'll need to construct our, basically what our schema looks like. And then from there, we're going to read the CSV data. We're going to display it and we're going to count how many records are there. So I'm going to go ahead and run this part. And you see when we run it, you can see kind of it's showing the top five rules, which is what we want to display because we put the, what we put the argument by there. And then we can see that there are 79 records. So this is just basically, we loaded the data and now we're querying it to make sure that the data is there. This is all this part is doing is just making sure that we have data in the bucket. So now what we're going to do is we're going to actually write to hoodie. And so what we're going to do is we're going to specify the table name where we want to write to, because hoodie offers two different storage table types. We have to specify this, the storage type that we want when this case is going to be a copy on right. The operation mode, the operation mode is going to be absurd. Hoodie offers different types of write operations like insert, absurd, delete, bulk insert and so on. In this case, we're going to absurd, which is inserts and updates. And then from there, we're going to specify the record key, which is the user ID. And then we have the pre combined field. The pre combined field is used to prevent for D duplication. So if you're working with streaming data, you might get tons of duplicates. In this case, you'll want to run with a pre combined field to prevent that duplication. So when you merge records and it's written to hoodie, you're not writing duplicate records. The ID here is that we're dealing, we are like, even though it's, we're dealing with mostly like a, kind of like a bulk load in this case, but ideally when you're dealing with ecommerce data, it's mostly going to be streaming, right, into your data light. So in this case, you will definitely want to define a pre combined field. And then all we're doing is we're just writing to the S3 hoodie path that we defined earlier. So we're going to go ahead and run that. And then from here, we're going to read the data. We're going to create a temp spark view. And then we're going to read the data. And then we're going to run some queries on it. So we're going to select the total records from cart status, and then we're going to select some fields, and we're going to order it. So I'm going to go ahead and run this. I think this is still running. There we go. So if we count the, the total records from cart status, there's 10. And if you want to see there, you can see all the user is used that exists here. And now we're going to do an update. So we're going to read the CSV data. And we just specify the headers true. And then we have the schema, which we defined earlier. So it knows how to display the data. We're going to show the last 30 results. And then we're going to count the records. So we're going to go ahead and run that. I don't know why. Oh, am I running? Hold on. Let me see. Oh, I don't know why. Okay. Oh, wait, you know what? I didn't double click. I'm so used to clicking on one, and it pushes off there. I didn't double click. That's why I'm getting that error. Okay, let me see. Let me actually, let me, let me try this again. Let me try with a new ID. I actually didn't double click onto it. So I'll take 52. Hold on. I hate when that happens. Let me get rid of this. I got to, let me rerun it. I didn't open up the right top for me. I didn't double click. It's a bad habit. I'm just running through the set of exercises that we had. I'm just redoing what I did earlier. Okay, I believe we're here. We wrote to Hitti. And then we're going to verify the initial loan. And then we have the table type. So I think we went through this already. Okay, so now, yeah, we'll start here. So we're on the D, D final spark that read format. And then we're going to show the five records. And then we're going to display the results and run the query. That's this part here. You can see the user ID in the event time. And then from here, we're going to read the incremental data. And we just provided the schema and we said that they're headers. So it knows how to read the data. We're going to show the last 30 records and then we're going to count it. So that's this part here. So everyone should be here. Am I going too fast or is this pretty, pretty good on everyone? Let me check Slack. Everyone here good so far? Okay, let me see everything. Okay, cool. All right, third question is definitely flagged. Okay, so then from here, we can see the records. So we can see that we've added to cart. Let's look at the schema really quick. We have action type, card, empty event, time date, last long done, last purchase user ID and then the ID. So these are all like the different activities that user are doing with the timestamp and what action they're performing. That's what this data set is about. So now from here, we're going to fully load this data, the update data into here. So what we have is we're providing the table name, the type of table we want to use, the upset record, the record key. And then we have the pre combined fields, which we talked about earlier. Now we have this payload class. This payload class basically defines the merge logic of how you're writing if there is a one record and a record already exists in hoodie, what the merge logic is to merge those two records and there's different types of merge logics that we can have and how you want to do that or you can specify your own merge logic as well. So that's what this is doing. The ordering field. So this in this version we're using hoodie 010 0, the current version is hoodie 013 0. So basically the ordering field is kind of the same thing as the pre combined fields. And 13 0 that's kind of been deprecated. So for the purposes, I wouldn't worry too much about this because if you're on the latest version of hoodie, most likely you're just going to specify the pre combined field. And then the payload event time field is mostly used for like metrics. At this point, we're not going to specifically use it today, but that's what this one is about. And then we're going to write to the hoodie path. So we're going to write the updates that are happening into our hoodie table. So I'm going to go ahead and run that. And it's still running. And now what we're going to do is we're going to read, we're going to create a temp view so we can actually query the data. So this is what we're doing over here. So we're going to select the distinct hoodie commit time from the cart status. And then we're going to select the specific fields we want and we're going to order it. We have the commit time and then we have the different users and stuff. So we have 13, 12, 16, these are all the updates that have happened. So that's what we're seeing from here. So if we go back here to earlier, we have to 16, 9, 5, 10. Now what we're doing is we're seeing there are updates that are happening. So now you see that with the updates, there are more records that exist here and these are all the updates and changes that have happened. So whether a user ID has had an update or whether there's an insertion of a new user that has activity as well. And then what we're going to do now is we're going to perform a delete operation. So let's say if you want to run a delete to comply to privacy laws and things like that, like GDPR, you can do that as well. So what we're going to do is we have the data set to delete the raw data. And again, it's a CSV file. Same thing, we're going to provide the header and then the schema, which we defined earlier. And then we're going to just show that the records, what the records look like and we're going to count the records. So this is just querying the CSV file basically. And you can see the deletes that were happening is 13, 14 and 10. So these are the user IDs that we want to delete. Now if we go over here, now we're going to write to the same hoodie table and we're going to delete the files basically. And then much of these are kind of the same. The only thing that's different is the parallelism. The parallelism is basically threads to do different types of operations to do the delete. So it just provides different, like shuffling the data and things like that. So that's what this part is, but the rest of the configurations are the same. And then now if you want to query the table and to see if the deletes happen, you should see that 13, 14 and 10 should have been deleted. So if you count, there are 15 records that you can see now, all the records exist except for 13, I think 14 and 10. So if you go here, 13 doesn't exist. But if you go back to the previous query that we ran, you can see, though this was querying the other data, 13 existed somewhere. I think I pointed it out earlier. Let's see. I think I'm blowing. Oh, here it is. 13. There we go. And now if we look back earlier and we go back to the latest query, you can see that 13 doesn't exist. So this is how you perform upserts. You can delete on hoodie. And this is how you'll do it through Spark. Now later, we're going to show you how you can query. We're actually querying through Spark, but we're going to show you at the end how you can query through Prosto, through the Prosto CLI. Yeah. So the question. So that delete, was it a hard delete? Did it actually delete the file? It's a hard delete. Okay. So I can't actually time travel and get that record back. No, it's a hard delete. But with the new CDC RFC that hoodie provides at 13-0, you can do incremental queries on hard deletes now. That's for the later version of hoodie. Yeah. You can also do a soft delete. Yes, you can. Yes. So just for the other folks, the reason why I bring this up is, again, you're separating kind of what the table actually is from the underlying storage. So if you do a soft delete, then when you query the data, it's as if the record doesn't exist. But if that data is actually still in the lake, then technically you can time travel back and see that data. Yeah. And again, this is all managed because hoodie keeps track of all this for you. And the end, but to the engine, you're just looking at tables. Yeah. So hoodie again has a sense of a timeline. So every commit or every transaction that happens, if you have a bulk insert, or you're having data that's being adjusted, it's all into a timeline. A timeline is how timeline is basically immutable. But it's how we keep track of what's going on with data in the tables. You can specify a hard delete or a soft delete. Yeah. If we go to the hoodie, oh, well, let me go. There we go. If you go to the hoodie documentation, and we go under docs, and we go under the spark guide, and you go under delete data, there are different ways to provide, whether you want a soft delete or a hard delete. So it's a similar thing. You just specify it. So I'm just repeating the question for the virtual audience. Oh, sorry. Was the hard, is the hard delete the default? Do you know the hard deletes the default? I believe it is. Yeah. But then if you wanted to do soft deletes, let's say if you wanted to look at Python, you can specify that. But yeah, usually it's a hard delete. Cool. Let's go back to the Jupyter notebook. So this is basically the COW portion of the lab. Let me check on Slack to see if there's anything questions. No questions. Okay. Any other questions? Okay. So now we're going to move over to the merge on read table. So we're going to go back to our slides really quick. And then we're going to start that. So let me go ahead and go to the slideshow. Okay. Now we talked about the copy on write storage. Now we're going to talk about merge on reads. So our merge on read table is a little bit different from a copy on write table with a merge on read table. And it's really effective if you're working with a lot of writes. You have a lot of writes happening, you're working with streaming data, depending on your workload. But basically when writes happen, they get written to these log files, which are avro files. And then you also have a base file, which is the parquet file. Now, when you have, when you have, when you do like your first load, for example, it will be written into a parquet file. But subsequently, any other writes that happen will get written into, if you look over here, there are log files. Oh, sorry. They get written to a log files. And because they're, because when you write to a log files, there's no like merge, they're not merging records at the time that it's getting written. And this is why the right amplification for an MR table is very low compared to a copy on write table. Now, earlier I mentioned that a copy on write table has slightly operational, no, sorry, less operational complexity than a merge on read table. This is because with a merge on read table, there is a service called compaction that hoodie offers that basically combines all these log files at some configuration that you specify. Like how often do you want to do compaction? You know, how do you want to merge these files? This is all defined in the compaction service. It gets all these log files. And then it merges it into basically a new version of the parquet files. So now all the log files get merged in and now you have whole parquet files. And then from there, when you have more commits that are happening, they'll get written to log files, a compaction service at some time will combine those log files and to a new form of the parquet file. And then the sequence keeps happening. So this is why merge on, merge on read tables have a low write amplification, but a slightly higher read amplification because it's merging the files later on through the compaction service. And then you can create the parquet files. So that's what our merge on read is. So if you look at the blue boxes that commit time zero, basically we're having our first initial load. Because it's the first initial load, everything gets written into a parquet file. We have, you know, ABCDE. So if you're running a snapshot query, which is the current state of the table, you're going to get ABCDE back. If you're running an incremental query, again, this is a first initial load, there has been no updates are going to get everything back. And if you write a read optimize query, you're going to get everything back read optimize query basically looks at the basically, it's kind of like there's a little bit of lag because it looks at the parquet file. So it's not like what your table looks like at this point in time. It's what it looks like when everything has been merged into a parquet file. So now if we go to commit time one, we see that a has been updated to a prime, but you see it's written into a log file. There's file one t one dot lock. So it just gets written to a log file there. It just, it just writes it straight to a log file. And if we look at the look at CDD prime also got updated. So this update also goes into a log file as well. So now if you're running a snapshot query, you're going to read what the current state of the table looks like. So this includes all the data that has been updated as well. So you have a prime B, which didn't get updated, C didn't get updated, but D got updated to D prime. So you're going to read D prime, and then he didn't change. So you're going to read E. So this is the current state of the table, where any updates, insertions or anything happen.  So I do want to slow down here because I think this is the key part that the big trade off between the copy on right table and the merge on retable. So notice that on a merge on retable, when you're doing a right, you're just writing what you want to write. You're not doing anything fancy. So from writing, it's very efficient. But the way you get your penalties when you want to read it, when you want to read it, you got to merge these files to get the effect of what the data looks like. So so depending on what type of query you want to run, depending on what type of query run, if you don't run the read optimize one, if you read the snapshot one, it has to in real time merge it. That's why it's called merge on read versus the copy on right, the copy and write all the all the heavy lifting happens when you write it. It's merging it when you write it. But when you read it, it's fast because the data's already merged together. And so this is the fundamental trade off that you pick. Are you are you are you want efficient writes and correct me if I'm wrong, you want efficient writes, you do merge on read is efficient, you write the delta, but slower when you want to when you want to read it. If you're like, Hey, I don't need to be as efficient on writing it. But when I read it, I'd better be fast. And so that's when you would pick a copy on right table.  If you're doing a backed workloads like once every hour or so, maybe like a copy on right table is more than enough because you can afford that type of latency. But if you're working with streaming data, you need to be very efficient on the rights. So you can have so you can reduce the right amplification. And there's a slight cost on the read amplification. So if we look at the incremental query, if we're just running an incremental query, we just get those updates, whether they're in a parquet file, they got written to log file, we just get those updates. And now if you read a read optimize query, you can just see just queries the previous state of the table. So it's not up to date of where it is. So there's a difference between the snapshot query and a read optimized query, depending on what you want. Now if we go to commit time to a got updated to a double prime e got updated to e prime. So there are two updates. So if we run the snapshot query, we get the current state of the table. So basically will it'll be a double prime. So a prime is now updated. So that's gone. Be didn't change seeded and change. D prime is just D prime because there's no update on commit time to and then we have e prime that got updated. So we're reading that and then apparently their f got inserted. I think this might be this might be wrong. The f actually should have been written to a log file. This is actually not technically correct. But if you have an insurance enough, which is not recognized here, it should have been showing.  And then if you run a read optimized query, it just creates the previous state of the table on the first on the first initial load. And then finally, if we do commit time three, what we're doing over here, you see everything in the parquet file. So at this point, we have run the compaction service. We have gathered all the log files and now it's all compacted to a new version of the parquet file. And now if you're running a snapshot query, you're going to get all the current state of it. So you're going to get a double prime, B, C, D prime, a prime and app. So everything got merged. All the updates and everything are now merged to new parquet file. So this is the original version. Now this is the updated version. If you're running an incremental query, you're just going to get the latest changes again. So between commit time two and commit time three is just the compaction service that was run. There is not actually any insertion of data or updates that are happening. This is literally just the compaction service. And then from there, if you run the read optimized query now, this is the biggest change is now you're reading the current state of the base file, which is a double prime, B, C, D prime, E prime and app. So now we have a new snapshot of the read optimized query that I will look at. Any questions here? Okay. Now if you want to look at the trade-offs for the hoodie for data lake, there is a data latency cost. So we talked about this later because the right amplification is slightly higher. You're going to have a little higher data latency lower on the merge on read because you're just literally running to parquet files. And as you looked earlier, if you wanted to run a snapshot query, you can get the current state of the data. So it has a lower data latency. If you want to talk about updates, the copy on right has a little higher update costs because there's a merge logic that happens so you can have those parquet files that you're writing. For merge on read, because you don't have that, it's a lower update cost. And then there is a parquet file size, and then you have the right amplification. What's not on here is the read amplification. So that would be lower. And copy on right is slightly higher on merge on read. And these are what to consider if you want to use copy on right or merge on read. So merge on read, if you need quick ingestion, streaming data, your workloads can change. Sometimes they can be spiky. Sometimes they are not. You don't know the patterns. Typically happens at streaming data. And then for copy on right, you can afford the data latency. You understand your workload. It's pretty consistent. And so on. And the difference between, again, merge on read and copy on right is the compaction service. If you're using merge on read, you would likely want to do a compaction service so you don't have all these log files that are taking storage and you can just combine it to a parquet file. Okay. So now what we're going to do is we're going to navigate back to the lab notebook. And then let me see if there are any questions here. Nothing. Okay. So now we're going to navigate to the base MOR versus the MOR.IPYNB. And we're going to do the same thing that we did on the COW table. So we're going to right click and duplicate. And then when you click the duplicate, you're going to right click and then rename it. And I'm just going to rename it back to like 52. And then enter and then make sure you double click. Don't be like me. And then that should work. Let me, there's something I want to show really quick. So on this part on the MOR table, we are going to show an incremental updates. And I actually, I actually rewrote a query to show that update. And I'm wondering if it's stuck. It didn't.  Let's go by it. I think I remember it. Okay. So the first thing we're going to do is after you rename it, I'll give everyone a minute to do that. You can just, you can use the same ID that you identified earlier. So I'll just pick one I picked 52. So when it goes to the ID, you're going to change that to 52. And then you can go ahead and run the first box. It's a really similar setup. Basically, we're getting our environmental keys and setting everything up over here. So we can actually access the data. And then I think you can see that. Yeah, you can just run it. Make sure you save it and run it. And then from here, we're going to create our app name. And we're just going to put the ID in the middle. There's no conflict. And then we're going to create an instance spark and go ahead and run that. And if you're on this side, if you're on this, if you're looking through this, we're actually in lab two now. And we're going through this step right now. And then what we're going to do is we're going to create our table name. We have different batch data that is stored in an S3. And then we have the hitty path of where we're going to write the data or do the updates and things like that. Since we already have a variable with ID, we don't need to do anything, we can just go ahead and run it. And now we're going to read the batch data set. So just make sure that's good. It's always good practice to do that. We're going to display five results. If you notice that we're working in JSON, so we didn't need to define a schema with the CSV, we had to define a schema so we can just automatically read the schema, read the JSON data. And if you look at this, we have updated cart, we have action type, app name, app version, cart empty, event time, ID, last long done, last. These are all typical things that you would want to look at when you're working with some sort of e-commerce data application. So these are basically all activities that user has had. Yeah, you probably, you'll need to probably denormalize it. Some databases work with nested objects. Yeah, that's a good question. And now we're going to count the records. So we're going to see what the status looks like. And we have a thousand records. So now we're going to do a full load into hoodies. So now we have the data, the batch data set. Now we're going to write to the hoodie table. And basically the only thing that's different is we're specifying that we have a merge on retable. The operation type is upsert and then we have a record key and then we have a pre-combined field which prevents the duplication. And then we write to the data and then we write to the table. Writing still. And now we're going to verify the initial load before we apply any incremental changes. So we're just going to create a temp view and then we're going to write some queries. So we're going to select the distinct user ID and then we're going to select the user ID and event time TS. So you can see all that, all that's here. And actually I have done earlier, but I'm going to write another query because I want, it's not very obvious in this query what the updates are happening. And instead of ordering, I'm going to see select user ID event time from there. Where, hold on, I'm just going to do where user ID equals, I think this is it. I had this done earlier. Let me see if this runs really quick. I'm just selecting the user ID right now. Oh no, it didn't do that. I'll come back to this later. Okay, so now we're going to provide the updates. So we're going to read the new updates that we're going to insert to the hoodie table. So we have a JSON data. We're going to show the results and then we're going to count the records. So if you look at this table now, these are all the updates that are going to be written into the hoodie table over here. So we have, where is the user ID? 1063, 1066, 1042, 1047. Okay. And then 1059 and so on and so forth. Okay. So these are all the updates that are happening. And then we're showing only the top 20 rows. I think this is the part that I wanted to write. Okay. So now if you go back to the bottom here, we're going to write to the hoodie table. And basically everything pretty much stays the same. And now we're just going to write the updated batch of data to the hoodie table. So you can go ahead and run that. So basically we had our initial batch load of raw data. There were some updates that happened. More users had actions. If we go back to the slide over here, where we talk about updates, we're just running an update. So we run queries, we can just see the changes from the updates. So now after we run that, we're going to verify that the updates and inserts have happened and 51 records should have been updated. And now this is kind of hard to read. I had written another query. So I want to, I'm going to actually do this on the fly really quick. And I'm going to show you what the updates look like. If I remember correctly, I want to say I believe I remember the user ID. And instead of event time dates, I want to show something else. Event time TS. I'm just writing a quick SQL query at Spark SQL. Hold on. So you can compare apples to apples. Let me go ahead and rerun this really quick. Okay. So 1047 had this. So now I want to, I want to show in an earlier timestamp that it did not exist or that it was updated. So this is the updated data. And if you look earlier, this is the, these are the incremental changes. And if we look before we did the incremental changes, when we ran the query, the initial batch data set, I think, let's run that. Let's see if that works. Okay. So if you look earlier, if we go back to earlier query where it says df cart status Spark read JSON batch one data, and I'm just selecting, basically selecting user ID and event time TS. And I'm just selecting a user ID so you can see the changes. You can see the ID 1047 had this event time TS 11 16 28 53. Now if we look later, and I query that same ID. Oh, oh, oh, yeah, yeah, I didn't rerun it. That's right. We run that and things really quick. And then we have the count. Yeah, I'm trying to find where I added the thing. Oh, yeah, there we go. This is the okay. So this is the updates and inserts. Okay. So this is that 28. So now let me go back to batch data one and copy that. This is the initial load. And if we run this, oops, where did I put it? I think it was higher. We have 10 this one is the initial load. No batch one. Let me go ahead and rerun everything. And let's go ahead and rerun this. Oh, it did an update. Oh, yeah, there's a lot of boxes. There we go. Oh, yeah, I have it over here. This is where it's at. Oh, it did an update. There was a there was an ID that I wanted to show that there were updates that were happening to the records. I forgot the ID that it happened on. But basically, if you were to query the IDs, you would see that there were different timestamps that were happening between the initial batch load and then the incremental update. So if you wanted to query the different user IDs, you'll be able to find one. I don't remember the ID that happened. I just did that earlier. But that's the point of the exercises show the incremental updates that were happening. I'll pause here. Are there any questions? Let me check the slack. Cool. So this wraps it up for the hoodie portion on how you can query your data and connect to basically, we're querying through Spark. And then we're going to show you how to query through the Presto CLI. You're going to query the hoodie tables basically. So I'll disconnect.


Hello everyone and welcome back to another Ethan Journal video. So today's video is a really exciting one because today I'm going to be talking about the reasons that I use macOS, why I choose to use Apple products, mostly Mac and maybe down the road I'll make videos why I choose to use iPad and iPhone and stuff like that. But in today's video I'm going to be covering some of the main reasons why I choose to use a Mac. Now I did make a video about this like over a year ago but it wasn't really that in depth and that's what I'm going to do in this video is cover this more in depth and more or less explain more of the reasons why I choose to use Mac over Windows or even Linux which I did use for a while. So further ado let's get into today's video. Okay so my first reason is the Apple ecosystem. Yes, pretty much anybody who talks about why they choose Apple products is because of the ecosystem but the truth is it really is the ecosystem. The ecosystem is truly what makes Apple Apple. The fact that they are making phones, tablets, computers and now smart assistance with HomePod it basically gives Apple a monopoly on people that buy one Apple product, the one that buy the other one just like that and that's what I had the same experience with. I've been using Mac since I was really little because my parents have always used Mac and Apple devices so I got used to it at a young age and then I've used my Mac for a long time. We had a family iPad which we actually still have. It was the Apple iPad too but we don't use it as much anymore but honestly I really liked Apple products at a young age and from that time I wanted an iPhone so the time I actually finally got my first phone it was not an iPhone it was an Android phone and at that point in time I was the only person in my family that had an Android phone but eventually my grandma upgraded her phone and I ended up getting her iPhone success and that became my first Apple product, Apple phone that I got to use. Now once I had that I was basically irreversible from going back. At that point I was able to receive messages from my phone and they pop up on my computer it was amazing. I was able to do FaceTime calls, phone calls, I was able to receive mail from my computer and my phone and AirDrop was amazing and honestly that is what just stuck with me. At that point on I've upgraded I've been through four Apple iPhones. I don't really have a thought of changing. Yesterday I've been some bugs in iOS 15 that I haven't liked but nothing can historically be wrong that maybe want to switch away yet but honestly the Apple ecosystem is really what keeps me and just how the Mac works with all the other Apple devices and makes it great and makes it all work together. Okay so my second reason is Apple's pro suite of apps. Now even their free apps that come with your device like Pages, Keynote, Numbers, iMovie and GarageBand they're all great. My first few videos on this channel were all edited in iMovie before I switch over to DaVinci Resolve and just a few months ago I got Final Cut Pro and I absolutely love it. Honestly Apple makes some amazing apps that come straight with your Mac and then you can pay like $200 extra to get their pro apps which are also absolutely amazing.  Yeah you get MS Paint but Pages is much better for doing not only word processing but design. I use it to make my app mock-ups and a bunch of other stuff that I like doing and honestly Apple devices just come with everything you need right out of the box which has been their advertising slogan for many years back when they had their first iMacs those get-a-max advertisements was always advertising how great iLife was and the truth is iLife is amazing yeah it's not called iLife anymore or iWork anymore it just comes with your computer but it is absolutely amazing and it is a game changer and honestly that is another reason why I just can't switch away from Mac is because I built myself on all these Apple apps that it is honestly just unethical for me to switch away and find a different platform because I already own them and I know how to use them and I know how to use them well my third reason is kind of a controversial one but it's truly the Apple devices themselves yes they've had a great design at least they are starting to now for years Apple would just seem to have been abandoning the Mac which is why I was stuck on an iMac from 2010 for so many years until I got my M1 Mac Mini but honestly Apple took a big break as it seems from the Mac for like 2016 to 2019 where they just make awful computers with a butterfly keyboard and all these other things that just made these computers awful and they finally came back to it with their M1 now the M1 chips or the M series chips have honestly been a game changer for a lot of people including myself because it has helped me so much with video editing and design work and it's honestly been a great bang for the buck and it's why I chose to stick with Mac when I upgraded from my 2010 iMac Apple's products may be pricey but you really are getting quality from choosing an Apple product most of the time at least with my Mac Mini I bought this for 1200 bucks and I have literally had no problems with it it unlike when I had my Windows PC that I had for three years I was constantly having problems with it it was so easy to get like spyware and viruses on there I've never had anything go wrong with my Mac compared to what I had with Windows now I will say I did use Ubuntu Budgie for about three months or so while I was waiting to and saving up for this Mac Mini and it was okay the only problem that I really did have with Linux is half the apps that I was used to using I didn't gain access to at that point in time I was just standardized on open office and LibreOffice which were fine in fact I still usually break office to this day for some tasks that Apple's I works we just cannot do especially for database applications and like the spreadsheets and stuff like that and I still use it and I learned a lot of the skills for those applications using Linux but honestly Apple's just Apple and Mac OS has just been a great solution for me and I honestly just cannot switch away and that about wraps it up this was a really quick video because I feel bad for not coming out with any videos I took a break over Memorial weekend to spend time with family and work and other stuff and schools out so I'll be starting to make some more videos very soon and I'm really excited to get back to my YouTube channel and start making some great content for you guys if you guys like to see extra videos and support me you can head over to my patreon at patreon.com slash Ethan's journal all of your support really helps and I'm gonna be starting doing patreon live streams with Q&A sessions and playing some startups and startup company simulator and some other games if you guys want to see that so be sure to head over to my patreon and pledge for as little as $1 a month other than that I hope you guys enjoyed the video have a great rest of your day and goodbye