Good morning guys. Happy Friday. Welcome back to another weekend of my life log. I am about to work out It's 8 a.m. I don't want to work out at all, but I'm gonna make myself do it. My set is after voices It's so cute. It goes underneath one of their new dresses, too. Love. I'll link it below All right guys, it's 10 a.m. I'm now at North Park. There's actually a skims pop-up this weekend and As we know I peaked and I work with skims So I want to stop by say hello. Also these sunglasses. I've been living in them. I have them in black as well From Amazon. I have been getting great. Oh my gosh. You know what speaking of you can't really see But like if you have a lot of hair and it's thick and you those claw clamps like never hold all of your hair Like I don't know if you can even see this one I found some on Amazon that actually hold it like I worked out today in this claw clamp and it didn't like mess up at All I Love my outfit today. It's so fabulous Fabulous when you worry this weekend's gonna be so good guys. We're going to buy our Nelson Sabu's back That's my best friend. We have been long distance for six years. So it's a very big day We have Friday dinner tonight Friday girls dinner tonight pains in town going to lunch with her It's a really good time to be alive. You know what I mean? Oh, so this is like always my problem Sorry guys. I've been like really active on tiktok guys Um Like it's like my number one thing that I've been I think I'm a tiktoker now It's like kind of embarrassing that I think that but you know, whatever, of course, I'm gonna make a tiktok Also, I'm fully vaccinated and like the CDC said no more masks, but like I am I don't know about that one You know, I don't know. I don't want people to think a certain thing. I think that's a main thing It's not the science. It's more so like I don't people Certain things, you know Okay. Well, goodbye. Okay. Here's my outfit by the way my cowboy boots I Love these shoes like I literally feel the most me and these cowboy boots. I like didn't even grow up in the country guys They grew up in McKinney, Texas, but suburbs golf course community not the country My favorite mall of all time North Park Very successful trip. They're here until June 11th by the way guys. So you can shop in person I don't think I even showed you guys my actual outfit for today. So here we are I have this cute Actually, they should technically be part of the haul because I just got it from nasty gal But it's obviously from nasty gal. It's just really cute. Just like button up. You can wear it as a dress You can wear it as like an oversized like top Put it over something. Yeah, um, even like a swimsuit cover-up. It's cute I have my favorite boots and they're from a Lambert's line all chocolate brown. So cute Okay, we're gonna do a little haul I asked you guys if you'd rather see this in its own video or in a vlog and you said vlog Thank God. I agree. We're starting off with what I got from skins. Thanks first. This is row. I Love it. I love it because it's cozy, but it's short. So it's not like too much I'm just gonna wear this like pretty much every single day Especially once the closet room is set up because like I'll shower my clothes are all the way over there Look, I'll be in robes. You know what I mean? Then I got another body suit from skins This is the same brown color that I love from my favorite one that goes up. It's like a high neck Body suit, but I love this one I love the little straps, but it was super cute and I get so much wear on my other ones See, even if I was wearing like this outfit I open this up and I was wearing shorts or something will be cute and then what I'm probably most excited for I got my first pair of pajamas from skins and I Love matching pajamas. I have another pair to show you that I wore last night We're matching pajamas that's bad like almost every night. This is just fabulous Also so soft and Carly from skims was telling me that this is like her favorite thing that she owns from them And she like works there.  I mean amazing I will be wearing this tonight. Also look what we have here Our besties merch so cute. You guys can get this or the besties tote. It's a limited edition order an ASAP This is from our live show super cute. So I've been loving it. I've been loving this Laura Mercier Lipstick as well. This is in the shade coral clear I really really like it though. It's like a super pretty pink next up. We have nasty gal. I got these cute little white like sandals, I guess I thought these were fun. I'm really into these for their summertime. I know they're really big last year as well But I just feel like I'll actually get a lot of wear out of them if it if there's too much of a heel I will like never wear it like ideally I would be one of those girls who can wear heels But like I'm wearing cowboy boots and it was like 10 a.m. You know what I mean? Like it's just I want to be that person and I'm working towards that But in the meantime, also, these are just more casual like daytime, whatever. I just thought they were super cute I would wear these to Byron Nelson if I Was like more put together. Oh by the way, I'm going to Byron Nelson tomorrow my first time going a very excited I got this set because I thought it was cute and I love that it was a set So you have this little top which these are definitely going to be like really small I like one of our triangles off. It's basically so difficult bottoms cute You can never get bottoms that actually cover your butt anymore and like it I actually really don't like it But it is what it is and then this skirt, which I thought was so cute to like put over That's really why I wanted to get this set Because I like the skirt alright, so this is the suit on without the skirt Okay, like I definitely have under boob It's definitely like not the most coverage on my top, but I really do love the skirt I don't really care for like laying out and stuff Also, I would just wear this skirt with like other things. So I think it's fun Another one of these I picture these honestly even with like cowboy boots and like a cowboy hat and a cute Instagram You know what I mean, but love This is like another one of those weird tops That are really in Alright, this is the last one. I really love the print of this. I think it's super cute I don't like how it's super like wide on my chest I don't know how much more I can like play around with the strings and tie that but I like this fit I just think it's cute So comment below which bathing suit of yours is your favorite and also if you have a larger chest the best place To shop for some suits I'll do an entire haul on it if I need to Because it's kind of annoying and then I got this which honestly looking at the top like this is gonna be way too small But like the good thing about this is that I go out on my rooftop of myself all the time So it's like not that big of a deal if it's like really showy because like it's not really like I wouldn't wear it Like tell the places and then I thought these bottoms were like really cute I just liked the top of this and the sleeves But I don't remember there like not being like a literal top, you know, honestly really pleasantly surprised by this top fitting I just think this suit is so cute Look, I love the sleeves and almost makes it look like a lot like nicer almost and then the bottoms are just really really cute I love the cut. I love the fit. I actually really like this too. I just feel comfortable in it You know what I mean? I got these cute matching pajamas, which I you know always steal cute stars Love that what I liked about this that I thought was like interesting is that the top was almost like cropped like it's almost I don't know it's cute. I wore it last night. Very cozy. Oh, this dress is so cute And I live in dresses in the summer, so I was really excited about this I'll probably wear this this weekend because I just like loved it so much and then okay I got this blazer, which I'm not sure how I feel about it and like I know myself I know I'm not gonna return it it's like a wool blazer and like first off it's summer So it is like a little bit too thick But it makes me think I actually might like layering this with different textures this summer if I'm like going out to dinner And it's like colder at night or which doesn't really happen in Texas or if I'm like on vacation or something I'm gonna give it a chance, but I'm not like crazy obsessed with the quality But again, it's not like this was like pricey But any means there's a few blazers that I really really want to go right now, but they're on They're out of stock, so that's unfortunate and I have one more thing two more things to show you from wave clothing I already wore two pieces. They sent me last night I'll put that on the screen super cute a girl literally yelled at me outside of my car because I just got in it and asking Where my outfit was from so like that's how you know to get off it, you know Um, I also bought these which honestly in another lifetime would be perfect to word of honor Nelson They're like kind of just like cute in like 70s. I love like the flair with them I would love these with like the white sandals And then this top I just like love this vibe It's just like super cute even wearing this over like a swimsuit. I don't know. I'm into it I got this cute dress again. Just me and dresses in the summer because even this with boots So cute that is my haul for today. I hope you guys enjoyed the print for the kitchen has arrived I'm not sure how I feel about it. I wanted something that was more like this That was just more calming and chill Because I've had the art prints for so long and I'm just so ridiculous like I realized my bedroom brings me so much peace Because it's very neutral and calming and just very zen So I wanted to do that down here as well And then the thought is to have the gallery while going here and obviously replace that light Going down the stairs and that's really fun and add something to it I know that a house is like pretty boring with the neutrals right now You guys are not up to date. There'll be an ottoman replacing that chair. I'll eventually get like a rug I'm gonna replace these chairs and then I think it's gonna be like a lot different Especially when I have those prints here. Hello, Coco. Do you need something? But then I thought about just getting a really big Like black and white photo from my society six in a white frame and having that here But I don't even think they have honestly a big enough one And I think at the end of the day like this is probably what I will like the most I got this from a great canvas. I think And I do like it a lot like it's exactly what I was looking for But I just don't know because I think I need the new chairs and I just like need new stuff in here to really decide Let me know what you guys think I just texted my mom and she's an interior designer and she's the one he told me to get that and I think like long term That'll probably make more sense. I've even thought about getting something in the corner He actually I won't do that. I don't like that But I need to have this hung Glow screen sunscreen is my favorite. I'm wearing on my face today. We're at every single day It gives you the perfect glow It like makes me not even wear makeup because that's how good it makes my skin look they came out with glow screen body So I'm gonna try it for the first time. I'm expecting it to be like life changing Okay, I got some comments from people saying that it has like sparkles in it And I actually kind of see the sparkles a little bit. There's a difference between my two legs I don't know if you can tell I can definitely tell in person this light looks so much better I will link this below, but like I swear by this line of sunscreen. Yeah, my vote. This is in I really like it All right guys and going to Chelsea corner. It's on McKinney Avenue in Uptown my first time coming here It's very popular to sometimes flowers. That looks cute Really busy Back from lunch with Peyton, it was so good. I didn't really vlog when you talk for three hours Peyton and I have known each other. I think for like five years now. We randomly met and got Alfred one day We just like met on Instagram. We're both from Texas and we're like, okay amazing And we've been friends ever since she's one of the coolest people if you don't follow her She's just so interesting. She's just like so interesting that she always has something to say in her stories You're like unreal. So like when I get when I get to hang out with her I'm like, okay, please tell me everything because like I just need to know, you know Anyways, it is almost four now. Um, so I'm gonna get a little bit more work done and then I have dinner plans tonight I mean, I've been in like girls dinners on Friday nights, which has been really fun My editor just sent me over a little trailer So I'm gonna check that and then just get some work done and then I'll probably see you guys when I'm getting ready for dinner I'm really excited for this weekend because Byron Nelson's gonna be so fun. And I don't know I don't know. I just am really actually happy and like content and really Like fulfilled with who's in my life. I'm just in like a really good mood and I'm also just like really happy So it's a good time. Okay guys I have to give you a look an honest update because upon Like looking I don't know like okay The dogs were really playing with my clothes it seems maybe I'll wear this tonight To give you an update on the sunscreen I did get comments from people when I talked about it saying that it has like sparkles in it And like I don't know if you can really tell um There's definitely like sparkles, which I don't personally love Like I don't I don't know. I don't know. I don't know I don't know.  This is the Coziest thing I've ever worn. I will actually be living this this. Oh my god. Like it is so cozy. Hi girl What's up? Yes, I get it I get it I mean guys like I'm literally dying for this Okay, I've changed. I feel like my vlogs are basically like out for the week videos now I'm wearing again my favorite jeans from princess Polly. I'm gonna get even more pairs I already have a few but like I'm just obsessed My skims body suit that I got today. This blazer is nasty gal These sneakers are one falls normal rings and jewelry and in my hair because it had been up in the clamp for a Bit my friends they're about to pick me up and we're going to dinner Morning everyone, I'm currently getting ready for Byron Nelson making some coffee. Have my cold brew My almond milk Very exciting, but if you don't know Byron Nelson the golf tournament and it's in Craig ranch Which is actually in my hometown So Sav's stepdad got us tickets and it's very exciting I've heard from like everyone who's gone before that it's just so much fun So I'll be vlogging on my phone today, but I'm just making some cold brew. Okay, like I barely have any left in here I'm making this coffee right now because I really want to try out my new glass straws Emily's pizza nado told me to buy them and I trust her Isn't this glass straw just looks so fabulous It really elements the experience you guys might be asking can see what is in your Byron Nelson bag so far I'll tell you now that I make such a golfer, you know Actually, my mom and I are gonna start taking golf lessons. So that's fun a clamp in one of the clamps I was telling you about I'll show you the box of the other ones. So I really didn't get it ready, but it's fine. Um These are the big hair claw clamps. I'll link them below I like them a lot more than the other ones I have because this will actually hold my hair like extra face sunscreen So much liquid IV because we're gonna be outside in the heat all day And then I'm in the middle of applying sunscreen even though. I don't think this is my favorite sunscreen ever I am gonna wear it today my sunglasses my lip products keys this mask cash guys like my bag is like legitimately all Lip products like so that is how what's in my bag video another outfit shot guys My dress is from wave clothing. I showed you guys in my haul I'm wearing my white Puma sneakers because I feel like these are the most like I don't know golf tournament appropriate I just picked up coffee Well, I got matcha for me and Sav and she ordered a venti cold brew with an extra shot of espresso Pray for Sav y'all. I have no idea what's going on with that girl. I was gonna try the matcha order Oh, I just drink it. How do you feel? You're not a monster person, so I don't know why it's not bad I mean, I wouldn't personally get it but I wouldn't say no Look y'all because you got this which is like normally me on like a weekday, but like Yeah, I'm like a weekend unless I'm like not thinking about this like I that's just like not what I'd go for I would go for that before night out if I'd like been out like tonight I might have to grab that before because we're probably out tonight, right? Okay, thank God. Um, okay, let me tell you guys about this order. I've been trying out different Starbucks orders on Tiktok and someone sent this in. It's just an iced matcha latte I get oatmeal with it, but they were out. No surprise here with peppermint. It is so good I know it sounds kind of gross, but it's very good. We have really important things to talk about for the summer for the rooftop I was talking to you about this this morning, so I could get another blow-up pool What's the state of the earth? I'm gonna find that video and put that in here. That was last summer I thought it was gonna be like a literal I don't know what's worse the fact that I bought a net for a Adult-sized blow-up pool or the fact that it a fish net. Yeah. All right guys, we have the pool cleaner Let's see sad. This is not real. Oh wow This is great, thank you. You're really doing a great job Or Yeah, somewhere in there, it's okay. Yeah, and then there's another thing. There's like blobs I've seen cut looking not like a blob from a camp, but like Yeah, yeah, you like put water in it and like yeah, it was like kind of colorful Yeah, I'm sure they're different designs, but like do I do I think I did the pool? Or that But that's gonna be easier to clean. It'll be I'm leaning more towards the blob Okay, I think we're gonna have to do some shopping. Maybe tomorrow Okay, well no one cares about my rooftop thing guys. I'm just To be clear making this summer the best summer of everyone's life I mean, it's like you in like 27 on my closest friends. We're literally having the best summer ever I have so many plans tennis clubs next week by the way We've made it to Byron Nelson and I just have to document this. We've only been here for like 40 minutes I'm walking the whole time. Yeah, I know and so I was already complaining about her If you go to Byron Nelson or like any golf tournament like and you wear wedges Blisters I can walk like a few people can do that probably like dolly part in the Michelle Obama And like I just would never you are sweating.  We're going to write right up for breakfast So I was never been it is our new favorite coffee shop in Dallas. It is so good. It's right by Nottingdonkey It's an uptown. It's so good guys this rain Awful done. Have you noticed what I'm wearing? Mottling uptown made for a brother uncomfortable It actually feels like the rainforest cafe. I finally got a matcha. I added peppermint in it. It's like my new thing with matcha. It's so good. Oh My god, the best sandwich. This is my third time getting it in the past week. So I've got a breakfast sandwich This is the best thing I've ever had in my entire life So unfortunately now it seems as though we're under tornado warning And in this situation like we're definitely supposed to go downstairs But we're also watching the Tiger Woods documentary And Coco's so comfy and like here's it like You can literally hear the sirens. Oh, I know I've heard them before You should have been here Dom's first tornado warning. Oh my god. I wasn't It it was not good. We obviously were fine, but it was not good for me. Yeah, I think it's fine like Guys like it's just unfortunately like pretty like common. It's very normal here If you live in Texas every time there's a tornado warning, do you actually take cover? It's worse in Oklahoma, right? Everything is No offense guys, I just don't like I don't know Those are like the legitimate sirens. Yeah I don't hear a tornado coming. I don't sense that There was a tornado that like took out like a like streets and Preston Hollow like two years ago I actually that is literally my like so Terrifying this is actually a really good question guys. What sort of drills did you have to do in school? Like we need to do Fire drills tornado Yeah, that was it Remember Were you in Spanish class freshman year during that one? I have a very vivid memory of like we were in a push during one of our like the security gun in this channel Yeah I just remember us doing some sort of drill freshman year of high school I just remember that like someone was like texting me really mean things I wasn't sad I ran into the person recently and um They were like it was like so I literally I never met them before didn't know them at all. Um It's still to this day. I don't think I met them technically speaking No, and um, I just remember it was like us in our Spanish class walking and I was like guys what's happening So anyways comment below what drills you guys did in school growing up because I'm sure it's it's obviously different Yeah, but they're our age some probably Also these clams guys It actually holds your hair like all of it. Okay, it's officially Sunday night. We laid around I cleaned Got work done. We're back on the couch Yes, we did um we're watching king of statin island. We just ordered Thai food That is it for this weekend of my life vlog. Hope you guys enjoyed comment below any video requests You guys have I love you guys so much subscribe and I'll see you soon. Bye the first thing we do We we interpret labs using a thinner Reference range so anybody that's listening right now will know. Hey when I get my Y lab I'm being compared to this reference range in my lab Well, we get that reference range from a statistical bell curve average of people who go to lab


The Ministry of Tourism invites public comment on the Tourism Development Bill. In the current Tourism Legislative Framework, there are many pieces of legislation guiding the development of the tourism industry, including the Tourism Incentives Act, CAP 15.30, Tourism Stimulus and Investment Act, CAP 15.03, and the Tourism Levy Act No. 11 of 2020. The Ministry of Tourism seeks to establish a single but comprehensive legal framework for encouraging and advancing development in St. Lucia. The deadline for comments on the bill is July 31, 2022. For more information, visit the following slu-tourism.govt.lc, facebook.com, min-tourism-slu, and slbs.org.


What's up guys welcome back to the vlogs today's vlog number five. I'm really happy. I get to do these it is Officially spring. They were still snow on the ground, but I wore a heavy hoodie today, and I am sweating But I'm gonna go and get a smoothie in a minute from McDonald's probably a mango and pineapple So you guys are gonna see more of that in these vlogs as the weather starts to warm up, but I'm not gonna share Any of it today. I'm not gonna be able to vlog until later tonight, too But I do want to give a quick little update on the vlogs themselves And I know I did say I'm gonna be doing a vlog every two weeks I've actually I decided I'm gonna do it every week for at least for now until it gets too much because I do have a Lot on my plate and then I'll put it back to the two weeks Every two weeks having a vlog it so for now it is gonna be weekly vlogs, and I hope some of you guys are happy about that I know they're not the greatest vlogs I know I don't do much, but I'm just I'm using my vlogs to kind of give you guys a behind-the-scenes Look to my videos to my planning to my editing to Everything I do even whenever I go out to the post office, which I literally just came from post office I didn't want to vlog any of that because I didn't expect to get anything today, and I didn't so nothing there I am going to actually I'll give you guys a quick update on my tooth because I mentioned in the last video So I'm not gonna make an appointment at the dentist yet. It's not hurting anymore The pain went away. So if the pain comes back and the pain stays for a little bit longer I will make a dentist appointment, and I will do that and you guys might see that in the future future video Vlog or whatever, but for now. I'm not even gonna bother with it. And also if you guys seen in the last vlog I left my keys at home for the post office had to go all the way back So I actually went to go and cut them today I actually went that day in the place. I went to didn't cut them which was my work But I went today and they were sold out So I'm gonna keep checking back so you guys are gonna see that in the future video But oh my god I need a spare key so bad just because these aren't my keys so I need my own personal key And I'm gonna be getting that hopefully soon as long as they can get them back in stock soon But as I said, I've already went there already went to the post office already did everything really that I had to do So I'm gonna give you guys some I don't know behind the scenes stuff of some of the editing I got a lot of videos to edit I recorded a lot yesterday. I do have to prepare for a video today I got to record that video today. I got to edit this vlog once it is done So I do have a busy day on me, but to me it is gonna be a great day and hopefully I know this intro is long, but I kind of gave you guys updates on some stuff So hopefully you guys stick around for the channel for my vlogs especially because that's a behind-the-scenes look and I know You guys get to see a lot of behind-the-scenes look of Everything I do in my videos, but this is more of a personal level level. So I hope you guys do enjoy that I did get something from loot crate yesterday came directly to the house. So actually once I get home I'll probably show you guys that first. So I'm gonna head out of here because I'm really sweating It is still snow on the ground. It is officially spring and it's warming up quick But I'm gonna head out right now. Sorry for a long intro I did guy give you guys a lot of updates, but I'll see you guys in a minute. All right, so I am home I'm gonna go in my room now and show you guys what I got yesterday to add to my collection All right, so I got this 1940s Captain America replica shield on a little stand. It is metal. It has a little bit of weight to it Really cool, and I also have the first Ash in comic it is issue number one it is a reprint, but it's the first appearance of Superman And I knew I can't find one this good a condition for the price I paid So I decided to get a reprints even though it is a reprint. I'm glad to add it to my collection I don't think I showed you guys this, but this is my Pac-Man bank Pretty interesting. So I'm glad I added those to my collection All right, so right now I'm gonna go and record a nether video It's probably gonna be up way before this vlog and that is the WWE 2018 Hall of Fame my reaction to who's getting inducted this year. So it's pretty interesting And I'm gonna go record that and show you guys a little bit more of what I'm doing for the night All right, so I just finished recording my video and that is the WWE Hall of Fame class 2018 My reaction response to who's getting inducted this year I'm really happy for my Academy. I wish you got inducted next year just based on now WWE seems to do One main guy each year and that kind of pushes everybody else back And I think my can you deserve to be a main guy that gets inducted? But I'm still happy that he gets inducted nonetheless.  I will just repeat that. I do I said it early in the vlog Until it becomes too much or I might just push it back and do every two weeks. I don't know I'm still trying to work it out comment down below what you think should I do weekly vlogs or every two weeks? Which would you guys like to see more?  I know it's my new biggest addiction Besides any other gaming or making my YouTube videos. It's a great hobby. I love YouTube, but I'm gonna be playing some fortnight. I've really I'm not level 29. I know I'm not the greatest But I've advanced so much in a short period of time that I'm getting within like the top five almost every game now and I haven't won yet But I'm on the road to I keep getting second place. So we shall see hopefully I will record that and make a video if I win But this is gonna be my night along with editing Doing a bunch of other stuff. So comments down below what I should do for my next vlog I know I've mentioned that a few times in previous vlogs. I'm sorry. I didn't give you guys much Contents today. I did share a little bit of my day with you and that's what it my vlogs are about They're not always about me doing something Every single vlog, but they're about me sharing my life my opinions my thoughts and giving you guys behind-the-scenes look to Pretty much my vlogs my days my life Everything so hope you guys did enjoy if you did please smash that like button I will see you guys in the next video. This is takedown Taking off peace


So this morning I did a presentation about making your own languages with Racket and about how it could be useful for creating data structures and data languages but are useful in a specific domain and this afternoon I wanted to take a bit more time to make you try by yourselves making a language. So I wanted this session to be a bit more interactive and so if you have working computers you can try live and see if you can do something. The code is on this repository on GitHub at umoe-slash-first-dem 2019 talk and there's a workshop directory. So if you pull that repository and of course you also need to install Racket. It should be in all the good package repositories around there. You can get to to this file once you pull it and and we'll see how we can make our own language. The first thing you usually do when you create a language I show I have already shown those lines this morning so this is the most simple way to create a language in Racket. The way it works is Racket has modules and every time you import a module it executes some code on this module before importing it into your runtime environment and running it and during this time you have the ability to read the code you are importing and modify it on the fly before it arrives into your into your code your running code. So the way it works is that any Racket module you create you just create a file and by itself it it can be used as a reader for over modules. So any Racket file have the ability to read over Racket files and this is the kind of module you want to make so there's a the magic is happening here. This function it's actually a macro but so this macro is the macro that is being called when you import a module and in this file we're going to change it so that it does what we want. So we are modifying the stand our way Racket works by putting our own code into this function. So the way it works is we just rename the function while we provide it. So providing a module is the inverse of importing it. Importing a module is done with require you just say okay for for this module I need I need this I need this I need this and providing is the opposite so it declares all the all the the functions and the stuff you're creating your file and you want over over files to use. So if you provide something that always already exists it's going to be replaced by the one you provided. So here we create our own module begin and we provide it as this name so it's going to replace it into the environment and the module begin we write is really simple it's like only those lines. It's just this is the way to declare a macro in Racket and in scheme. So we declare a macro that is called module begin and takes an expression as an argument and we call the actual the real module begin with our own code here. So this is the part where we we change the way module begin works is we just call the old module begin with our code and the code is doing only one thing. It just provides some variable that I called the ranch. Yep so there you go. You got it. So it just takes the expression that is inside the module and puts it into a variable and provide the variable. What's happening here exactly is I'm going to show you interactively in a moment is that the data file I want to process is this one. So this is the this is an example of a simple language I just made. I designed this language so that you can create a ranch and have ponies in it and describe the battle cries of every pony and so this code is going to generate some function that I can call. So I'm calling them just right there. You just call the ranch and you give them the name of the pony and it should print their battle cries. So if I do this just there I call racket on my main file and as you see they're just trying out loud what they want. So the way it works is that I want to take this data and transform it into actual code. The easy way to do that is through a macro. So I just match for I just create for this example I create only one macro the ranch macro and so the ranch macro is going to run and analyze stuff in there and generate some code and it happens just there. The macro is there so I define a macro called ranch that takes a syntax and so this part is a bit hairy but you get used to it when you work with a syntax boss. The macro can have two keywords ponies and pony. Every time it stumbled upon one of those keywords it just doesn't care. It's just therefore as an information that okay we are in this structure this is the pony section that's it and the way it pattern matches the data we saw is there. It looks out for a section called ponies in which there are any number this is any number the three dots there any number of the section starting with pony then a name as an ID and a battle cry as a string. So this line here describes the structure of our data. If we can look at our data again we have a structure starting with ponies and then any number of structures starting with pony. So here we start with ponies and any number of this stuff and we want to replace that. Oh yeah sorry and you want to replace that with code. So the replacement part is happening just there. I just create a function that takes a pony name and switches over it and if it equals the pony we just met then it returns the cry. Over rise this pony does not exist and we'll be able to see in real time what this macro is doing step by step using a very handy tool which is called the macro stepper and if I step it's gonna open up yeah. So this tool allows us to see in real time the macro over the code. If I step it's gonna define a module it's gonna define some stuff for the racket system and then at one time it's gonna reach our code and there okay. So this is before the macro hits and this is after and we can see that it transforms our structure into a lambda with a cone and an if statement for every pony in the data structure. So in C or C++ this would be really efficient because you're transforming the data structure into actual code with direct inlining of the data. Here the condition is directly in line for every piece of data. So you actually generate code and a big if and it happens that the cone keyword is actually macro so if I step a little bit further we're gonna see the cone macro transforming our code into if statements. So here the cone defined into into racket is transforming to a bunch of if. So once this is compiled into actual binary it's extremely efficient it's not it's not data it became code it became compiled code. And so when I discovered this I thought wow this is like the the best advancement we have in data oriented programming programming systems. It's like we can write data and have actual code at the end without anything in between. It's like your data is your code. And so I started to to look at how I could create languages for for my needs. And so that's why I'm doing this talk right now it's because I thought it was really exciting to just write just express what you want from the program from a data point of view and get actual code in the end. So this is the the the trickiest part to understand it's the macro here but there's a lot of resources on the internet from over racket programmers. There's one that is Greg Henderson he has a really great blog about writing macros in racket. So you should really check that out. And at first I thought I would stop there and make everyone try this kind of stuff but four minutes before the the talk started I got something else working here. It seems you might be missing package. Maybe. Yeah so you have an issue with. Yeah. Okay. So he's wondering if there are any dependencies missing. And I thought I would I was using known but maybe I'll check that out later. Yeah. Oh yeah that'll work. Yeah. And so right now the language we created looks like S expressions. So it's the standard way of writing least languages. It's just parentheses everywhere and atoms between between them. And racket can go further and provide you here I'm using the default SX reader but you can actually use any reader you like you can create your own readers for syntaxes that are not S expressions. And so I created another example that the other example requires dependencies external dependencies. I changed the language into that. This is completely arbitrary. I could have used any any way of writing this. I just thought it was cool that way. So if I want my language to look like this and then make it so that it generates this then generates the code we saw. There's a step further. We can write a reader and a parser. The thing we wrote right now was the expander. So the expander is using macros to expand the code into bigger stuff. The bigger functions as we saw in the macro stepper. But there's over layers and one of the layer is the parser. And there's actually a language called Bragg which I'm using here. And this language allows you to define a syntax for a language and use that syntax as a parser for your language. So here I wrote a syntax. And so I wrote that a wrench can be an optional new line then a marker for wrench begin then a new line then a number of ponies then this is yeah this is a typo. Then wrench and then new line. And then ponies is any number of ponies and ponies is any number of spaces then extra extra. And so this syntax is actually a rocket language and this gets processed and transformed into a parser. And it generates exactly what we saw there. It generates this kind of stuff. And the way it works is we have to define what wrench begin is as a token what pony begin is. And so it is done in the reader. And so the reader overwrites an over part of rocket. Instead of overwriting module begin we overwrite two over functions read and read syntax. And those two functions are called when you actually read something from a file when you import it. And so you can define written text to be a parser. And here what I'm doing is so this is the parser I defined. When this gets compiled it is compiled into a function called parser called parse. And in the reader I require it so I get parse. And as you can see we can visualize the dependency. So parse is coming from this parser. And so I can call this function to generate tokens from my file here. And I define the tokens here using a lexer. This is a new line. So I define that every time it finds this character it generates a token that is a new line. Every time it finds this suit of characters then this is a wrench begin. Every time it finds this this is a wrench end. Every time it finds two characters that are alphabetic two or more characters that are alphabetic this is an ID. And anything from double code to double code is a string. And so by defining your tokens there you can then use them here to define what the syntax of the language is. And it will take this and process it into this. So this is a really powerful system that allows you to define anything you want. Any language you want. Do you have any any questions about what's going on or is it okay? Yeah, so here what I'm doing is oh yeah. So how am I using the parser here? How am I requiring it? So one of the issue with the Bang-Lang syntax is that it allows, I cannot put directly something like that. Let's say my reader.orca team. It doesn't work like that. Only actual modules are authorized here. So I need to create an actual module. And I make it here. Let's see. Okay. So this file is just declaring that the reader is workshop slash reader. And so workshop slash reader is our reader here. So the way it works is you just have to define a special module that is called a reader. This is something a bit specific that I had a lot of difficulties working with this at first. That's why I'm showing this as an example right here. And I hope a lot of people trying to do this will not have to struggle with this as I struggled. But yeah, right now I declare that the module Pony when I require it, when I use it, the reader is going to be my reader. That's it. So this is all I can show you about this Pony language. So I made a lot of over language if you want some examples. More complex ones. And I encourage you to take this example and push it further and make your own language out of it. You're free. Yes, there's a question. Yes, I tried it. So yeah, I'm going to repeat the question. Is it possible to create some indentation based languages like Python, for example? So I tried it. And it's actually possible, but it's a bit complicated because Bragg, as I showed here, will not be enough because indentation based languages are not regular grammars. They cannot be computed using the kind of grammar I showed here. You need a recursive grammar. And you need to keep the state of what is the number of indentation. And this is so this is getting a bit more complicated. It's possible, but it's more complicated than the example I got there. So one of the languages I made with this was it's one of the first languages I made with this was a language for generating emulators from CPU description. So you just describe what the CPU is, what are the operations, and it generates an emulator. I called it virtual MPU. And let me see. I'm going to show one of the one of the files. Let's see. So this is the kind of file you can create with this language. You declare an MPU, you declare a name, then you declare the registers. Okay, so VCPU has an A register, B register, a status register. The stack pointer register is 16 bits, stuff like that. You can precise, you can declare that the status register is SR and that all the bits mean this. So there's one bit that means it's the carry, one that is the overflow, et cetera, et cetera. You declare the interrupts, and then you declare all the operations. And I made my own somewhat macro system inside this operation stuff. The operations are starting here with the branches. So those are the different styles, the different kind of branches in the emulator.  And some of them required, some of them add exactly the same way of working for a lot of stuff. So I made macros, for example, there, subtraction, addition. And you can declare them on the top. So here, subtraction and additions are described here. You just say that you want a result.  So this is an example of the kind of language you can make. In this one, I prefer to use S expressions and not a special language. But I also did, in the same project, I also did another language, which is actually an assembler. And so I created a language that passes assembly language. So this time, I needed a special reader and stuff like that. But this code right here is parsed and transformed into an S expression and transformed into racket functions and then executed. And it's just assembly language at first. So for this one, the reader looks like this. There's like boost tokens. There's like a token for numbers. So numbers can have a dollar before them. They can be in hexadecimal format. There's comments. So there's comments with a semicolon and stuff like that. There's a special section for data. You can put data in there and stuff like that. And the parser looks like this. So an assembly is just lines, any number of lines and new lines between. And then a line can be an instruction or an assignment or data declaration. An instruction has a tag, an optional tag, then any number of space, a mnemonic, then any number of space and operands and stuff like that. So it's like 20 lines and it parses assembly language. And it took me, I've been working on this project for maybe one month and I knew nothing about how BRAG works before. And I just learned how it worked. And in 20 lines of code, you can generate a parser for assembly language. What can I say? Yeah. What is actually expanding into like the assembler? Does it expand into packet code that works? So right now it's assembled, it's expanded into binary. So the way it works is it parses the assembly and then generates opcodes for all the assembly stuff and then put that into a file and so it compiles a binary. And I put it there, multiple options. You can generate the Motorola SREC format or the Intel format or stuff like that or direct binary and stuff like that. So yeah, it's a working assembler for the CPU I wanted to emulate. Try to stand. So Racket, so I wasn't in the personal talk. Racket is like a variant of the list which defines your own language. So yeah, Racket is a list language from the scheme family which allows you to define your own language.  In this case, assembly, you parsed it and you're compiling it to what? So he's wondering how, I'm generating Racket code, but how does it compile to something else after that? So in this case, I think I can show the expander. So I'm going to show up the GPU. The interesting file here would be, there, okay, the assembler. So it is compiled into structures, Racket structures. So the, yeah. So at first it compiles into Racket language and then when the Racket language executes, it generates a binary. It generates a binary from the assembler. Like the assembly language is assembled into a binary so that it works on the target machine I wanted it to work. So what I'm doing is I'm generating code that is executed to do what I want. Instead of just executing some code, I'm generating code that then it executes. So I parse the assembly into a Racket structure. Then the Racket structure is parsed into Racket code. Then the Racket code is executed and the main function of that execution is write to a file. The first function that is called is write this to a file and then everything else is the data that I scrambled and mashed together. I could have written, I mean, it's a way to write an assembler. I could have written an assembler in any language. But the way it works in Racket is interesting because you define your program as a language instead of just a program. So instead of saying, okay, I wrote this assembler, you just run it on some code and then you get a binary. I wrote language that is actually an assembling language and so when you use it, you assemble into binary. So it's a bit like changing the way you think about programs. Instead of using programs, you create languages and you use those languages. It's the way Racket promotes. Any other questions? So what's the benefit doing this? One of the things I like with this is that you define, instead of thinking in terms of I need this program that handles this data this way, instead of thinking into pipes that you put together to achieve a result, you think into how can I best describe my problem? How can I best describe my issue? What is the thing that I want to do and how can I describe it so that it's obvious when you read the code what you want to do? And so Racket allows you to open a file and say, okay, so my problem is about bakeries. I want to generate a cell bread and croissant and stuff like that. So I have a bakery and it says a lot of stuff. So it has products and there's croissant and there's bread and baguette and stuff. And you start by thinking about your data and then you make it so that this data is actually executed. It's actually a program and actually makes what you want. So the benefit I find is that instead of thinking about making programs that takes data and outputs something, you just write your data the best way you want your data to look and then it executes into something. And that's what I found a bit magical about this way of working is that it redefines the way you program. It changed a bit my way of seeing programming and languages in general. I don't have to think about, okay, in Python there are those kind of data structures that I need to use or I remember with my colleagues we had really long talks about we should using this kind of classes and then this class inherits this class and then it generates this interface and we need to change the interface so that this customer can work correctly with the interface and this class interacts with this class. And it forces the way you think about programming is you have tools around, you have, okay, I'm working with C-shop for example, you have classes, you have reflection, you have a lot of tools around, but I think only those tools in your hands, you only think about how to use those tools. The only way you think about resolving your problem is using those tools. So if instead of thinking in terms of tools, we think in terms of what is the actual program and the data that I want to process and what do I really want to do, you write the language that gives you the best tools for this task and this could be a different language depending on the domain. So you still have to write the tools to translate your data into actions and what I like is that the way you write the tools is through a language. So they are not just tools, they are a language that you can make evolve and that you can document, that you can write specification for that language and that you can share with colleagues and that you can promote and that is adapted to your company and the product you're working on and the clients and it doesn't change completely the way we work. We still write tools and those tools are executed so that we get what we want.  I'm going to need to rewrite this from scratch. And instead if you have a language, yes, you will need to change it, you will need to adapt it, you will need to modify the way it works to adapt to the situation but it's a complete language. It's not just tools around. And so it has consistency, it has a logic behind and that's what I like. Anyone want to try Racket now? Raise your hand. Great. How do you use Racket in your work? I mostly use Racket in my free time because in real life I'm actually doing JavaScript. But actually I'm doing something that is in relation between JavaScript and Racket recently. I've been working on a project that has been started by a student at the University of MIT and he's been working with Racket a lot and he has written a lot of thesis about a lot of stuff in Racket. And for his PhD, he invented a language that generates JavaScript from Racket. And this language is really interesting. It's called Erlang there. And so this language uses Racket and generates actual JavaScript. So this is how it looks. So you write something like that and it generates this kind of stuff. And this is really interesting because this is similar to the work of the team at Babel. The way Babel works is that it's a generator for JavaScript and it pauses the syntax of JavaScript and it understands what you need when you want from your JavaScript code and it transforms it into other JavaScript code that runs on any browser. And Babel is an awesome open source project. And this is the kind of stuff that allows to do the same thing as Babel but using the power of macros in Racket. So everything here can be used as data, can be parsed, can be modified, can be analyzed so that you can generate different JavaScript depending on the situation so that you can generate JavaScript depending on the browser or stuff like that. And it happens that the Babel project has a system called macros that they wrote from scratch by hand and it works mostly the same as Racket. The only difference is that Babel is like a really big open source project with a lot of contributors so it's way, way more developed and there's way more features. But one interesting thing is that using Racket here, you don't have only access to the macros defined as Babel defined then. You have access to all the Racket environment and all the other languages. So one of the things I was showing in my talk this morning was an example from a language I'm making. So this was my talk this morning and there, okay, I'm going to put it a bit smaller. And this is an example from, this is something that I try to achieve. It's not ready yet but I'm trying to make something like this. I'm trying to make a system in which you can write API servers, web servers and this is, for example, a response from an API and inside you have code that fetches in the database. You have code that generates HTML, you have code that generates CSS, you have code that generates JavaScript, everything in the same syntax and in the same function. So you just focus on writing your complete website as data, as function, as just one function and then it generates everything, it generates JavaScript, it generates the CSS. For example, the style system here could be a system like less so that you can have variables inside your style and generate your style dynamically. And the same thing for the JavaScript part, it could be something like Babel. So I'm trying to use Erlang inside this part there. So I generate JavaScript but inside the JavaScript you can put some data, you can modify and generate some JavaScript code and stuff like that. So this is the kind of way of programming I'm trying to, I'm exploring and I'm trying to see where it goes but I think it's kind of cool to be able to forget that there are different languages and just write in one language with a lot of syntaxes to generate bits and bits from the server. Yeah. Is it more complicated if the data is not a tree? It's like a graph? It can be, sometimes it can be, I know some people in the racket community that are working on that kind of structures with cycles inside and stuff like that. The racket parser and the racket expander and the macro system is extremely powerful so it handles the cycles and stuff like that. I don't know in details, there are people with PhDs that would be way more able to answer this question but yes, it can handle data structures with cycles and stuff like that but I guess it would be a bit more complicated and you will face some issues. If nobody has any other questions, I think I'm done.


is a female. I'm the only female here. I'm hoping that that improves. That's always a challenge to get female in STEM. But thank you for having me. Today I'm going to be talking about how to build a production big data pipeline. So before I start, I just want to get a raise of hands so I can adjust my speed at which I'm going. Is everyone here familiar or have dabbled with big data? Can I get a show of hands of who is familiar and have dabbled with big data? Okay, perfect. Then my talk is, there's like three people in the room, so my talk is apt because I lay out the context of what is big data. So if you know all of your familiar, I'm going to skip, but it's good. So before anything else, a little bit about myself first to introduce myself to you guys, to break the ice. First things first, I love data, of course. So I'm in the right field and I'm in the right team. I have 15 plus years of software development experience, mostly focus on data. With the past five years, focus on big data. So for the past five years, I've been the engineering manager for what we call the Autodesk data platform, or ADP for short, that is the de facto data platform used by the entire company, the entire Autodesk. It's built right here in Singapore. Now it's being maintained globally, but we started building the infrastructure and the whole idea concept here in Singapore. So also last 2017 and 2018, I led a team of UC Davis Masters of Business Analytics students into a project of what we called System Incident Prediction. During that time, there was no term for it yet, but 2018 is the period where everyone started, they want to be able to predict system incidents and now we have a term for it, which is what we call AI ops or AI operations. What I'm passionate about, apart from data, is strategy, technology and innovation. Also teamwork and team culture, me being an engineering manager, I want there to be a good culture within the team. And I love sports. I love tennis primarily and skateboarding, both of which are on hold because I just busted my knee. But there you go, a little bit about myself. All right, so what I'm going to be talking about today, I want to introduce what is big data, what's the big deal about it, right? And also a little bit, I will talk about the architecture of what we have for the Autodesk data platform. Granted, it's a bit high level, but I think it's just appropriate for what we have here. And I'm going to walk through how do we build a big data pipeline. What is a big data pipeline to begin with? Why do we want to build it? So I'm going to talk a little bit about that. So I have reserved some time for Q&A towards the end. So if you do have questions, I would just kindly ask you to, I'm welcome questions, please do ask them. I would just ask you to hold it till the end so that we have enough time to go through the entire presentation. Is that okay for everyone? Cool. All right, first and foremost, big data. What is so special about it, right? So if we think about data, can someone, especially for those people who haven't touched big data, which is most of the room, when I say the word data, what do you normally think about? How do we, you know, what is the technology that we use for data when I say data? Databases, great. That's right. A lot of numbers. Okay, that's great, which is great because those are the two things actually, two illustrations that I have. People usually think about numbers. I kid you not. In this day and age, people do still use Excel sheets as their database. Honestly, I don't, I don't know. But yes, the Excel sheets and databases, right? So when we talk about big data, what is the difference then between big data and data? What makes it so big? Right? So let's think about this for a while. In 2019, this is what happens in an internet minute in 60 seconds. 18 point, my eyesight is not so good and I don't want to block the camera, 18.1 million texts sent, 4.5 million views. I don't know, 87,500 tweets, 347,000 Instagram. I'm not going to go through all of it, but you can see in one minute, a lot of things are happening right now. And this is largely due to the advent of the internet, right? So can we all agree then that if I look at the numbers here that this, this data is not small, right? This is, this data is big actually, this data is, is massive. So it is, actually it is, sorry, why I had that. So let me share some facts about data with you, right? Actually 90% of all the data that we have in the world right now were collected over the past two years. So it boggles the mind that this been like, we've had like what, more than 5,000 years of history, but the amount of data that we have has been collected in the last two years. That's just how it blew up, right? And by 2025, it is estimated that we would have around 463 exabytes of data. That is why I put this, just to give context, what is an exabyte of data? Exabyte is here. It is 1,000 to the power of six and there's a lot of zeros over this. So if I say it's huge, it's massive, right? So if we look at that and we agree that this is big, right? What are the aspects or what are the different areas of this that makes it big data? First and foremost, it's the volume. We can see 1 million, 1 million, 2.1 million, it's big. It happens in 60 seconds, just to say. So volume is quite huge. The velocity at which this data is coming through also is quite fast, right? In order for us to get 2.1 million snaps in 60 seconds, it has to go through at a very fast rate and it's not a rate that is easily handled, certainly by an Excel sheet, right? And then also if you look at the third aspect of it, it's a lot of different variations of data. It's not just texts, there are videos, there are music, there are pictures, it's a lot of different kinds of data. So going back to traditional thinking, right? Do you think that these are going to be easily handled by a database? Honestly, I think not, right? And not only, so basically when we talk about big data, usually big data comes with three Vs and these are the three Vs, volume, velocity and variety. But as more and more data are collected and as more and more people and companies are processing data, we come up with new Vs every day. So I'm not going to talk about all the Vs, but for big data, this is the three primary Vs that you have to be aware of, right? And so if we're going to talk about that, if we think about it, how do we collect and store these kind of data? Again, database, yes, maybe you can store it in a database, but it's not going to be usable. And then how do we process these data to find two more Vs? Veracity. How do we know how true the data is with all the fake news and fake information out there? How do we know if your data has high integrity and has high accuracy? And the next one is how do we unlock the value of this data? In order for us to unlock the value, we need to be able to do, to process the data, to do some analytics on it, right? To slice and dice and finally do predictive analytics on it, right? In order to do that, we need the traditional technology that we have used in the past is not sufficient. It may be able to store the data, you can store it in a database and a blob. When I'm talking about database, I'm talking about relational databases. Relational databases are not really good for these kinds of things because if you think about it, how do you store music? How do you store pictures? And storing is one thing. How do you enable processing on videos? How do you enable processing on pictures? Databases, traditional relational databases are not good enough. So let me share a little bit on how we are handling these kinds of big data in Autodesk. So a little bit of context. For big data at Autodesk, we are talking primarily about product usage data. If you're not familiar with what products Autodesk has, these are AutoCAD, Maya, Inventor, Revit, these are our hero products. We have over 100 products and so when all those 100 products have usage data, have telemetry data, we collect them, it becomes huge. So the good thing about the data that we collect is it's kind of semi-structured. There's some common schema that we slap on top of it so that it's easier to process. It's not like, you know, blob here, text there, video here, it's not. It's text so it's semi-structured and it has a common schema. It is also high volume, high velocity data and just to give an example, our average volume of data that we are receiving per hour is 110 gigabytes, uncompressed, and 73 million records per hour and this is growing. This is on average. At a max peak, we were receiving 157 gigabytes of data per hour and 110 million records per hour. It sounds big but it can be small so let's keep that in perspective and do a comparison. So in comparison, the average tweets per hour is only around 21 million, which if you do a rough calculation of a byte math, it translates to around six gigabytes. Six gigabytes is a conservative figure because in Twitter, although you have the character limit, you can also post pictures, you can also post like GIF. So even if you blow that up by 10 times to 60 gig, it is still only half of what we are processing in data, in Autodesk for our data. So this is just a perspective of the volume of data that we are processing. So to handle this kind of high velocity, high volume, semi-structured data, I'll talk a little bit about the architecture of what we've employed in Autodesk. So it's not a silver bullet that is able to handle this. You have to have data collection. You have to have a storage strategy. You would also have to have different tools to access and process these data. So I've kind of separated it into big blurbs. And I'm going to talk about the blurbs in a little while. But like I said, basically we have data collection. So we have two patterns for data collection currently. We're increasing the patterns. But basically what we want are ingestion patterns, not use cases that, oh, I'm going to ingest this. This is a use case that I have right now. I'm going to create one mechanism or one ingestion for it, but it's not usable. What we want are patterns, patterns means that it's plug and play. Today it's going to be used by this team and the company tomorrow because it's a pattern, it's a plug and play for another team. So right now we have two of that. We also have data exploration. So once we've collected the data and once we've cleaned it, we want to be able to unlock that data to the internal users. We want them to be able to do their crunching and their analytics based on the data that we collect. And ETL, for those of you who are not that familiar with data, it's okay. ETL usually stands for extract, transform, load. This is the industry term that we use for batch processing, basically. For the data that we collect, we want to do some aggregations on it to come up with different numbers that executives are keen on or are interested. And ETL is what we call. And finally, ETL and the data exploration serves into the data visualizations. So the dashboards that management see, actually there's an engine behind it. It's not raw data into a dashboard. That is in the realm of science fiction. So there's actually a lot going on before you can populate a dashboard. And I will talk a little bit about that later on. But first, let me talk about the data collection pattern one. So for us, like I said, we have a common schema. And so for each and every product, we do have an SDK. It depends on whatever programming language that they use, but we have a multitude of different language SDKs that they can integrate into their products to collect the data that we have and to collect the data and follow the common schema that we have. There are also some services, meaning internal services that we use to authenticate users, to authenticate licenses and all these things. They also instrument using our SDKs and they send data to our central data lake. So all of them are, this is an abstraction of it. There's a lot more happening in the middle, but ultimately it lands into AWS service called Kinesis Firehose. And Kinesis Firehose is like a queuing. It's like a message bus. It's like a message queue. It's able to handle high capacity, high volume data with a lot of reliability. So recently, our architecture was not as simplified as this. So when we built the Autodesk data platform, it was five years ago when I started out with the team. And five years ago, Amazon does not have big data offerings. So we used to have to do it on our own. We used to use Kafka. We used to use a lot of messos to orchestrate all our clusters, but recently we moved to what we call hug the bear. We want to make the most out of AWS services so that we don't have to develop this on our own. So now it's sending to a Kinesis Firehose. That's a pattern number one. And actually, on top, you have the users. So these are different kinds of users. So for data, I know it's like a kind of a big dump into you guys, but there is a steep learning curve for data. And I'm trying just to give you guys a high level understanding so that you can get into why I want to build a big data pipeline. So for big data, there are three, mostly three kinds of users. One is a data analyst. Data analyst, what he or she does is look at the numbers as it is right now and do some analytics on it to get some insights from the data that we have right now. Another role in data is data science. Data science is more around predictive analytics. Based on the data that we collect right now, what do I project the pattern to be? What do I project the trend to be? Therefore, if the trend is going to be this way, we should follow this trend and do these certain actions. It's more prescriptive. And last but not the least is a data engineer. What does a data engineer do? The engineer does all of this work on the side, on the back, to set up infrastructure so that the data scientists, the data analysts, and other people in the company can make use of the data. So things like this, a data scientist will not know how to integrate them together. Data analysts forget it. They will not know. So these are the users. I went on a tangent. But there are users. And for the SDKs that we have, we have what we call remote control data collection. So if you're familiar with, you may not be familiar, but there's a recent law that came out from EU, which is called GDPR. It's all about data privacy and data protection. So because of that, we cannot always be collecting data. It cannot be always turned on. There are only certain spurts when we can collect data. And it should be for a purpose. So this is what we call ethical use of data. And in order for us to facilitate that, we have this service called the Experiment Portal wherein this serves as the remote control to trigger the data collection to toggle it on or off. So once those data analysts say that, hey, there's this question that I want to answer. How many users is using AutoCAD version 2020 release this feature? They will do some things on the Experiment Portal, push that out into all the AutoCADs that are released out there. Those AutoCADs use our SDK. They're instrumented. They will start collecting data. In this experiments, they have parameters. I just want to know in the U.S. how many people, so they're going to limit the scope of it to U.S. So that's what the Experiment Portal is. It is backed by Amazon Aurora, which is a key value database basically. And our storage is Amazon S3, object storage. So that's ingestion pattern number one. The second data collection or ingestion pattern is just simple. Any unstructured or semi-structured data, we just need our users to dump it into S3. And we have an automated system backed by Lambda to ingest it into our raw data. So the raw data is a red bucket. And there's a different bucket for process data, again, because of GDPR, data privacy, and security. And also not just that, for processing needs in order for us to be able to process fast and return results fast, we need to be able to segregate the data. So the S3 buckets here, here, and you will see another bucket later. This is what we call the Data Lake. Data Lake is a standard industry term as well for big data. It's deep. It's murky. There's a lot of confusing data around there. You need to make sense out of data. So next, I will talk about ETL. So after the data has been collected, we need to do some processing on it. In order for analysts, in order for data scientists to be able to make sense out of the data, first and foremost, we have to clean it. Because even though we are collecting through SDKs, some instances can happen that causes data to not be clean. For example, some packets are lost, then data is corrupted. Or a Weisscrack engineer did not instrument it properly. It's expecting an integer data type and they send us a Boolean. So these kinds of things, they have to be weeded out. They have to be cleansed in order for data scientists to be able to use that data that we collect. So the underlying big data technology that we are using is Spark. And I will talk a little bit about why do we need a big data technology to begin with. And the underlying scheduler that we are using is Uzi. Again, I will talk a little bit later about why we need a scheduler. Why can't I not just use a cron job? So just keep that in mind. I will talk about it in the next few slides. But our big data technology that we're using is Spark. Our scheduler is Uzi. And what we do basically is the data analysts and the other data engineers in the company, they use these technologies to build aggregations. Again, like I said, what are aggregations, right? So when the data is in the data lake, it is deep and murky and it's huge. So in order to answer one particular question, you don't need all of the data. You need an aggregation, a group buy of that data, some number, group buy this field, give me this number. What are all the licenses that we are using? How many licenses are being used? You don't need what do you call that? You don't need product information. I just want how many licenses are being used. If I want to know how many licenses are being used per product, then yes, I need the product information, but I don't need the country information. If I want to know how many licenses are used per product, per country, then yes, I need the country information. So these are different kinds of aggregations to answer very specific questions. And those are what we call data cubes. Once an aggregation is built, it becomes part of the data pipeline and it produces a cube that is usable by data analysts and data scientists. And then we put it to the green S3 bucket over there. It is green because it's open to the public. This one is red because its raw data access is primarily restricted. And then after we do some cleansing, we do some aggregation on the data, we open it up for data exploration. So what is data exploration? Some analysts in the company or actually some product manager in a company. Let's say I'm the product manager for AutoCAD. I just released a new feature, feature A. I want to know how well is my feature A doing? How many crashes have been caused by this feature A in the product? So all the data is there in the green bucket, but how do I access it? I am able to access it through these technologies. Hive is our meta store. So it's basically metadata about the data. And then we use the Presto engine. Presto engine is developed by Facebook, but it's a columnar database, basically. And for big data, usually columnar patterns and columnar file formats and databases are a lot faster because they get rid of all the other things that you don't need. If I just want the country, I can get the column of the country. I don't need all the rows of all the data that I have and then filter by country. So it doesn't work that way. Columnar is the way to go. And also, some of them are data scientists. I want to do predictive analytics. For data scientists, mostly they use R or Python. We need to provide interface for them to be able to access our data and use R or Python language on top of our data. So that's why we provide that Hive. It's a JDBC interface, basically. So they can connect their Jupyter notebooks there. They can connect SQL workbench for SQL-like query languages and other things. But basically, these are the supported ones. And finally, after all the crunching, research, and exploration, we finally, after all that, get to the visualization. So for visualization, our de facto tool, BI tool that we support is Looker. And it is backed by Amazon Athena for faster querying. Looker is what we support, but we can also have Tableau. We can, oh, finally, a female. Welcome. Yeah. So Looker, as I was saying, Looker is a de facto BI tool that we support. But because of the JDBC interface, we're able to support Tableau. We're able to support ClickView as well. Yeah. So this is overall the big data infrastructure, architecture that we have to handle the product usage data. Right. So remember I said, why do we need a big data technology to process big data? Is it not as simple just as a database or querying? And why do we need a scheduler? Why not just Cron? So for the big data technology, usually, it operates on the concept of MapReduce, right? And why do we need MapReduce? So what is MapReduce? MapReduce operates on the notion of instead of bringing data to the processing, which can be costly, we bring processing to the data. Now, I give you a very basic example. For example, you're using, you're doing a word count, right? And you want to write an application for word count. Imagine that the file, your input is 10 gigabytes. If you write an application using shell script, let's say, you know, grep and then WC minus L, right? You can get a word count. That actually downloads the 10 gigabytes of data into your local or wherever, whichever environment you are running your application on. And that download alone takes a long time. It's 10 gigabytes. We're talking about 110 gigabytes here. So instead of bringing the data to the processing, what we do is let the data stay there. We bring the processing code to the data, because usually the code is small anyway. The source code for it is MBs at best, right? So this is what MapReduce is all about. So for MapReduce, this is what it usually does. So for example, a word count, right? There's XBB, CBA, XAC. Actually, there's character count. I want to count how many characters there are. So what they do is actually they split, oh, by the way, sorry, I forgot to mention, one other thing that is great with MapReduce is they can parallelize it. Of course, you can also write a script that, you know, has threading and all these things, but it becomes very difficult to maintain, very difficult to track. So for MapReduce, they do that innately. They parallelize it innately. So you have three lines here. For example, they will have three nodes, what we call partitions, right? And they will allocate like the first line on that node, the second line on the second node, and the third line on the third node. And you will employ the same script, the same logic for counting in all the three nodes. So what it does, it is on the first node, it will split. I have one instance of X, one instance of B, and another instance of B. So X1, B1, B1. Same thing is, same logic applies in the second node. I have one C, I have one B, I have one A, and so on. And the third node, I have X. And what happens is that there's a shuffle that happens in between to make sure that the nodes only, what we call the partitions, only have one character allocated them. In this instance, it's one character, but in a different example I'll give later. You can understand a little bit better what I mean. So for A, for the partition of A, I have two instances of A based from this once they shuffle. Same thing with B, I have three instances of B, two instances of C, and two instances of X. And after they combine and sort in a step that we call reduce, which is the last one, the reduce will actually give us the counts. So all together I have two A's, I have three B's, I have two C's and two X's. This is what happens in MapReduce. Now it's a very basic example, how do we use it? For example, again, we are collecting product usage data. Data comes from AutoCAD, Maya, Revit, and Venture. If I want to segregate their data, how do I do it? It's the same pattern. We will just partition it by product and count the number of whatever filter that I want to give it. Count the number of licenses per that product. It will operate in the same manner like this. Okay, so that's about MapReduce. Now, why Spark? Okay, so MapReduce, this one has been around for a long time. It started out with the technology that we call Hadoop. Some of you may have heard it. It's that elephant. So Hadoop actually, when it came out, it was a really great, it was a huge jump in the processing time of processing big data. It was a breath of fresh air. But actually, MapReduce has a lot of improvements that can be done on it. Although it's fast, they're always iterate and they'll always come up with something better. So why Spark? In traditional MapReduce, data sharing is very slow. Because why? Because in order for them to be able to process, read and write the data, they have what they call Hadoop file system. And that is how they're able to split into different nodes. They have to have their own file system. But in order for them to share that data for intermediate results, let's say you have MapReduce one followed by a second MapReduce in order to get to the final result. The intermediate results, they have to do IO. They have to write, persist it to disk. And that is why data sharing is very slow in traditional MapReduce. So it goes something like this. For example, data is on the disk. They will load it to their Hadoop file system. They will do the maps first. And then they need to write whatever they map on disk. So if you look at this example, they need to write this to disk first before it can be reduced. And that causes IO. So it spends around 90% of the time on serialization, replication, and IO just because of this. So what Spark does is that it came out with this notion of what they call, it's a shared memory. So what Spark does is that it processes data in memory. And that's what makes it 10 to 100 times faster. And they came out with this notion of what they call resilient distributed datasets or RDDs for short. So in between, instead of writing to disk, they have a Spark context. And as long as you are in that Spark context, you can share the memory. And that's what makes it fast. So in Spark, it removes expensive operation by introducing shareable memory, what I said, resilient distributed datasets. And this is how it then looks like. So we have an iteration of map first, and the memory is distributed, it's shared. And then the second map, and then the third map, and then finally, another thing about Spark is that in Hadoop, every map requires a reduce, even though sometimes logically you don't require a reduce, it maps reduce, maps reduce, and writes to file. In Spark, you don't need to reduce. You can map, and map, and map, and then finally reduce in the end. This is also an optimization. So Spark has a lot more optimizations on why it's a lot faster than Hadoop. And right now we are at Spark 2.4, or 3.0 is like in beta, but 2.4, there are a lot more. But I'm just talking about the biggest thing over Hadoop that they have on why it's 10 to 100 times faster. So this is why we need the big data technology. Because of the amount of volume, the volume of data that we have, it's very expensive and sometimes impossible to bring the data to the processing. We have to bring the processing to the data, and that's why we need the big data technology. So next, why do we need a scheduler? Why not just cron, right? So we are talking about, again, I need to remind you guys, it's big data. And for big data, sometimes you cannot, actually, let me take that back. Most of the times you cannot process in sequence. You have to paralyze it. So let's say start action, you can fork into three actions, and then those three actions further fork into three actions using cron. Yeah, possible, but is it the best way to do it, right? So we have what we call a directed acyclical graph. So it's just a fancy word, honestly. It's a conceptual representation. So we break down the words, a graph. It's a graphical representation. Directed means a single directional one way, and acyclical means there are no loops. So directed acyclical graph, right? It's single directional flow with no loops that's represented in a graph. And then it's most often used in data processing for a series of computations run on data to prepare for one or more ultimate destinations. And there can be more than one path in the flow. So an example of a directed acyclical graph looks like this. And it's very small, but this is taken from our production system. So here you can see it's a fork. In this particular instance, it shows this fork, and it's still running. Yellow is running. This means it hasn't succeeded or failed yet. So for the DAG that we are using, we are using Uzi as a scheduler. Okay, so enough about theory. Let's put it all together and let's build a big data production pipeline. So no, I don't have live demos. I will just share with you some scripts and some code on how you do it. Because in 30 or 45 minutes, honestly, it's we're not going to be able to cover all that if I do a live demo. So basically, first you create a MapReduce function. So I know, oh, it's a little bit small, but this is our production code as well. So if you can see this one is cleaning the raw data. So we've specified three kinds of cleansing that we want to do here. One is the valid logs. Second one is invalid logs. And third one is corrupted. Very simple, corrupted. It's not a JSON cannot parse it. Our file format is JSON. Invalid means there's a file, there's a data type that it's expecting, and the incoming data did not conform to that data time. For example, I'm expecting a string. They gave me a Boolean. It doesn't conform. So, so we have to do. So this is based off of the programming language used here is Scala, because Spark is natively built on Scala. Spark does support Java as well, Python as well, but for us, we chose to write in Scala nearest to the underlying technology that is processing the data for, you know, to don't to not lose the memory overhead for JVMs, basically. So in the, I will, I'm creating a log path array. And in that array, I have specified three, right? So valid invalid and corrupted. And then I'm going to do dot par. Par means parallelize. So I want to parallelize this execution in a dot map. Again, dot map, like what I showed you, it just segregates it into the category. So what this does is that given the inputs, I'm going to segregate it into valid invalid corrupted. That's what it does in parallel. And then, so validate is an internal function doing something else. Let's forget about that. And then finally, if we do some, we do some counts, obviously invalidate. So invalidate, if we do some counts, if it passes, then we just say that the counts of the valid, corrupt and invalid are correct. And it's successful. And then we go ahead and save this RDD. What save RDD does is just it writes to S3, nothing else. And then if not, it just says error basically. Oh, and by the way, again here, rdd.par, parallel mapping saved to RDD. So parallel saving to S3. That is what makes it very fast. So this is very simplistic and very simple example of a map reduce function in Spark. Then after you've compiled this, this runs, you know, you've verified it, you've debugged it, it works fine. What you need to do now is schedule a workflow. Again, we're building a production data pipeline. And in the end, I will tell you why do we need to build it. Okay, so we scheduled a workflow. This workflow is the action, this is the action. Okay, I tell it that you do the raw clean for me. This is the configuration. And that's that. And I schedule it every day. Right. So Wozzy's in taxes. Don't ask me it's XML. Yes, it's outdated. But this is what it is. Then once it ran, of course, you have to verify your data. So I've redacted some data over there because I don't want to violate anything. So can you see, is it very small, the text here? Oh, I'm so sorry about that. It looked fine in my big screen. But basically it's a select from that table that I was creating. And I just did the partition pruning on the date. That's it. Simple. And a limit by 20. So this is the result it gave me. That means my job ran successfully. This data is now there in the in S3. And then after everything, I create a visualization on top of it.  And we have a second version, a second part of the processing, which has abnormal and delayed and all these things. But basically, this is a dashboard that I've created. Now to answer the question, why do I need to build a big data pipeline? It's because this have to be updated every day. So when executives are asking questions, how many licenses are being used? The questions are not usually just a one time basis. They're usually asking on a daily level, how many licenses do we have today? How many subscribers do we have today? How many subscribers do we have tomorrow? Yesterday? How many? How what's the trend? How do we compare? So in order to establish that trend, and in order to populate dashboard that updates on a daily level, you need to build a production data pipeline that runs hourly, daily, monthly, depending on the need. Just a one time run, scheduling it on a cron, it doesn't work. And it's a little bit nuanced, but in order to create a production data pipeline, it requires a little bit adjustment into the mindset into the thinking. It's not your regular programming, regular programming that sounds like TV. It's not like your regular application development. It requires a little bit thinking of parameterizations for dates, parameterizations for certain fields. It's not rocket science, but it does require a different kind of thinking. So I reserve some room for the Q&A, but I just want to let you know that I've put some of the resources here as well. You can take a look at Spark if you're interested. You can take a look at Uzi if you're interested as well. And the first two are basically around what is big data. And the third one is an introduction to Spark. Why is Spark better than Hendoop? So that's the end of my presentation. I reserve some room for Q&A. Yeah, go ahead. So you mentioned at the beginning that you're moving everything from in-house to AWS. How worried are you about vendor login? Great question. That's the first question that I ask our architect. So honestly, there are advantages to vendor login. One being in an enterprise. We don't want an enterprise where what we're building in this VPC cannot talk to another VPC purely because they're different vendors. The second one is honestly, if you take a calculated guess, AWS is not going to go away anytime soon, at least not in my lifetime. So there's very little worry around that. And actually Autodesk, what we've done is we've taken one step further to partner with AWS. So a good point that I wanted to bring up. So I mentioned previously, right, we built this in-house. So when we were partnering with AWS and they were coming out with these new services, their services could not handle our load in our use cases. So that partnership was really helpful for them, for them to be able to ramp up their services. At the same time, it's helpful for us because then it becomes managed. We don't want to manage it. We don't want, you know, our 10 people, 10 engineer team cannot battle with their like 100, 100 engineer team, right? So it's a win-win in that case. So we've partnered with AWS as well. So kind of that sense of fear and that sense of apprehension, it's not there at all. Yeah. Anyone else? Yes? Okay. So the, okay, I know you said apart from managed, but I want to kind of take a step back there because when we say manage, it's not just someone is managing it up and down. It's also around SLAs, right? For the data that we are collecting previously, just to give you an example, when we were using Kafka, we were losing around 30% of our data. And it is because the amount of data coming in is a lot bigger and a lot faster than what the Kafka can handle. So when we switch to AWS, it's not just about reliability and someone maintaining it for us. It's also around their SLAs and their capability to auto scale up, auto scale down based on our needs. And the amount that we pay for that managed service is pennies in comparison to, you know, having one or two people dedicated to maintaining this Kafka stream to auto scale it, auto scale it down. And at the same time, if we've already lost data, you know, yes, we can retroactively fix it, but there's no going back to collect that data again. So in terms of why, what's the benefit, it's not so much technical and not so much managed as the promise that AWS has given to us that there's no data loss. And true enough, after we've switched to AWS, we have 99.9% of our data. There's 0.001 loss, but that is acceptable. Thank you for asking. Any other questions? What else? This is a good question. It depends on what the product is. So because Autodesk has like 120 plus products, most of them are around manufacturing and construction. And most, well, our biggest one is for design for AutoCAD is both design and manufacturing. So there are instances I know, for example, for AutoCAD, there is what we call telemetry data, every mouse movement is tracked. So that is why there's a huge amount of data being collected. Now that's for AutoCAD. Other products, let's say, for example, for structural analysis, every action that AutoCAD takes also introduces or also calls them a thousand times. So imagine that one slight mouse movement generates a thousand calls, we get that data. And for AutoCAD as well, any every mouse movement generates that thousand. So that's why the data is huge. Yeah. Yeah. It's mostly around how commands are used and mostly around, you know, different geographical locations basically they've put in their code. When you hit this, get the data. When you hit this, get the data. Thanks for asking. We have time. What else? I think we have time maybe for one more question if you have any. All right, I guess that's it. Thank you, everyone.


The Department of Education, Innovation and Gender Relations wishes to inform parents and guardians of students who sat the 2021 Common Entrance examinations that results for the exams will be released on Wednesday 21st July 2021 at 10.30 a.m. As is customary, the presentation of overall results will be first made to district education officers and principals and then results will be presented to students by their respective schools. As a result of the ongoing challenges with the COVID-19 pandemic, this year's presentation will be made via Zoom to all districts and will be streamed via the Department of Education's Facebook pages. Analysis of overall student performance and performance by institution and subject area will be disclosed at Wednesday's presentation as well as the way forward for the Common Entrance examinations. This year a total of 2,148 students sat the exams of which 1,058 are male and 1,185 are female. The Evaluation and Assessment Unit of the Department of Education, Innovation and Gender Relations thanks you for your patience and apologizes for any inconvenience which may result from the decision to execute an online presentation.