With Linux serving as the backbone of modern technology infrastructure, from the Internet to the cloud, mobile phones, supercomputers and more, it is essential to understand the security challenges that can affect almost every system. The Linux Security Fundamentals Training Course from the Linux Foundation will ensure you can assess your operating system level security needs and readiness and implement security options as required. The class explores many of the features for securing Linux, which are either built into the Linux kernel or added by the various Linux distributions. This comprehensive course provides experience that will further expand your awareness of security issues and preventative measures. Anyone involved with any security-related task, including implementation technicians, developers and managers, will gain valuable experience from this course. Enroll today and up-level your security posture.


It's DDK and I'm back today with another Dordaz video. Huge shout out to everybody tuning in. If you're not subscribed, make sure you smash the subscribe button. I need everybody to stop the video right now and give me a thumbs up. Just so you guys know, 80% of you are not subscribed. So it costs this month, $0. All I need you to do is hit that red button and say subscribe. Today is Monday, December 13th. And we do actually have a few Walmart orders right behind me. We have two on her phone for $17 and then we have one person on my phone for $15.50. I hope you guys enjoyed the video. Again, smash the subscribe button. It's time to get to this money. Let's go. We're at the first delivery. We called them but they didn't answer the phone. I'm gonna try to give it to them because it's very light. And if they're not here, I don't know what to do with it because it's gonna blow away, it's real light. So I'm gonna hand it to them in their hand. If I can, I did see a car in the back. So maybe they're here. But sometimes people even if they're hungry they never answer so let's see what happens. Hello, this is your Walmart order. Oh, thank you so much. You're welcome, you have a great, I can carry it for you. Oh, thank you so much. You have a good one. Thank you, have a nice day. You're welcome. You have a good one too.  I don't know what it is. Hello, how are you doing? I have Walmart orders for Louis Micah. Okay. Alrighty, thank you very much. Thank you. You have a great one. You too. Oh, and it's a Walmart order. It might be bad, it might be good. We don't know. 6.4 miles, 11.75, let's go. Here baby, this is actually pretty sweet. Bring it up to the bell, I got a card in there by the answer. Put your hands up here. I didn't want that photo, but y'all already know though. Y'all know I didn't want that photo man. Click that. I don't know if that bell worked. We gone. Hot belly, 4.5 miles for 6.16, let's go. Hello, I'm gonna have a doordash for Rhiannon D. All right, thank you, have a good one. We got us one Japote late, two miles for 6.75, let's get it. In this thing baby, we are here baby. Throw out that first delivery. Hello, I have a doordash for Chrissy D. Chrissy, here you go. Here you go. All right, thank you. You're welcome, you have a good one. Here we are, here we got a school again. Schools love to get this doordash. I might get a doordash for Rhiannon. I might have a doordash delivery for Rhiannon. And what's the last thing? I honestly cannot pronounce it. This time I'm supposed to do it. Okay. I'm not even gonna try it. I don't want to chop it up. All right, thank you, you have a great one. Got us one Mesca restaurant, 4.6 miles, 7.75, let's get this money. In this zone baby. Here we have a doordash for Andreas. We are here. Hopefully your dogs are gonna get us. Here and start barking here, so. All right, here we go, doggy, doggy. Here we go, doggy. Y'all like those dogs that they be feeling. They will bite you too though. Y'all they want the photo? Y'all already know though. Do you want me to help? We're gone. Got us a jewel, 2.4 miles for 9.75, let's get it. We added on this other order for $5. There's three items it says. I don't know how many miles it is as you guys see it doesn't tell me, let's go. Two, one, two, one, two, one, two, one, two, one, two, one. I think we got a couple of them. What you hear up? Let's go. They're making a park over here, so. Hello. Here we go. Oh, I'm gonna have two doordash orders. One for Stella, one for Sally. One for Steve. What's that? What is it? I'm gonna put all hers up top. They got the feeling. You got these models off, it's not that much, bro. I'm gonna send them on. Appreciate you. Thank you. Aw, he's mad. You want to know one thing? I know you mad. I know you mad by job. I understand if you mad, but you gotta understand. When people are old, they just get to do what they want to do. Okay, we are here. Luckily, this one isn't a permit, but it's only four bags. You guys can see what we want. We have to wait for the bothers then. It says she has a work, so it's gonna take her time to get to there. It's the same as a permit anyway. So you gotta take your time, take your time, baby. Hello, this is doordash. I'm here with your jewel order. Okay, I'll let you in. Thank you. Okay, thank you. Hello. Hi, how are you? I'm fine to yourself. I'm good. Is it nice out? Yes, ma'am. All right. Finally, we got one. Would you like me to put it on here? Just on here. Okay. And this one is your bread. I'm just gonna sit it on top. That sounds good. All right, thank you. You have a great afternoon. Thank you. You too. She got us right. That one's for $10.22. Now we're going to the next one. This is actually the same apartment building. This one is upstairs. Hopefully we get the elevator here. Hello. Hello, ma'am. I'm here with your jewel order. Okay. We got one royal tie, 7.5 miles for $9.25. I know. I know. I know. It's not $2 a mile or close to $2 a mile, but we have to go this way anyway, so we're going to take it. Let's get it. We are here. Let's get this food. Hello. Hi. I'm gonna have a doordash pickup for Steph and Kay. Okay, we are here. Drop this right there, ma'am. I didn't want the photo, but y'all already know the photo. Here we go. So it is 5.45 and we're about to start the dinner shift. I know we're a little bit late. We got a lot going on. We got the drip coming. We had to take some pictures of a house. We had to do an inspection on a house, so we had a lot going on. But I hope you guys are enjoying these videos. I love putting out the content. And if you aren't subscribed, make sure you smash the subscribe button. Make sure everybody give me a thumbs up. I need you guys to start comment below hashtag DDF. It stands for doordashfamily in the comment section. And before we left, we were at 90, 91, 91.38, which is not bad, because we did have to stop in the middle to take some pictures of a new house. It's about to come on the market. So I hope you guys do it again. Smash that subscribe button and let's get back to this money. A 950 for 5.7 miles to the Mexican restaurant. Let's go. The order is $6 to 1.8 miles. Let's go. Dang, baby. Let's get this money. Oh, it's ugly. What's been going on? Uh, Mohammed K. The first house. It's going through. Let's see if she can hear me. The other one is actually over in the same complex. So it's pretty good. Hello. That's one. It's one more bag. Thank you. You have a great day. All right. That order was for 1475. Let's go to the next one. Number two off. It's weird saying extra number two, two or three. I don't think it's going to work though. I don't think I know. The other ones make noise. Let me give them a call. Hello, Mohammed. This is Jerome with DoorDash. I'm outside with your order. Building? Yes. 1921, correct? Yes. Uh, let me talk to you. Okay. Are you in the white car that's in the middle of the parking lot? Yes. I'm in the white car. No, I'm actually by the door. Oh. Oh. There you go. You have a great one. 11. We's only supposed to get six, but we got 11. Let's get it. 3.2 miles for 5.57. Let's do it. The building, baby. My peace. We're here, baby. This is right over there. This one also says handle to customer. So we're going to hand it to them. I like this little four here. It's pretty nice. Hello. How are you doing, sir? I like it. I'm fine to yourself. Good. Thank you. You have a great evening. All right. Thank you. 5.5 miles for 8.75. Doesn't make it happen. I'll do an extra cleaning. Excuse me. I had a door there for Ed. Both of them? Okay. Thank you. You have a good night. This is beautiful. Look at the vast level. Hey, y'all see what's going on? They live lavish out here. Look at the door. The door's huge. Oh, no. It's very dark. I don't know if the photo... No, no. Photo. We got it. Wendy's for 8.50. And then we have Chick-fil-A for 9 dollars. They're both going the same way. So we're going to try to take both of them. Hopefully the drive-thru... I mean, hopefully the diner is open for Wendy's. And if it's not, if the drive-thru is fine, then we might go through it. I don't know. But if it's too critical, we're going to cancel that one. Then take the Chick-fil-A one. Let's get it. Lamb skeet. We got a game coach. Hopefully they don't take forever. We have a tennis player doing that here for some reason. Oh, we might be doing that here. Yeah. Kevin C. Yep. That's how I feel. I'm already getting busy. All right, thank you. You have a good one. I don't think I saw a barbecue show. All right, thanks. You have a good one. We are here. Can I see the front door? Put it on to the side. Fold it enough down. Fold and bang. I don't want that photo off the chart right now though. Fold it down. We got it. 9.25 instead of 8.50 for that first one. Let's get back to this money. Yeah, since they had leaving it at the front door. Leaving it at the door, baby. What's with this? Almost there, don't laugh, baby. Daddy, what's the photo? That sidewalk was uneven. Got that.  Come on, man. Got that. No bill. We got it. It's 2.7 miles, 6.75. Look at this money, baby. The chili orders on 0.6 miles for 5.65. Let's get it. Here, lady. This is John. He's getting hard to the lady for the last time. He's got some other man. Yeah. Thank you. Drop off number one. There's that door. They want that photo, but y'all don't remember the photo. The old photo. Let's see. 2.15. Hello, I'm here with your chili's order. I can bring it up, man. Okay, thank you. You're welcome. I try to be. I can't leave you hungry. Thank you, you have a great night. Thank you so much. You're welcome. 2.5 miles, 6.75. Let's go get this money. Let's get this money and let's make a stack. How you doing? Have a do that for Brooke H. Brooke H. All right, I appreciate you having a good one. I appreciate it. All right, you have a good one. Thank you. Smoke coming out that morning. So it's the end of the night. I want you guys to come below and let me know how much you think we made. Five, four. And don't be cheating. Three, two, one. Today we made 173.85. Honestly, I think it's pretty good. I wanted to get like the 200, 225. But we didn't reach the goal today, which is not that big of a deal because we did kind of miss a few peak moments in the day because we had to do a few other things. But I hope you guys are enjoying the new content. I hope you guys are enjoying the videos. Again, if you aren't subscribed, make sure you smash the subscribe button. If you aren't subscribed, everybody give me a thumbs up. Oh yeah, by the way, the 80% of you that aren't subscribed to the channel makes you smash the subscribe button. I don't even know why you're playing. I know you be stabbing and dabbing every day you come over here. Look at a little video next day come. This one's just press the subscribe button. Hit that notification bell. And go ahead and get tuned in and join the family. But I see you guys tomorrow. We going, we growing, we flowing. You already know it. It's DDK and I'm on my way.


These Baltimore Ravens choices for their decision that they're gonna have to make on Lamar Jackson this offseason They became a whole lot clearer YouTube team keep it clean What's going on is engraving here with another video and this is probably going to be one of the hardest videos that I will ever Have to make because the detail and the numbers and breakdowns and all that I'm I really want to try to make this as clear as I possibly can for everybody So we can all have a full understanding of the possibilities that lie ahead When it comes to Lamar in the Baltimore Ravens, so y'all please bear with me And if there's anything that I said or miss spoke on feel free to let me know because I make mistakes plenty of them Probably more than anybody But we'll try to make it as clear as possible before we get into it though I gotta give a special shout out to all of y'all. Thank y'all so much because we are slowly slowly Creeping up on 63,000 subscribers and that's insane to me It's this is a lot of crazy people in this world It's a lot of crazy people in Florida alone But the fact that almost 63,000 people decided to subscribe to this this channel now There were a lot of long way that unsubscribed to but 63,000 people decided to subscribe and stay subscribed to the channel after everything. What really? Hey, but I appreciate y'all. We all crazy in this thing together and also special shout out to the newest team Keep it clean patron my god John appreciate you and just thank you to everybody really man. So anyway Lamar Jackson we know Lamar Jackson is a pending free agent now We know Lamar Jackson ain't gonna sniff free agency But the Baltimore Ravens they have some decisions to make when it comes to Lamar Jackson One of those decisions that they could make is offering him a long-term deal and coming to agreement with him on a long-term deal And that would be the hope that they can get it done But in my opinion, I don't think it's gonna be something that ends up happening before a lot of other dominoes fall into place Things that we've talked about already like the offensive coordinator thing That's gonna be important. What the Ravens do in free agency is also gonna be important how they move and then a very pivotal point in this offseason, I think is the draft because If they're gonna trade Lamar Jackson I think they would do it before the draft because they're gonna want to get something right here right now They're not gonna wait till next year to get draft picks for Lamar if they trade them They're gonna want it this year. So if that's gonna happen, it's gonna happen before Draft time, but if Lamar Jackson is still here After the draft and that'll increase that'll increase the likelihood that he remains with the Ravens It won't be a short thing, but it'll increase the likelihood that he does stay But I think before Lamar Jackson signs anything. I think he's gonna want to see how the Ravens move but With how the Ravens move that can be dictated by a lot of what happens with the NFL but the NFL today According to Tom Palacero the NFL informed teams today that the 2023 salary cap will be a record 224.8 million dollars per club So the salary cap is a little over 224 mil. So now that's that's clear So now every team knows what they have to work with they know how they can move and operate and maneuver and finagle and do this They know now the Ravens the cap space that they have according to Brian McFarland Raven salary cap Y'all know him. He is on point with all that salary cap stuff I don't know how he does it, but he does it, but he said that the Ravens with that number This will leave the Ravens with 27.8 mil in cap space So they have a little under 28 mil in cap space moving forward. That's what they're pending free agents That's what the guys that they signed to the future reserve deals. That's with everything everybody who's on the books That's including Roquan Smith his escalators I think Devin Duvenay is escalated matter BK and maybe some other guys too But that is what the Ravens have right here. Well, not right here right now, but the start of the league year That's what they started with unless they make some other moves, but that's the number that they're starting with now This is where May get a little confusing, but I'm gonna try to make it as clear as I possibly can So again, my apologies in advance if it is confusing, but I'm trying my best so Brian McFarland he tweeted out that for There's a couple of different franchise tags that the Ravens could apply to Lamar Jackson There's three different franchise tags. One is the exclusive tag One is the non-exclusive tag and one is the transition tag So we are going to talk about all three of them So we'll talk about the non-exclusive tag first the number the salary cap number for the non-exclusive tag Would be thirty two point four one six meal. So a little less than thirty two point five meal That will be the cap hit that the Ravens would have to take that will be the number that Lamar would take If it was a non-exclusive tag now Let's look at exactly what the non-exclusive tag is in this I'm referencing a CBS article that they put out Breaking everything down so it says the non-exclusive tag allows a player to negotiate with other NFL teams But if he signs an offer sheet with another club His team has five days to match the offer if the offer is not matched His team will receive two first-round picks as compensation from the signing team So that was pretty self-explanatory say for instance just to use it as an example say for instance the more that if the Ravens give him the non-exclusive tag and The dolphins just it's just an example, but if the dolphins sign Lamar to an offer sheet and they sign them to Just for example 250 meal deal fully guaranteed The Ravens would have five days to match that offer The 250 meal fully guaranteed and if they don't then Lamar Jackson will become a dolphin But the Ravens they will receive compensation in the form of two first-round picks from the dolphins So that was that's how the non-exclusive tag works now For the exclusive tag The exclusive tag is the average of the top five QB cap numbers as of right before the draft and that should be around 45 meal so With the exclusive tag the exclusive franchise tag it's about 45 meal But a player cannot negotiate with other teams with the exclusive franchise tag So there's no negotiations with the exclusive franchise tag. You either take it or you leave it That's it. The team has the control in that one. It can't know other teams in a fear. They can still trade you now With every tag you could still be traded or you could be moved or whatever But with the exclusive tag you can't talk to nobody well legally at least because you know tamper and be going on but anyway So that's that one And again, that one is around 45 meal now There's another tag there's a transition tag and you know what my apologies cuz I did not see how much that one would be but That one is pretty irrelevant because you know Ravens ain't gonna use it and let's listen to why the transition tag Is based on the average of the top 10 salaries at a player's position using the same Methodology as non-exclusive franchise tag calculations, but the teams have the same writer refusal as with the franchise tags But they do not receive any draft choice compensation for declining to match an office sheet. So Basically with the transition tag Another team could offer will use Miami again for an example. They could offer Lamar Jackson a deal But if the Ravens don't match it Okay, they don't match it Lamar will move on to Miami But the Ravens will not receive anything in return. No draft picks. No compensation.  So we ain't even really got to worry about breaking that one down But still you know how it works now. So again, we got the non-exclusive franchise tag That's where Lamar and they could talk with other teams other teams to sign them to an office sheet and whatnot And if he signs and the Ravens matched and the Ravens keep them But if he signs the offer sheet and the Ravens don't match whatever he got offered Then he will go to that other team and the Ravens will get to first-round picks. That's the non-exclusive tag the exclusive tag He can't negotiate with anybody and then the transition tag He can negotiate and if he goes to another team if Ravens don't match the offer then they wouldn't receive anything But now what the Ravens are gonna do. It's not clear to us I mean, I'm sure they got their different plans in place, but Everything is a lot more clear now because they know what the the salary cap is gonna be and They know how much salary cap space they have even though the cap is kept But and then they know the amount of all the different franchise tags and what they entail So I really hope that we explain this clearly to everybody my apologies if this confused you even more But I just really wanted the information to be out there so people could understand All the possibilities and what everything means so when you hear like cuz I know when when you hear like franchise tag Or you hear non-exclusive tag or transition tag or or Exclusive tag it can be confusing. I know it's confusing for me It's definitely confusing for me So looking at all this this helped me understand it a lot more so hopefully the same thing happened to y'all But anyway team keep it clean. I love y'all. I appreciate y'all and just like I hope Lamar Jackson isn't with everything


my god you guys we are inside of an abandoned hospital and the links are creepy in here my comments is the same my solution to player stacking is bonding outside spawn to spawn out what all right what if I need some milk what is that noise oh god what is up you guys your list of game here today we're gonna be playing the haunted hospitals and we're here without the shark timeless bad place roblox slash red cat and tweeter and that weird noise in the background I don't know what it is open door alright let's see let's check out this computer somebody was looking at cats what okay it says cameras let's check out the cameras real quick pressy oh god this looks spooky that's a hallway what's that noise oh god this is creepy man why whatever let's let's begin with this adventure click to open okay no oh who want to open the door somebody open it because I want to have a flashlight out I hear something coming from this room wait hello anybody here we go oh wait we can't go in here what's that noise okay that's the weird noise god this is scary man hello there not be any ghosts in here because I will fight you man we will fight the ghost we will all beat you up together the whole squad will okay let's check in here oh I hear the noise anybody here okay guess not man because there's nobody I guess we're safe for now how about in here I mean let's check every room you know in case there's something inside oh god okay nothing seems to be happening yet so serenity comes here not okay that's a real out that door is locked oh my god why is there blood coming out from this room what there's blood on the bed oh what what did that guy come from what in the world hello mister I think he's asleep or something you just thought from the ceiling oh my god that don't push me all the way okay guys be careful to stick together never separate because that is the worst mistake we can do because then they will take us out one by one oh god what is that just a table well that is a scary table okay seem like there's nothing here okay let's let's keep going okay that was scary is there somebody here you got all like we could go in the vent let's go let's go off the shark we're right behind you buddy come on guys you've got to get out of here oh god that was super loud and scary what is this for okay seems like we open that door oh I hear music that's coming from here okay this is this is spooky why is there music playing and who even played this music like there's nobody around okay let's go this way then oh no so you're late at timeless right behind you man okay let's open this door oh my god this is a giant hospital what welcome oh god this is super creepy why is there wheelchairs in the circle like as if there were somebody in these chairs move I don't want to be in that room okay should we go down here okay let's go check out the single door over here I don't see anybody yet what about here no oh god okay let's over here restroom let's go check this out man oh god what happened here why is everything so rusty what there's blood oh my god is it leading us somewhere look at guys there's blood on the floor it's trying to tell us something where is it going okay seems to be going this way okay you stopped maybe here why is there a wheelchair oh okay you guys think we're getting somewhere oh there's a key here key one we got the key okay so this key must be used for something why is there a camera here that is weird and it's recording okay that doesn't look like it's looking into this room oh okay let's go back this way someone come back for me oh where's Ben oh Ben got separated you see that's what I was talking about oh there's somebody right here Lily bear oh wait let's go check out that other room let's go over here okay those blood trails are still there anybody here what dude someone just appeared there what does that say okay we've got it I can't Jonathan I can't have you watching television while you're on duty if a patient were to come running through it in an deranged rant while you're watching TV there's no telling if you would close the door fast enough before he makes for the front entrance you may listen to the news if you'd like you must keep one eye on the door at all times okay so he's talking about that door right so something better not be coming up in that door because I'm gonna fight that dude found another one what this is empty okay let's check out this door what's here another letter oh god why are these pages empty oh no okay think oh my god what who is this guy why is it coming after me okay he's going after timeless now no don't get out of here about about you guys I'm not gonna let that guy get me oh god what's that loud humming hello anybody oh but is that dude that's loud with some a reset someone has to go for bed okay we have a key so it's okay this is what the key is used for why don't you put steps where's everybody where those footsteps coming from you guys should we go on without the other guys okay let's just see where this takes us okay we have to go down oh no alpha alpha just jumps right down there why would you jump down there boy you didn't know what's what we're stuck so we're stuck here oh that oh it actually works okay so we're going up oh the music where's this thing taking us I have no idea okay the music went away oh god that was loud okay so where are we at now we're still in the hospital but oh my god this is scary there's a doll here with toys okay let's see what this door takes us okay there's a letter here you guys hear that music that is not cool man wait it's the same letter why is it the same letter oh no what does it say that says something else right it says what does that say I really must insist that we that we get a red copy with different color my god there's footsteps a different color this morning Sarah said that it reminded her blood Janet okay whoever Janet is I don't know why she would be up here it is so scary what to that help what oh my god my real name is Jonathan nice man how did this girl not even get up what in the world is going on here guys oh let's get out of here man this is already getting spooky okay that's a dead on that's not a dead end let's go check out this hallway dude that dawns just got up and fixed herself okay this isn't that oh my there's a ghost no wait where's it going there's nothing here it's just a dead end oh my god time this I don't know where that guy came from wait what happened to the other guy where's the alpha shark oh no think you left with that let's go let's continue this oh my god the box is just did you guys see the boxes one cuz there's definitely some go say hey Lily kind of toes oh god oh dude this goes all over this place okay why is that painting everywhere though oh my god I am super scared guys y'all forgot us and real what I heard something break oh my god there's a chair moving where's that chair movie okay there has to be another key here or something there's a computer excuse me is there nothing in that room I guess you check that one already cuz nothing here nothing here nothing to see here ladies and gentlemen there's a chair that's knocked over in here dude that little creepy music is freaking me out why don't hear a ball it sounds like a ball my god I think there was something in there can't see a staircase right there nothing here okay I don't think there's anything in these rooms oh my god okay I guess this is the only choice we have to go down I wonder where this is gonna take us though so scary I don't want to be here anymore guys I just want to get out of all timeless and Lily should we go in here okay that is locked what about the key the key does not work there okay guess not it's not meant to be keep going down wait we keep going down and down that is not good you never want to keep going down okay we got somewhere what we're back in the beginning are we oh my god there's someone right here alright that's a player I'm about to say what all that's a door you can't even open okay so we are definitely getting somewhere guys oh god nope I'm sorry I'm out of your ghost I did not mean to bother you Lily yet to come this way oh my god we made it out of the hospital I didn't know that was possible I guess we made it that's it oh wait let's see I see a door over here but I think that's it yes okay that was like help us guys we made it out but I think we're we're stuck out here because you guys hear that music that's like some Victor music isn't it I heard something because it seems to be coming from the hospital I think this is it I think we made it we made it we got out of that stick place but we're still stuck in here maybe the teleports is no nothing happens guys I guess I made it out of there well not technically because we're still inside the hospital but nothing seems to be happening all right you guys let's say bye to her there you go thank you guys for watching today I still made support and I would see you guys next time oh no she's scared bye


Whenever you're consuming a product, be it food or beer or cannabis, you want to make sure that it's not going to harm you. The foundation of the cannabis regulations is the strictness of the rules of one of those which being the testing requirements. We start with obviously the potency of the material, so THC, CBD are the ones of interest. Then we move into some of the sort of things that aren't supposed to be there. So we look at heavy metals. We are looking at what we call aflatoxins. Any solvents we use in our processing, you have to make sure they're gone. Y'all in NatHealth Canada has published a list of 96 pesticides that everybody must conform to. Knowing that you're consuming something that's been looked at for all these contaminants versus going in blindly and consuming something that may be more accessible or less costly essentially, it's a risk that many people probably wouldn't take if they knew what the implication was.


to Healthy Planet, the show for people who care about their health and the health of our planet on the ThinkTech Live Streaming Network series. I'm your host, Dr. Grace O'Neill. Joining me today in the studio is, or on Zoom, is Darlene Lume, Butterfly Rancher and owner of Sharing the Butterfly Experience. Today we're going to talk about monarch butterflies. So let's get into it. How did you become interested in monarch butterflies? So I had this deck which is above my garage and it was just empty. No one used that space. So I had this epiphany one day to think how I could use that space. And I wondered if I got crown flower would butterflies come? And so one day before Christmas, almost 10 years ago, I went to get some plants out from my friend in Waimanalo. And he wasn't my friend then. He just had advertised. But I bought plants and then I got caterpillars and they ate all of my plants and then they started to disappear and I was wondering where did they all go? And I found out that there were predators eating the caterpillars. So then I invested in habitat and it just snowballed. It's just from a thought of what if. So when you're saying you got habitat, what do you mean by habitat? It's just an enclosure to keep the caterpillar safe. So in the very beginning, I bought a chameleon cage and not realizing that some of the predators can get into the chameleon cage. And so eventually through trial and error, I was able to get screened tents. And that's what I use today to raise all of my caterpillars. And you just have them, they're potted plants on the rooftop or how are you doing it on a rooftop? So the caterpillars, so the butterflies fly in and out of the garden all day. They eat, they mate, they lay eggs. I gather the caterpillars and then I keep them safe in the mesh tents in what was built for me, which is a nursery. It's five feet by six feet by three feet. And it has lights, it has a roof. And all of the tents are in there safe with 19 gauge wire all the way around. Yeah, it's pretty secure. And that's through trial and error, you know, when I've had rodents get in. Wow, rodents. Yes, and they ate the chrysalids and the caterpillars. What kind of other predators will eat the butterflies besides, I can't even believe rodents eat them, that shocks me. Yes, that's what I learned too, because before it used to be just window screen all the way around, but they ate right through that. Some of the other predators is the Bulbul bird. Yeah, so it's a black cardinal with the red under its tail. That bird eats butterflies and caterpillars and is voracious. It is a very smart bird. So it will catch the butterfly midair. Oh my gosh, that's because sometimes I see the butterflies, they're just hanging when the wings first dry and it's trying to dry its wings and it can't use them yet. And I wonder if it can be attacked in the garden and, you know, they won't go near it because I'm there, but I'm wondering if when I go away, it's sweeping in and taking them. So if you look, the caterpillars typically try to find some place safe to hang because they're hanging in the jay for very many hours. Yeah.  So that's why sometimes if you're not watching them, they'll crawl under a table, under a ledge, under leaves someplace where they can hide. Yeah. So it's good to put them somewhere that's more safe. So I had heard that after, you know, someone told me this was a rumor that after they get to a certain size that they're not appetizing for Bulbuls. But I guess that's not true. They're always appetizing. So if you if you see just wings on the ground, then it probably was eaten by Bulbul because the Bulbul will eat just the abdomen and leave the wings. That's terrible. Yes. It's heartbreaking for people that are racing, they go through all of that work. They release the butterfly and then hair swoops down a Bulbul and steals that butterfly as soon as it's released. Is the Bulbul, is that, I mean, where is that from? Is that a native bird? No, invasive, invasive. So in the late 70s to 80s, it was introduced to the islands. And you can Google that. There's a story about it being brought in by the Audubon Society and how they escaped through a windstorm and then they just thrived. So this bird is very smart. It will jump from branch to branch and look under the leaves. Oh, wow. Yes. Very smart. This is scary. And I heard there's these, actually, there's these wasps that will attack the butterfly and eat the butterfly too.  I mean, what other predators are there? So the wasps typically are females and they will be attacking the caterpillars, not the butterflies. Oh, okay. Okay. So they eat the caterpillar and they regurgitate it to their young. That's what they're feeding. They're young. So there are tiny wasps that use the chrysalis for multiplying. There's flies that will use the host. They will lay an egg in the caterpillar and eventually when it becomes a chrysalis, the maggot will crawl off and the chrysalis is dead. Oh, geez. So in nature, less than 10% make it through the whole cycle. So this, but just this February, this last month, we had a huge boom of caterpillars. And I'm attributing it to the number of people that are racing. Interested in them. Yes. Yes.  So I saw that these caterpillars, they also, some of them, they come out with the crumpled wings. And that's a special parasite, right? Right. It's called OE. It's got a really long name. And that is, it cripples the butterfly. If you have any of those, you should put them in the freezer to euthanize them and make sure that you disinfect everything, including your hands so that it's not transparent. Oh, geez. And then I saw one, I had one that actually tried to come out of the chrysalis and it couldn't come out. It was stuck for days and then it died. I don't know why that happens. Is there a reason why that happens? Sometimes it's pesticides, but it is, if it's not strong enough, it's not going to be able to come out. Yeah. Yeah. Yeah, my gosh. So let's talk about the other kinds of butterflies in Hawaii now. That's the monarch. And is that a female or male? Yeah. So what I tell people is think of a peacock and fishes and birds. The males are always more vibrant. They're prettier and the females are very bland. And so when you see a dark bean butterfly, a monarch butterfly, that's probably a female. There are also two hormonal dots on the bottom of the male's wings, but it's easier to spot from a distance if you just look for orange because the monarch males, the male monarchs are much more orange than the female. And I tell the kids it's because they have to attract the females. They have to work it. It's the opposite of humans. Right. Right. So let's go on to the other butterflies now, Max. This is the gulf fertility and tell us about that one. So every butterfly has its own host or food plant. And the food plant for that butterfly is lilycoy leaves. So they don't eat the fruit. They just eat the leaves. Yeah. And isn't it, I see this color here. It's more pinkish, but I was actually trying to grow the lilycoy in my yard. And then, you know, it looks as a caterpillar. It's spiky. And so it attacked my lilycoy and ate every single leaf. I don't even know if the plant is going to survive, but it hadn't even flowered yet. I was so angry. But you know, these, the butterfly, the one that I have, it looks orange and black. Is that a different species of the same thing? Or is it a different kind of butterfly? It's kind of like a monarch mimic. The ones that I'm seeing in my yard now that like came out from, you know, my passion fruit. Okay, there's only one butterfly that we have in the state that uses lilycoy. And so that has to be the same. So on the top side, it's black and orange. On the underside, it is camouflage. And is this an invasive species or is this endemic? It's not endemic. So we only have two butterflies that are endemic. One is the commamia, and the other one is the koa. And the koa is tiny. So I've actually not raised any. The commamia is on the endangered list. So the endangered list, people are talking about putting the monarchs on the endangered list. It is my opinion that is a bad idea. Why is that? Because if you put them on the endangered list, you and I will not be able to raise monarchs. Really?  You can't have them. Okay. That's what endangered means.  So my thought is, what if we had, we were a whole bunch of people were able to raise commamia butterflies? How many more butterflies would we have if they were not on the endangered list? So what is their primary source?  Mamaki. Mamaki is their host plant. Yeah. So that's, I guess you'd have to buy that.  But you're not supposed to be able to touch them or hold them. No, because they're right. That's right. Yeah. Yeah. Yeah. So they're interested in the, you're correct because there's so much interest in the monarch butterfly because I was talking to a few colleagues at work and everybody is interested. Everybody's interested in it. Right. Because there's a lot of discussion about monarchs on the mainland and the migration. So in Hawaii, our monarchs do not migrate. There's no reason for them to migrate. We have them all year round. The only reason that they migrate in the mainland is because of the weather. So monarchs actually aren't endemic to Hawaii then? No, they're not. How did they get here? Probably on food plant. So Queen Lillio Kalani loved butterflies and she was gifted a crown flower plant way back when. It's my understanding. That's a crown flower in the picture. Right. And so in the mainland, they call it giant milkweed or colochipus gigantia. And we just call it crown flower because of the flower. It looks like a crown. So that's milkweed. I'm just showing everybody. I'm just talking about the milkweed. Smaller. It's a smaller one than the crown flower. But the picture before, if you can go back to that Max, you said there's a white monarch butterfly and the one that we normally think of. Yes. Right. So the white monarch is a recessive. It has recessive genes. So they're not very common and they're not very strong because it's recessive. I've tried to breed them and I've had success for two generations and then they just peter out. Wow. And I guess if you could show the video now. If you could tell us how can we, after these monarchs, after they come out of their chrysalis, if we want to have them in our yard, what kind of flowers can we grow and how do they, should we feed them when they come out? Does that make them stronger and how can we feed them? If you could show the video, Max. Yeah. So the video will show how the butterflies eat. They have a proboscis that they put into the flowers. And the flower that that butterfly is eating is pentus. It's P-E-N-T-A-S. P-E-N-T-A-S. And that's one of its favorite flowers. I also have lantana. So that is tropical milkweed and silky gold milkweed. And honestly, if the butterflies had a choice, if you put crown flower, silky gold, and tropical milkweed side by side, they will be laying eggs all over the tropical milkweed and the silky gold. That would be their first preference. Yeah. It's just like they completely obliterate that. It's a smaller plant and it's basically all gone. But you know, the crown flower grows bigger. So I guess it has. Right. You'll just see a caterpillar and it's like, oh gosh. So, Darlene, tell us about your business a little bit and what services you provide for people? So I provide butterflies for all kinds of events, weddings, funerals, parties. People can release a butterfly from a basket, a mass release basket, or individual butterflies.  How do you get the butterflies to like in the envelope in the first place? Oh, just put them in there. You put them in there and then when they're cool, they sleep. And so that's how we transport them and then when you're ready to release them, then you take them out of the cooler and they wake up within a few minutes. But butterflies are the most friendly. They will stay on your finger. They will fly to flowers, lay, creating lots of very special moments. How do the butterflies mate? Back to back. They're back to back. So and they stay mated for hours. Oh, wow. Yeah. And the males, the males can fly carrying the female. Oh, wow. Yeah. Is the male bigger? Not necessarily.   So is there a certain time period that, you know, that they would, is it like they're, they're chrysalis, like their caterpillar is this many days and they're chrysalis? I know it's like seven, 14 days or whatever.  Can you go through the life cycle? So from egg to chrysalis, it's about 10 days. From chrysalis to butterfly, it's about 10 days. And in Hawaii, from my experience, the butterflies live, the monarch butterflies anyway, live about three weeks. Oh, wow. Is that just because of the bullbill? My gosh. No, in that three week period of time, they are laying 300, 400 eggs. Oh, wow. Yeah, that's, I didn't know it was only three weeks. I thought they were surviving three months or something. I didn't know. If they're migratory butterflies, they will live longer, but our butterflies are not migratory. Yeah. What's I gonna say? I was wondering, are there any, is there anything besides like the crown flower and the milkweed and we're talking about some other plants? Like what other kinds of flowers is it mostly? I see them going after a lot of things that are growing naturally from the ground. Like I guess people would say they're reeds and everything. And you know, just the things that I would not necessarily even grow in my garden to be pretty, you know, like daisies or sunflowers, which I'm sure everything is probably, will they pretty much pollinate anything? Butterflies? They have a proboscis, a tongue. So they have to be able to put their tongue into a flower to eat. So those are the flowers that they're looking for to get nectar. Yeah. And if you were going to try to feed the butterfly, like would you just use a little honey and water and then put it in a cup and then try to feed the butterfly like that? You can also use sugar and water and put it either on a cotton ball or a q-tip or a cloth. You just have to watch for the ants because the ants want to join the party. That's true. I mean, I see some of them too. It's funny. Like some of them, you know, have them in the net and then some of them will kind of die. It looks like they just, I don't know why they die.  I don't know if it's because they didn't get fed soon enough. They were trapped in the net or something like that. So on the first day that the butterfly is born, it won't be hungry. But by the second day, you have to feed them. If they don't eat, by the third day, they're going to be on the bottom of the tent. So I tell people, just release them on the second day. They're stronger on the second day. If you have what they want, they will be back.  I tell people, if you're looking for my house, just look up and follow the butterfly. Absolutely. I'm very interested in this Kameha Meha butterfly too.  And then maybe you would be able to see them in your yard if you got that plant. So the Kameha Meha butterfly is on exhibit at the zoo. Oh, it is. And I have a butterfly exhibit that's opening July 1st. It runs through August 14th. It's at Pro Ridge. We did this every year. This year, it's going to be on the Makai side. The Kameha Meha butterfly will be there at least two Saturdays, maybe more. Even though it's in danger, it's okay. They have a permit just to, yeah. Wow. So the caterpillars will be there. The Kameha butterfly will be there. The food plant. People will be able to ask all kinds of questions. Usually I have Dr. Wil Haines come and join us, but he's not going to be able to make it this year. So the zoo is actually sending staff. Oh, wow. That's awesome. And who's Dr. Wil Haines, by the way? He started the Pule Lehua project on tracking the Kameha Meha. And he's part of the Department of Land and Natural Resources. And he raised hundreds of Kameha Meha butterflies and released them, trying to increase their population. What's the exhibit called if people want to go to the exhibits? How can they get on the internet? Monarchs and more. Okay. Monarchs and more. But if you just go to sharing the butterfly experience at, no, sharing the butterfly experience dot com. That is my website. And I'm on Instagram and Facebook as well. I really enjoy sharing the butterfly experience. And that's why I named it that. It just the joy that it brings people and the positive energy is just priceless. Amazing. Yeah. Agreed. I mean, thank you so much for answering all the questions. So yes, that is great. It was a pleasure to talk with you. Yeah. And to talk about my passion.  What is that called again? International Butterfly Breeders Association. And how many people are there in the state that do that? Is it just you or? So there's one other member and that is Paradise Monarchs. And Paradise Monarchs sells the chrysalis boxes. I've seen that at the farmer's market. Yeah. Yes. Yes. So we work very closely together. We have a good partnership. That's great. I heard that though, you know, I always wonder how, I mean, I know you said it's easy to get them in the envelope, but I actually had a couple of chrysalis that fell and then, you know, on the ground because I have them in these nets and they make a chrysalis on the net and then it like kind of falls. And so what I've been doing is I try to use some glue and tape it up there. And I see how he does it and it seems so easy for him. But I find it actually really hard because I glued myself multiple times. Okay. So just so you know, the chrysalis does not need to hang for Butterfly to emerge. But the reason that it fell maybe because it's not healthy. And so even if you hang it, when they make the jay, when they're forming the chrysalis, they have to hang. But once the chrysalis is dry, it does not have to hang. Oh, I didn't know that. I thought they had to hang because it made it easier for them to become a butterfly. As long as they have something to crawl up onto when they emerge, they can dry their wings. They can dry their wings and then just hang on to maybe the side of the tent. Yeah. I mean, what if it's been, you know, I don't know if I brought some inside and they were, I put them in a cup and I put like some rope and I glued it and everything. And it's been, I feel like it's been really long and it's probably just never going to become a butterfly, right? Because it only takes a certain amount of time. So maybe it's good. Yeah. Yeah, it's just. But in nature, less than 10%. Make it through the whole cycle. Yeah. No. I'm sure you're getting more than 10%. Oh yeah. Yeah. My husband says there's expected casualties, but I just hate it. Yeah. When you see like a butterfly that's crinkled or, you know, the chrysalis, you know, I mean, but I don't know if it's, I heard though that the chrysalis hatching time or whatever is due to the heat, the amount of heat we have like in different places and in the colder climates that it takes longer. Is that true? That is true. That is true. So I thought maybe because I brought them inside that it's a little bit cooler and that's why they hadn't come out, but. But the variation is just one or two days difference. Yeah. Yeah. It's not a week's time. Yeah, it's very sad. I'm just gonna have to give them. I'll probably just bring them outside. So you just have to focus on the ones that you were successful with. Yeah. Yeah. And never mind, you know, the ones that you weren't because you were successful more than 10%. Yeah. You're better than nature. Yeah. I did. I meddled in nature big time. My crown flowers, I mean, I don't know if you know much about crown flowers, but I feel like it's on the verge of death because it was like every single leaf has been eaten because I keep putting nuts around it. So I've actually been trying to fertilize it and water it. And I don't really think they need much actually because I had one at my old house. I never meddled like this. I never put nuts around it or anything, but I started to become interested in seeing what would happen if I did it. And so now like my crown flowers completely obliterated by these butterflies, you know. It's because of the February boom. But if you, I use Miracle Girl and it will come back. Yeah. It's very resilient. It's a very resilient plant considering all the suffering that it goes through, you know, with these butterflies. But that's why we have the plants, right? Yeah. That is why we have the plants. Not so that the plant looks pretty. It's because we want to feed the butterfly with the caterpillars. That's true. All right. I guess we should probably, it's probably near time, Max. I think I'll start wrapping it up. This is Dr. Grace O'Neill. This is Healthy Planet on the ThinkTech live streaming network series. We've been talking with Darlene Lou McDowell about monarch butterflies. Thanks to you all for being here. Thanks to Max, our broadcast engineer and the rest of the crew at ThinkTech for hosting our show. And thanks to you, our listeners for listening. I'll see you on March 17th for more of Healthy Planet on ThinkTech, the show for people who care about their health and the health of our planet. Our next show features guests, Laura Leigh Blanchard of the Lailani Farm Sanctuary. If you have ideas for the show, please contact me at HealthyPlanetThinkTech at gmail.com. Check out my website at greasonhawai.com for more information on my projects, including future show guests. I'm Dr. Grace O'Neill. Aloha, everyone.


Hi everyone. Thanks for coming out to my talk today. My name is Travis Adair. I'm an engineer working at Uber on our AI platform. And today I'm going to talk about Horovod and its role in our Michelangelo machine learning platform ecosystem and how we use it to build deep learning pipelines end-to-end on top of Spark that are scalable both at training time and at inference time. So to kick things off, I want to spend a little bit of time talking about the background on the Horovod project and how it came to be and what specific problem we're trying to solve with this technology. So the early days of Horovod at Uber were centered around solving the problems of scaling machine learning training for the ATG group, which is the group that works on autonomous vehicle technology. And the problems that they were facing were mostly around their models taking a very long time to train on a single GPU. So they had these big detection models. They had many GPUs available in their cluster, but they had no way to tie them all together. And so training using a single model and a single GPU was taking on the order of days or weeks. And they wanted to bring that down to hours or minutes. Additionally, some requirements they had was that whatever solution we came up with had to support both TensorFlow as well as PyTorch, which were both being used at the time. And also to make it easy for the ML engineers to add distributed training or scalability to their neural network training without needing to become experts in the infrastructure of how to scale things up. So there's a quick background on how deep learning training works. In this very simplified diagram, we have some data coming in, some training data. We do some preprocessing, some future engineering. And then it goes through several layers to output a prediction. And then after the prediction is made, we compare it with the true label and we get a gradient that tells us essentially how off we were and then that's used to adjust the model. Now these gradients, there's some problems if you were to think about how to parallelize this in a very straightforward way using something like MapReducerSpark, namely that there's very limited opportunities for parallelism in that these layers are typically applied sequentially.  So in practice, there are a few ways that people have approached distributed training in the past, model parallelism and data parallelism. And model parallelism, the common approach is that you will take your layers and divide them across different GPU devices in your training. And in data parallelism, what you instead do is take separate replicas of the model itself and use a separate portion of the data to train that model independently and then do a global update on all the models at once using the aggregated, the aggregate gradients from all the models separately. So in practice, data parallel training tends to be the preferred approach that most people use. And one of the earliest examples of it was an approach known as the parameter server. We essentially have this single shared piece of state that all the separate replicas are sending updates to. And then at the end of each step, the parameter server will send back the combined updates to each of the models and they'll use that to update the parameters of their model. Now, parameter servers have some advantages. They are fault tolerant by nature. They can be used to run asynchronous SGD, which allows different workers to update the model at different frequencies. But there are some downsides as well, one of the biggest ones being historically usability. So tight coupling between the model and parameter server is how some of the earlier frameworks approached it, where you had to write a lot of scaffolding code to get your model to be distributed this way. And scalability being the other big challenge where you have this kind of mini to one communication bottleneck and so suddenly your parameter server is adding an additional layer of hops in the networking that slows everything down. And while async SGD may seem like an advantage at first glance, there are actually some problems if you choose to use it. Convergence being a big one where an update will come in, but the update is no longer very useful. And as a result, it actually does more harm than good to the model. So in practice, most people will use synchronous stochastic gradient descent, not asynchronous. So Horovod was brought in as an alternative to the parameter server approach. It was based on a research paper that was published by Baidu back in 2017, I believe. And its approach to the problem was to have a framework agnostic layer that can be used for TensorFlow, Keras, PyTorch, or MextNet and uses various high-performance APIs and features, including NVIDIA's Nickel library for fast communication between the different workers. GPU Direct, in order to speed up communication between GPUs, RDMA, as well as a special algorithm that we call TensorFlow Fusion that we use in order to reduce the number of independent network calls that we do in the system. And it's very easy to use, just add a few lines of Python to your additional training script and you're good to go. And it's been open sourced as part of the Linux Foundation and can be installed very easily with just pip install Horovod. So in contrast to the parameter server-based approach that I showed you before, Horovod's technique is based on an MPI concept called AllReduce. So in AllReduce, the gradients are sent around in some sort of decentralized topology, usually a ring, where each of the workers forwards its gradients to the other workers and then at the end, each one of the replicas has its own copy of the model parameters without having to go through a single shared piece of state. And in practice, this scales very well. We've observed over 90% scaling efficiency on common perception networks, as well as many common natural language processing networks up to hundreds of GPUs. And in fact, the Horovod has been used to win the Gordon Bell Prize for the first deep learning experiment to ever reach one exaflop per second of performance. And that work was done by Oak Ridge National Lab with something like 27,000 GPUs on the world's largest supercomputer. So now that I've hopefully convinced you that Horovod performs well in isolation, let's talk a little bit about the unique challenges of doing deep learning in production. So one of the recent trends that we've observed at Uber is that deep learning is now achieving state of the art performance on not just image classification and natural language processing tasks, but also on tabular data-based problems as well where existing tree-based models now need to be migrated to deep learning. And in these sorts of problems, what we find is that there are many features, but every feature has low average quality as compared to image classification or natural language processing. And as a result, there tends to be a lot of iteration between the feature engineering steps and the model training steps. So going back to this diagram for a second, what this means is that in this pre-processing step here, this is where we do our feature engineering. We would typically do this in, say, something like Spark, and then we need to do our deep learning in the green boxes to the right using a different framework like TensorFlow, PyTorch, plus Horovod. And that we need a way to kind of seamlessly migrate back and forth between these two as we're doing our model development. Now, in research, that's what it looks like in isolation, but in reality, in a company like Uber, what we end up having is that the deep learning itself just ends up being one small part of a much larger picture. So we, in reality, have data coming in from a stream residing in a data lake that needs to be processed into features stored in a feature store that then ends up being used to train models that are pushed to a model store that are then used for serving online or batch predictions that themselves are either written to a data lake somewhere or sent back to a client who's maybe using their phone to get the latest updated ETA for their Uber ride, something like that. So in reality, we need a lot of infrastructure to support this, not just fast training using something like Horovod, but it all needs to fit together seamlessly. So the first part of this diagram that we want to talk about is the feature store itself. So in Michelangelo, we have data residing in our data lake. So this would be something like HDFS that we run ETL jobs to process that into features. We have a feature store that we call palette that we've written about before that indexes all of these and makes them available with very low latency to different models that need to use updated features like say what the seven-day average click-through rate was for a particular user, things like that, so that individual data scientists don't need to constantly be recalculating these features. They're just available as a service provided by our platform. So once we have the features that we want to use to do our training, how do we actually do the training on Spark? So historically, Spark has not integrated natively with systems like TensorFlow or PyTorch, and indeed there are a number of challenges that come up when you want to try to fit deep learning into a Spark environment. So one of the first ones is how it relates to the problem of pre-processing. So in pre-processing, we have two different types. One is very straightforward, what we call example dependent, just image resizing or color adjustments. You only need to look at a single example to do them. But others end up being data set dependent like string indexing or normalization. And so this ends up being a kind of model itself that needs to be fit to the data before it can be fed into the machine learning model. And in Spark terms, what we call this is an estimator that needs to be fit into a transformer, and then we can chain these transformers and estimators together into a pipeline. So what we would ideally want to be able to do is take all of these steps of our Spark transformation and create the estimators out of them, and then additionally have an estimator that represents our model shown on the right. And this all can be wrapped into a single pipeline that we can use to do training and then later prediction. Now in practice, there are some limitations of Spark as I alluded to before. And one of the big ones is that the RDD Resilient Distributed Dataset format that it uses is very good for kind of these embarrassingly parallel problems, but not particularly great for deep learning. And so at Uber, we have a framework that we've open source called Petastorm that attempts to solve this problem by taking your RDD, materializing it as a parquet file and then feeding the data into your TensorFlow or PyTorch code by doing efficient, by making efficient judicious use of sharding the data, streaming it, shuffling, buffering and caching that data so it can be fed quickly to the GPU for training. So in practice, what this looks like is something like this, your tasks, your Spark job will have the data residing in an RDD format somewhere and then it will write it out to your storage layer, maybe HDFS or S3 as these parquet row groups. And then Petastorm workers running on each of your trainers, training workers, will read these row groups, break them apart into individual examples and fill what's called a shuffle buffer with these different examples so that the system can then sample these individual elements to form a batch that can be fed to the training process downstream. Now, in a larger context, what this looks like on Spark 3.0, which now provides something called resource aware scheduling, is that we can do our data transformation ETL feature engineering on CPU resources, write out the intermediate data to HDFS and then transition over to GPU resources in the same Spark cluster to do our training using Horovod. And then finally, we can run our post-processing back on CPU. Now, if your Spark cluster doesn't happen to have GPUs, which is actually the case for us at Uber Internally, what we do is we have a system called Horovod Lambda that we've run, which we're in the process of open sourcing now that allows you to run your data processing on CPU with Spark and then use GPUs residing a remote cluster like in the cloud somewhere to do the actual deep learning itself. So as an example here, you can see we have this AWS job context that we use to spin up some number of instances. And then we fit this into our Keras estimator here as a backend that we can then use to fit our model. And this will do the training automatically for us in our remote cluster. And here's a visualization of what this process looks like on premise.  So that's the training side. What about the prediction side? So in practice, the way we serve it is that we have a front-end layer that serves the initial request coming in for the prediction, which is then routed to a particular instance. So we have these different service groups that handle different models so that each service group has its own isolated SLAs in terms of how many instance replicas it's going to have, et cetera. And then that online prediction service backend is going to spin up a Java interpreter that will run the Spark transformer steps very quickly. And then for the model prediction itself, we have a separate containerized environment that we use to execute the trained model prediction that uses shared memory plus GRPC over the Unix socket in order to minimize the amount of copying of the data that goes on. So with shared memory, this ends up being a very low overhead process. And this is all facilitated by a framework called NeuroPod that was recently open sourced by UberATG that allows you to take a model written in any of a number of different frameworks including Keras, TensorFlow, PyTorch, et cetera and then wrap it into a single API that can be used kind of framework agnostic in order to make your predictions. And NeuroPod supports a system, a technique called out-of-process execution. And what this allows us to do is have a single server that is running an arbitrary version of, with an arbitrary set of dependencies that can then support inference on various versions of PyTorch and TensorFlow by just having them available somewhere in the system and then quickly spinning up a separate process in order to do the actual prediction without having to load multiple versions of TensorFlow in the same processing environment, for example, which would of course be problematic. All right, so stitching this all together, let's talk a little bit about workflow authoring itself. So in our framework that we provide at Uber on the Michelangelo platform, what this looks like for our customers is something like this. So they first have their ETL step where they will query their initial set of features from the data lake. So something like this Hive query or Spark SQL query. And then we have some set of features that we want to join on this raw data. And so then we make the request to Pallet, our feature storage layer, to get the actual materialized features from that raw data. And then the next step is we want to process these features in some way. So for example, maybe we want to convert strings to floats or indexed strings in some way. So they're given a numerical value. We can do all this using typical Spark pipelines and standard train test split. And then finally we do the apply the transformations to the split data. And then next up comes the model construction itself. So all of this is just normal TensorFlow slash Keras code. You want to take every one of your labels and create or every one of your features and create a separate input to your model for it. And then create your standard in this case, feed forward model. And then as well as have your output layer correspond to the label column in your data. And then finally create a Keras model itself from all that. And then we can take that model as well as an optimizer and a loss function and then create what we call this Horovod Keras estimator from that model and give it a set of parameters say, you know, what batch size do we want, how many epochs we're going to run for, what validation split do we want, et cetera. And then we just call fit and then the Horovod spark estimator will go ahead and do the work of using Pedestorm as well as Horovod, the two frameworks I mentioned before to do the actual training. And then it spits out this model transformer, which we can use on our test data set to get our final set of predictions and run evaluation. And then once we're happy with it, we have internally this part has not been open source yet but it's coming soon now that NeuroPod is open source as well. Finally, we call build servable model to create a NeuroPod which we can then stick into our serving pipeline with our data transformation steps. And then finally upload this to our Michelangelo project with a description, et cetera, and some test data for sanity checking and do the actual deployment into production. So that's the state of the world today in terms of how machine learning and production works at Uber. I want to quickly spend a little bit of time talking about what's next for us. So one of the big things that we're doing in the Horovod community is working on what we call elastic Horovod. So this is a fault tolerant Horovod that is capable of recovering from a single host going down or a new host being added to the system dynamically. So in this example you see, for instance, host zero has an error of some kind but host one is fine whereas host two over there has just been discovered. So ideally what we'd like to do is go from this scenario to something like this where we remove host zero from the system, add host two, and then let training proceed as normal. And so we already have this committed to Horovod. You can go try it out today if you'd like. At a high level, the way we do this is that we have the driver, this driver process similar to a spark driver that starts up and creates the initial workers. They synchronize them amongst themselves. Their state training proceeds as normal. Once an error occurs, the worker that's still alive will send someone we call a reset request back up to the driver, which will then look for new workers and then decide to go ahead and launch it at which point training proceeds as normal until either a similar error occurs or things go back to normal or training completes. And so I encourage all of you to try this out. So as I mentioned, Horovod, Pedestorm, NeuroPod, all of these are open source available on GitHub free for you to try out. Please let us know if you have any issues or concerns with anything or would like to contribute. What I'll do quickly is answer some questions here for those of you who have asked some questions. Thank you very much. So the first question, are there any container orchestration features baked into Horovod? So that's a good question. So we don't have anything that's tightly coupled to a particular container orchestrator today, though there are a number of tools that have been written in order to run on top of different container orchestration frameworks that are out there. So for instance, if you want to run on top of Kubernetes, there are some utilities out there that can do that for you. Internally at Uber we primarily run on Kubernetes as well as our internal orchestration framework or orchestration layer called Peloton. But we will be actually contributing some more container orchestration specific capabilities to Horovod in the near future, particularly for the elastic Horovod work because it's a little bit, I guess what I should say is that today it's fairly straightforward to run Horovod yourself in a, just kind of an bare metal way, but running elastic Horovod is a little bit more complex and requires more integration with your container orchestration framework. So we will be adding support specifically for that and that's coming soon. So the elastic Horovod ETA, so the question is what's the ETA for elastic Horovod? So as I mentioned, there is a preliminary version that's already been committed to master up on GitHub. So you're welcome to go and download master, try it out yourself. We have some documentation there on how to use it. It's pretty straightforward to use elastic Horovod for basic fault tolerance today. And it's going to be, and it's, I should say the running on top of Spark with elastic Horovod isn't a pretty good state as well. Running on a particular container orchestration framework, as I alluded before, is one of the areas that we are still working on in order to make it convenient for people to do so. So they don't have to write a lot of code themselves, but that should be coming soon and we actually have ongoing projects internally to do that. So my estimation would be sometime later this summer, probably August or so, we'll have hopefully our 0.20 release, which will have full support for elastic Horovod. And I encourage you to continue following along with our progress there. And definitely if you try it out and anything doesn't work right, please let us know on GitHub. I'm always happy to fix any issues that come up. All right. So next question is, what is the comparison between Horovod and Databricks MLflow? So this is a good question. I would say that in general Horovod solves a problem. Horovod itself solves a problem that's pretty different from what MLflow itself is solving. But I would say that MLflow is very analogous to the Michelangelo machine learning platform at Uber as a whole. So some of the key features that MLflow provides would be experimentation, tracking, model management. These are things that our Michelangelo platform provides as well. And I would say model management being one of the bigger ones. But MLflow is pretty agnostic about how you do your training, how you do your productionization of machine learning. And so in that sense, I would say that there's actually a very little overlap between the two. So I think that's both a good thing and a bad thing about MLflow. So MLflow can fit into just about any environment you can imagine taking everything I showed you here and adding MLflow as an additional layer to track your model versions, your model experiments, et cetera. But the downside is that it doesn't give you as much expressive power, I guess, as some other ML platforms that are out there. But in general, you can definitely mix and match the two. All right. And don't see any other questions coming in yet. So unless there are any more questions, we can definitely stop there. Thanks, everyone, again, for coming out to the presentation. And if you'd like to follow up with any of the topics that were discussed here, you can reach me on GitHub at tgaddar. And of course, always happy to answer any requests about the project or any of the things we've worked on.


Let's do a little recap of the Dodger's big inning last night in game two of the NLDS against the Giants. They're up two to one. It's top of the six inning. Gossman's in there against Trey Turner. 0-1 count. He's going to smoke that one for a double. Down the line. Easy enough. Lead off runner on second base. Helmet was bobbling the whole time. Couple guys let this helmet fall. A lot of speedsters end up without a helmet, not Trey. He needs it because he needs to do the celebration. And that looks like it's shot, zoomed in from the moon. Will Smith takes ball one. Will Smith takes ball two. Will Smith takes ball three. Will Smith takes ball four. The takes on pitches two and three in that sequence that I just showed you are so impressive. Look at this pitch. That is in the zone, in the zone, out of the zone. And he took it the whole way just spat on. In the zone, in the zone, out of the zone. Good pitches by Gossman. And they're right in the same line and they go different ways. It was just a really impressive at bat. Good takes by Will Smith. So hot boy Gabe says, let's bring in a new guy. That's a ball out to Chris Taylor. And that's a ball out again. That's in play. It's up. Is there going to be a play made on it? Wilmer Flora is running the right side. I think a couple years ago, before that netting was mandated, he might have had a play on this. Because look at him lunge here. That looks like a confident lunge like it would have went right into his glove. Instead, it hits the netting, fouled back again. Two twos to count. And they really wanted that pitch for strike three. They really wanted it. It's top of the zone. It's right above the belt. I don't know. Pretty borderline. I think it goes either way. So now the count is full. And that's ball four. Which means bases are loaded with one out. Which means a fly ball scores a run. Which means a wild pitch, pass ball scores a run. A walk obviously scores a run. So as a batter, if your bellinger coming up, you're thinking, well, fly ball scores a run. I should swing early because he doesn't want to get behind in the count. He doesn't want to walk me.  He's missing the zone. Be patient. Make sure he comes to you. Two different schools of thought. Cody Bellinger, he takes route one. Says I'm going to swing early as soon as he fills up the zone. Sends that one to the Bank of America sign. Is it out? No. It's off the wall. It's a two run double. Belly comes through in the post season. Got to feel huge. He does the helmet celebration. He calls himself safe. Guys in the crowd are screaming, belly, belly. Don't like that. Let's go, belly.  OK, cool. Nice swing by belly. No one really reacts because they're the home crowd. In that shot, they reacted. And then a pollock comes up, and he smokes that one down the line. And that's two more runs. And the Dodgers have themselves a big inning to blow the game open to tie the series. Pollock jacked up.  Ryan Roberts, giddy. Ha, Tim, let me see what I'm looking like in the background. Well, Smith, did you see my walk? Cody, just looking around, taking it all in. This one brought to you by the shop at Johnboymedia, shop.Johnboymedia.com. We've got the JM Essentials line. 15% off, and new products are dropping every week. We use promo code PLAYERS for 15% off JM Essentials items throughout the postseason.


So TikTok is always changing, and one of the biggest complaints about the platform is that self-promotion, well, falls really flat. Since when you make self-promotional posts on TikTok, well, they flop more commonly than when the television networks do another terrible show where adults talk about how they don't get pronouns to these damn kids with their TikToks. For the last half year, I've really seen that change as users have learned to make self-promotional TikToks more intriguing. I mean, you see these videos of people lip-syncing into the camera? Those are working well. And yes, many of the unknown DIY artists this works for are sometimes posting videos of them doing this in various ways 30 to 60 times before the earworm of the song hits the people who regularly get to serve their videos, and they watch them through. And after the earworm is in their head, they watch them repeatedly and drive up the video till it starts to go viral. Since self-promotional TikToks seemed not too much, the next deduction was what's the purpose of all that time and money you spend on a music video if it's useless on TikTok? Since TikTok breaks so many artists, why bother with your limited budget on a music video, right? But I've seen how you can make music videos that will build relationships with fans and get them to listen to your full song and deepen your fan relationship while also making them work on TikTok to be great content for your music that really does numbers. So this video, I'll show you how to turn your music video into TikTok content the algorithm likes. So one of the more weird things I've experienced when I'm consulting with musicians is that lately they really are skeptical that spending money on music videos is worthy of their budget. Since when you see the potential spread of TikToks, reels and YouTube shorts, well by the numbers it seems weird to invest in the music video since you want to go mega viral which I have to say is a little shortsighted since what I hear every musician who has a viral TikTok or a real or short complaint about is that it doesn't then equal streams and what they really want is streams yet a music video is actually streams of a song and they're thinking about neglecting that. I seem to recall all of you wetting your beds when all the major labels drop your favorite artists for not hitting numbers but you say the artist is everything to you but they're dropping them because they're not getting numbers and you're concerned about those numbers. Funny how that works, isn't it? Then the other thing musicians complain about is when they just get one song to blow up on a playlist or on TikTok is that no one forms relationships with them but the number one relationship builder with an artist is a music video and yet they keep thinking about cutting that from their budget but no one ever said musicians had logic unless it's installed on their Mac. Thanks everyone, I'll be here all week. But why not think of a strategy that tries both short form views on TikTok and can build relationships. Since like many things there's more than one way to skin a cat. Wait, why are we skinning a f***ing cat? I love cats and I definitely don't want to see any more people wearing leopard prints since I grew up in New Jersey and saw enough of that as a youth. Anyway, this looks at the problem as black and white or an either or issue. Instead of asking the question how do I film a music video that I can turn into numerous TikToks that will do well in the algorithm while making a music video fans will love and watch repeatedly to build a relationship with, which then gets those fans to share it and send it to their friends in its long form and your fan base to spread. You can do it all in one. And my friends, I'm going to show you how Caroline Paulcheck did just that. Now if you're not familiar, Caroline is one of the most innovative artists of our time. Really ahead of the curve and frankly can sing better than 99.9% of the planet without being the least bit American idol show off cringe stuff. And while she's been on a major label in the past, she's technically a DIY artist who owns her own label right now. She's also familiar with TikToks going viral since her song and dance for So Hot You're Hurting My Feelings turned into a trend on TikTok and spread the song massively. Anyway, so I've seen this technique done a little bit, but it really hit home when I saw her do it and then I was listening to her appearance on my favorite podcast How Long Gone and she gave up the game a bit and it all made sense. Let's listen. So welcome to my island video, for example, had 18 scenes and the idea was that each one of those scenes could get posted individually as their own reels or TikToks or Twitter posts without compromising the integrity of the video at all because there is no narrative, the sequences of no significance and the chaos of the jumble as the whole point. So no matter how you consume it, you're getting the point. Each one of these scenes is the one stop. So technically the edit was very simple. So what does she mean by this? So if you watch the video for her bagger and one of my favorite songs of 2023, Welcome to My Island, you can see it's exactly as she describes 18 scenarios where it seems like she filmed them through the entirety of the song and then shows which part of the video to use each scenario for.  And all she needed to do was chop them down so they'd loop well for TikTok. And this is important since I would argue the relationships a fan makes with a video and the repeated listens make it way more valuable to invest your money. But like I said, this isn't an either or situation and instead one where if you think of the video properly, you could have no compromises in your video. And if anything, make it more focused and make it so you can turn it into short form videos in minutes by really thinking about how you frame someone in it so it converts to 16 by nine and how you're going to make it so that these scenarios can be used in short form as well. Since let's always remember no one who smarts goal is to just keep users on TikTok. It's to get them to build relationships with you and music videos are how you do that best. So if you show all these people who are bored enough to sit through hours of Biden, Trump and Obama gaming while surely seeing that a song they like has a music video is often a better alternative because these are sure getting old fast. Now let's be honest here, some of you who actually have functioning brains are like, Jesse, if videos where people just sing in their bedroom or working for getting viral TikToks, I'm just sticking to that. And that brings me to another lesson that we have seen time and time again. So if you're familiar with the concept of divergent streams, this is what a divergent stream looks like. Essentially, it's a theory I often talk about where each side is going further and further away from each other as time goes on and the middle becomes increasingly irrelevant and something no one consumes or puts any attention towards. For example, throughout music, you have things like in the 2000s, you had in syncs song pop along with Shania Twain that are the most maximalist over productions in music history. Seriously, listen, these productions are way more overproduced than anything on the radio today. And then on the other side, you had the strokes and the white stripes who were achieving massive success with records that were barely produced at all and as raw as can be throughout time. This exists even today. And it's exactly why you see the popularity of a bedroom pop song like Steve Lacy's Bad Happen exist.  Anyway, the way this relates to your videos and TikTok is you're able to do both with the music video as you're able to take advantage of the high production music video scenes you shoot and still do some lo-fi stuff. You shoot yourself on your phone singing in your backyard and take advantage of two things audiences love, both the lo-fi and the high-fi. The raw, honest vibes of you and your camera in a relatable backyard scenario and then the spectacle of a great shot in your music video that has one hell of a vibe help to show your range to people and see that you're an artist worth paying attention to. The fact is what works on TikTok right now is a variety of promotions and not just throwing content at the wall, but instead making content that has a lot of movement and is intriguing and both raw and polished can be intriguing. And like I was saying before, so many TikTokers are doing 30 to 60 TikToks of the same song before it blows up and just pushing it until it goes viral. So if your next video you take the simple philosophy that's been done for videos for the past 50 years, where you perform the song in various scenarios all the way through, you can then take from each scenario you filmed in the part of the song that has the hook that you shot and simply crop it to work in a 16 by 9 portrait frame. And well, you've got yourself short-form TikToks. You can repurpose and help drive views to your songs and streams. And with that music video, you now have some of that higher quality content that can build relationships. And if you pepper that with low effort content like lipsticks, well, you're on your way to having a ton of content to promote each song with. But some of you may have noticed that Caroline doesn't do exactly what I say, and she plays different parts of the song in her TikToks and not just the hook. Now from what I've observed recently on TikTok, the real key a lot of these unknown artists who go from let's say zero monthly's on Spotify to hundreds of thousands. What I see over and over again is the trick is repeating the hook. We have to remember Caroline's pretty established. She's been around for a minute, has a preexisting fan base. She has to play by different rules than you. And while you can learn from her, I think what I've really learned from all the underground artists who are breaking with this technique is they're repeating the same hook of their song over and over in the videos. They choose one part of it, and that's every video over and over and over again. So I would encourage you to get the color correct inversions of each scene in your video where you sing the hook and then make your shorts for TikTok, YouTube and Instagram from those. Since that takes minutes. And if you have 18 scenarios in your video like Caroline, well, that's only 12 different times you need to lip sync your song in the shower or twerking on top of your grandfather's head. If you want to take this even further, I've seen Samaras sing their future singles in video scenarios for future content and then use that later. Since let's remember our goals with TikToks is to just be in scenarios where people will watch to the end because they're intrigued by what they're watching. If you got a good scene, well, that doesn't go stale fast and you can use it for a couple songs. Another great trick I talk about all the time is that whenever you're filming a video, half the battle is getting the lighting and setting the scene. So once that's done, film a few costume changes in that scene and get various songs in those different costumes. If it works well, so you have tons more content to pick from. And oh yeah, when I say costumes, don't get intimidated use slabs. Sometimes I mean just changing your dirty jeans and maybe changing your shirt. I know that's a lot to ask from somebody. And a lot of you may still feel like you don't have the budget to do any of this. But honestly, I see ours every day making amazing videos where they have one of the recent generations of phones, one 200 ish dollar light with some diffusion that can create a lot of light in a room. And then this $100 gem from aperture, which is called an emcee that can do light in any color. And honestly, that's all they need to make videos that get tons of views and look really good.  A little color correction tutorial after you film some stuff and you're there. So now that you've learned how to make your music video go viral on tiktok, I bet you'd like to know how to make the best music video possible. Well, you're in luck because I made a video on that which is on the screen now or in the description, which you should really learn since it will help give you another piece of this puzzle. Thanks for watching.