The greatest Asian-American sauce right now is having a huge shortage to the point where some people are reselling bottles for almost $120. David, could this be the beginning of the end for Hoi Fong Sriracha? Yeah, well, probably not to be honest. If it's going for $120 a bottle, it's just like, it just goes to show you how much people are fiending over it. I mean, CBS News has an article saying where did all the Sriracha go? Sauce shortage hiking prices to $70 in online markets. People are debating the reasons why. People are really getting into this, Andrew. And it's not just within the Asian community or Asian-American community. This is within non-Asians as well. No, I would say this is just general American foodie news because everybody loves Sriracha. Sriracha is a household name now, by the way. I sure like that spicy ketchup, man. Ever since I got introduced to it 10, 15 years ago, I can't get enough of it. And it is uniquely American. So anyways, guys, we're going to go through the comment section. We're going to talk about why there's a shortage and then also talk about what are some other hot sauce alternatives that you might be using. But anyways, guys, please hit that like button, check out other episodes of the Hot Pop Boys. Andrew, people are talking about this, like it's like Bruce Lee's death or something like that. You know what I mean? There's non-Asians spending their days online debating about the reasoning. Is it because of dispute with the suppliers? Is it supply chain? Is it climate change? And then people are tying it to this and that. Does that go to show you how deeply Sriracha has sort of penetrated the mindset of the young American millennial? Yeah, I would say Sriracha is the greatest Asian-American food item that is uniquely Asian-American. So there's Panda Express, which is a brand, but I would say Hoi Phong Sriracha specifically because it's made in America by an Asian immigrant. With California ingredients, right? With California, it is very Asian-American. So we should, Asian-Americans should be proud of Sriracha, for sure. Long story short, Andrew David Tran is a refugee from Vietnam. He's Chinese-Vietnamese. He came over to America. He started Hoi Phong Foods in 1980, and he sort of used local California ingredients to reinvent a local Thai sauce from a place in Thailand called Sriracha. Yeah, and you know, as a Thai person at first, I just wanted to educate everybody that Sriracha was not authentically Thai, but since it does make Sriracha sauce so popular, now we get to bring in more authentic Thai Sriracha sauce. Yeah, hey, Thai people are pretty chill. They weren't mad at him. No, I don't think they were. We simply seek to educate. No, we not mad at David Tran. Anyway, let's get into the comments section. Of course, our own takeaways, Andrew. Somebody said, hey, yeah, I've got a sealed bottle from Costco from a couple of years ago. Somebody make an offer, man. No lowballing. I know what I've got in my hand. What do you think about people paying? Somebody said I paid $50 for two bottles. Yo, that's crazy, but if you love it that much or you need it for your business because a lot of restaurants do use Hoi Phong Sriracha flavored things or use it in their recipes, yes, I'm sure you are very worried right now. You know why I think people love it, Andrew? It's because it is a cheap sauce that tastes expensive and you can put it on cheap food to make that food have a lot of flavor but without tasting cheap. Because sometimes if you pick an extra hot Tabasco, it's very thin and has a vinegar vibe to it. It can ruin a dish. Yeah, yeah, yeah. But Sriracha doesn't do that. No, Sriracha is a good sauce but I will say over the years I ate a little bit less of it. I stopped putting it on everything because I'll tell you the other chili sauce alternatives that I use even for my pho. Anyways, let's keep going. Somebody says in Vietnam, people don't even use Sriracha. They use Chin Su or Tolu Mex. Yeah, so I think this is the thing that a lot of people don't realize is that Hoi Phong Sriracha is from America. That's, even though you see it on the tables of all the pho restaurants in America, it's not from Vietnam. So that's not what they're eating in Vietnam. Right, it was just made by a Vietnamese refugee. But it was based off of a Thai sauce that they don't even really use in Vietnam. Yeah, I heard that because Sriracha got so popular here at certain pho shops in Vietnam, they started using it but really it's not traditionally a Vietnamese thing. Yeah, and I believe, I don't, yep. There was a documentary about this in 2013. He was exposed to it by either living in a refugee camp in Thailand or knowing some people who did. Right, I think he did spend some time in Thailand, yeah. Shout out to David Tran, Andrew. We've actually met him. And yeah, I spoke Cantonese to him before. All from the six to six, man. Hey, shout out to Rose Mead. Somebody said, is this only in the US? Because here in the UK, it costs the same as it has before. Yeah, I would say basically overseas, this effect has not rippled and been felt overseas. But believe me, if Sriracha really is in a shortage, then you're gonna start to feel it overseas. I think Andrew in the UK, they're not buying it up as constantly. They're probably still on the shipment from like six months ago. They're not like putting it on everything. Right, now is maybe David, this hike in price, is the shortage built in and there's not a shortage at the moment? Or is there? Well, long story short, I heard the big dispute and we're gonna get into this in the next comment that it's a lag between the legal thing, the pepper shortage from the drought of Mexico and how it translates to the production process. So you might have heard about the thing that caused the drought a year and a half ago, but you don't see the drought until now because that's how the supply chain works. Right, because the sauces take some time to be made. Yeah, so anyway, Andrew, let's get into the drama surrounding what underwood farms. They got, there was a crazy lawsuit involving huay fong foods where there was millions of millions of dollars involved, Andrew. Why do you think so many people are interested in this from a legal perspective? They're almost studying this like a crime podcast. No, I would say this, Americans, and I mean Americans as a non-Asians, are kind of taking on sriracha as their own. Like, they care about it is what I'm saying. And that's why I would say, sriracha really blew the lid open as far as like spicy food. That and Sichuan food really made like, for example, white people get into super spicy food. So, and I think that a lot of white people can eat spicy foods now due to kind of being introduced maybe via sriracha. Andrew, there have been multiple articles saying that Americans' median spiciness level for their flavor profile or their tongue has just been going up and up and up every year. And sriracha is a huge part of that. So is Chipotle, you know, a lot of the Mexican sauces from Mexico. Right, right, like a Valentinas or... Cholula. Cholula, yes, my favorite. I forgot the name. Moving on, Andrew. Somebody said there are other hot sauces that are alternatives. For me, I like Kroger. I like this one. I like that one. What do you think? Because it's true. Ever since sriracha, so sriracha's been around since 1980, guys. I looked it up. They said it didn't blow up and get this cult non-Asian following until 2010. So that following's only been going on for 13 years. That's people dressing their kids in sriracha shirts, by the way, which is crazy, Andrew. You can find on Google image search more pictures of white kids dressed up as sriracha bottles wearing sriracha shirts than you can ever find any Asian kids. Is sriracha the most accepted Asian product in America? Because think about it.  That would be even more weird. You mean like Kiko Mons or something? Yeah, because sriracha is a little bit more whimsical. It's red and green. It has a cool look. It's also, I think for some reason, like even though white people may not always think about how it's from America, I think it being from America does help. Yeah. Dude. I think it is. There needs to be more investigation. I'm not saying there's not some YouTube videos analyzing this sort of like cultural pop cultural sub genre trend that happened around sriracha in the non-Asian community, but it needs to be examined even further. Because what you say, Andrew, in the Asian community, people always use sriracha, but they would just use it almost as a sauce. Yeah, I don't love sriracha any more than I did 10 years ago, but I still like it.  Like that teacher in third grade that told you you could be anything you wanted to be in your life. Well, you know, I got to give sriracha a lot of credit. You know, after eating that in college, I really got into Asian food and Asian cuisine. Then I went on a trip over to China and Vietnam, and that really changed my life. Now I've got an Asian wife and half Asian kids. Now I'm married to a Chinese Malaysian woman, and I got three kids. Dude, honestly, that obviously is an extreme 11 out of 10 case. It's probably happened. Somebody said the Kroger's version is actually pretty decent. I got something to say about the Kroger's sriracha. I actually like it, but unlike real sriracha, if you use it more than three or four times in a row, you will start to feel sick of it. I think that Huay Fong food sriracha, Andrew, you don't get sick of it. Yeah, another notable one is Trader Joe's is not bad. Yeah, those are not bad. What do you think about this argument? Somebody said, it's not even that good. Real foodies know that sriracha from Huay Fong is actually such a mid-hot sauce. I've heard this even from Asian people before. Not a lot, though, but I've heard this from a lot of different types of people. So can I tell you this? I don't think it's mid. I wouldn't use the word mid, but- That's extreme, that's extreme. But I use the garlic chili sauce and huay sin sauce with my phone now. I do not dip my meat in sriracha. I think meat in sriracha, the sriracha's too overpowering. It's actually too strong and too spicy and kind of takes over the meat. So for me, I just don't understand how people could squeeze the whole sriracha bowl. You know what?  I would compare it to, and to me, Huay Fong food sriracha is almost like the Rolex watch of srirachas. You know how you can argue that like, Patek's or like, you know what I mean? Like AP's got better movement or even like this Grand Seiko has better movement than a Rolex due to the intricacies. But at the end of the day, people aren't just gonna get a Rolex Submariner. Right. It's just the one that made it in. Is it the Toyota Camry of hot sauces or it may be the Lexus? Or BMW 3 Series.  You know what is?  Nobody's going against it. Right, right, right. Somebody said the main thing that I think the sriracha has good is the flavor behind the heat because it's the first ultra hot sauce that's really not that hot in an over way. It's only hot in a sweet, warm, low way. Right. No, I mean, it is still a good sauce, guys, for sure. And I think it is still very unique in its own way. David, of course, this comment had to make it political. They were like, yeah, I used to like this product, but I'm not buying it at these Joe Biden prices. Right, obviously connecting Joe Biden with crazy inflation and et cetera, et cetera. I mean, but it's funny because Andrew, like we said, anything that makes it to the non-Asian world, it becomes politicized.  It's ruining our favorite hot sauce because now they got droughts in Mexico where Huay Fong gets the chilies from. So this became politicized on both sides. Right. Anyways, David, going to the takeaways. First of all, I'm just gonna go ahead and say, no, this is not the end of sriracha. I think they will figure this out. Droughts in Mexico are somewhat common. However, it has been that they are more common as of the last few decades. Now that's why a lot of people attributed to climate change. So, you know, and, but also people think, oh, climate change is not necessarily man-made. It just happens all the time. So you just kind of have to deal with this. Dude, everything just opens up such a multi-layered onion of arguments in 2023. What do you think about people saying that sriracha is just a passing food trend? Because they said pesto was big in the 80s and 90s. Then it's sort of- It's still around though. And then it switched to ranch. You know, for a while, people were super into ranch in America. They're a low-fat ranch. This ranch, Hidden Valley Ranch versus Paul Newman's versus, I guess ranch was really big. And then it switched to barbecue sauce. And then it switched to Chipotle, which is more like opening up to the Latin American vibes. Interestingly enough, Andrew, Piri Piri from Mozambique to Portugal to all around Europe and Canada has still yet to trend in America from like Nando's like Piri Piri. Yeah. Collaboration chili oil is not that popular either. Let me tell you this, David. Sriracha will always be around. You know why? Because it mixes well with other things. You mean Asian food, non-Asian food, everything, right? It mixes well. It mixes great with ketchup. It mixes amazing with mayo. Think about it. Tabasco doesn't do that. Yeah. Yeah. Sriracha does. So I'm saying, guys, I think sriracha's around forever. And I mean, if it's gonna take some other Asian billionaire or even some white billionaire that's like, well, you know, I see the opportunity here. I'm gonna get you a farm, David. Mr. Tran, I'm getting you a farm. And you know what I think the key is to, Andrew, is it tastes kind of expensive, but the price at its retail pricing is actually very cheap. Yeah. I don't know who's buying it at $120 or $60. That's crazy though. But anyways, guys, we have our own chili oil that I wanna plug real quick. Shameless plug. Smala, click on thebeacons.ai. Link down below. You can sign up and get the updates for when the pre-orders start. Also more information on it at the Instagram. Andrew, is this the next wave?  Guys, it's from Smala, from Sichuan to Sicily. This is where this takes you. It is an Italian, Chinese-inspired chili oil. I'm very excited about it. Anyways, guys, Shameless plug is over. David, any last things?  And you know what? Bruce Lee, also an Asian-American icon, but also because he was validated by Americans, even though Asians still liked him, he was, after being validated and cemented as a legend by non-Asians, that's when he's like, God stats. Yeah, I think if non-Asians never got on sriracha, Asians were never gonna dress up as sriracha bottles on Halloween.  Yes, an American thing now. All right, everybody, let us know in the comments down below what other hot sauces do you use? What do you think about sriracha being the greatest, most well-known Asian-American sauce ever? Remember, Kekomon is from Japan and all these other brands are from Japan. I think, Andrew, my final word is that if you make me pick all the different sriracha alternatives right now, I'm still picking huaifeng foods. Well, even over the authentic Taiwan. Yes, for me. All right, everybody, let us know what you think in the comments down below. We're the Hop Hop Boys and until next time, we out. Peace.


I'm actually shocked that it sold because bread and butter item. Like I said, I cannot believe that this even sold on Poshmark. Hey, Bolo Buddies, thanks for watching. Amazing, right? All right, let's get started. Hey, Bolo Buddies, thanks for watching. All right, in this video, we are gonna be pulling orders, orders to ship. So it's basically a what sold video where I'm gonna tell you where I got it, what I paid for it, and what it sold for. Different format than I usually do. I usually do screen sharing, but this one is gonna be reseller vlog style. I'm also gonna mix in some plush unboxing that a viewer sent me. She sent me some free plush. I actually went and picked it up and got some incredible items to share with you guys. So stay tuned and thanks for being here. Should we? Bolo Buddies? Be sure to like, comment, subscribe and share and let's get started. All right, we're looking for tote 23. There it is. All right, so I sold a beach ball and I got it from the Goodwill Bins.  This one right here is a mega beach ball and it's a 44 inch. Now this one had like a dollar general store tag on it. And I don't think it's super vintage, but it's pretty big. And I sold this for $20 and the buyer paid shipping. Okay, we are heading over to tote 43 right here. And I'm looking for those brets.  Well, this is one of those brets, I do believe. And I sold this bret right here. It's this gold one. It's made in France. This bret, I sold this for $15.50 buyer paid shipping. So nice little Bolo right there, bread and butter Bolo made in France sold pretty quickly. All right, you guys, this is bag four of the free plush that I got from a viewer.  So I was able to go and pick these items up. This is I think part four. So we're gonna get started here. All right, I got my light box lit up so we can really see them here. And of course my dog's gonna eat. She always seems to eat. This is a Kohl's Cares. I've talked about that before. I don't typically pick up Kohl's Cares, but the Rebel reseller does and I trust her judgment. So I'm gonna start listing these if I get them. I don't know if I'm gonna source for them, but maybe if they're a quarter or something. This is, I don't know what kind of dog this is. I know that the Kohl's Cares most of them are not super valuable. Maybe Rebel will do a Kohl's Cares video on which ones are valuable. If you haven't checked out the Rebel reseller, she does videos on plush and she tells us which ones to be in the lookout for. This one's Batat. So those are those dolls. So I know you can take like a dog brush and brush these out. I am super lazy though. I never like to do that. So it's got something around its neck. What is happening here? All right, so this next item, I pick it up all the time if I see it at the bends. Now, if I'm at a garage sale and it's overpriced, I will make sure that I do my research before I overspend on it. So you gotta be careful with these. This one's a bread and butter. It is a bitty baby. And I got it at the Goodwill Bend. So I'm gonna show her to you. All right, she's over here in tote 17. And I pretty much bag everything anymore just to keep it organized. A lot of times I will put things in bundles and it just makes it easier for me to find stuff. Sometimes I leave it in the bag, sometimes I don't. But this is the bitty baby. I sold her for $15. Now, why did I only sell her for $15? I'm gonna show you why. She's just not perfect. But she can probably be cleaned up a little bit. I don't know.  I don't know if you can just throw this in the washing machine and put like a pillow bag around it. Anybody know? And just wash it up. It's just a little dirty. But it is American girl. And it is, and I just took good pictures and disclosed everything 2017. So she's not super old.  Is that creepy? Hello. Listen. Oh, her eyes stuck. Is that her eyes? Listen, is that weird? Hear it? All right, plush tote nine. This is called a jingle pals. And I got this also at the Goodwill Benz. And you can see right here it's got an on and off switch. I pretty much pick up anything that's animated. And I did put a video in the description of my listing. I sold this for $13.64 plus shipping. And it's just a little cutie. So that's going out.  Okay, so the funny thing is anything that is new old stock, switch this around, anything that is new old stock and it is lightweight, I'm gonna pick it up. Even if it's just a bread and butter. So this item that you're about to see sold for $9.92 plus shipping. It's just a bread and butter item, but it sold pretty quickly. And I'm not even really sure what it is. All right, it says that it's in this tote. Okay, so it says Dremel. So I think it's some sort of like, but it's in here maybe. Is it in here? Here it is. Look, right there. I think they're blades maybe. Yeah, five piece. So almost 10 bucks plus shipping, just for this little thing. And it weighs like, I don't know, maybe three ounces. It's $1.89 at my Goodwill Benz. So that's fantastic. So far, all five items that I have showed you have all come from the Goodwill Benz. It's like the original, so it's still got the original thing on it and it looks like that. Some of them stuffed animals, they just get scruffy fast. Oh, this one's cute. This one says good stuff. It looks like it's some sort of sports bear. Boston Red Sox has the original tag and look at that little cuteness on his foot. All right. Stay. I think there's a big guy in here. So this video probably won't be super long. All right, this one is a Martha, Russ Berry Martha. She looks like she's the fourth of July bear. She should be pretty easy to look up. This one here is Natalie. She is an angel bear by Gans. It's interesting. Never seen one of those. Okay. Ooh, baby's first doll. I like that. It's a rattle. I don't know if it's big money or not, but it's by Fisher Price. It's from 2008. And I'm gonna bet you that somebody is looking for that for sure. We're heading back over here to tote 17. This one came from a thrift store, paid $3. I'm not sure if this was like the best deal ever. So these furbies, I believe they're McDonald's. They all do like little different things. And I gotta check and see what I sold them for. Sold for $13 plus shipping, but here's the catch. They're going internationally. So the buyer also had to pay customs and all of that good stuff for these. So I don't know what they're all in for, but 13 plus tax plus shipping, I have to send it to Kentucky. I do use global shipping. I like global shipping. I know there's a new shipping program that's out. I know the Rebel reseller is using it. So far, I have not opted in. I'm happy with global shipping. I'm gonna stick with it for now and let them work out all the kinks and all that stuff. I've heard good and bad about it. So usually I wait. I'm kind of like, I'm comfortable with what I'm doing. I'm just gonna stick with that. So check out the Rebel reseller. She has talked about it in some of her videos. I know that Cha-ching King also put out a video on it. I have not watched that yet. Guilty, because I'm just really not interested in switching yet, but I may go over and take a peek at that and see what he has to say about it as well. His videos, he's got a lot of videos that are great for beginners. So if you are new to eBay and you're learning how to sell, check out his channel. But even if you've been a seller for a long time, check out his channel because he is a big money Bolo Finder and that is the Cha-ching King. Okay, so this next item, I sold it for $10.54. The buyer paid tax and shipping on this. This is an item that I always pick up. It's a toy and you guys, I'm turning on the lights. So get ready, lights. I always pick it up. I love toys. I love them because I can usually find them very cheap and I can make decent money on them. So this is what I sold. I think this is the one. Let me make sure, I'll check the date. There is a date on the back. So I do pick a lot of these up. Let me see if there's another baby Jesus in here. Oh, there is. So I have to look and confirm, did I sell this baby Jesus? Or this baby Jesus? I think it might have been this one. So they're different years. So let me check real quick. It was this one. And this one is 2001. So again, put that date in the title. That's gonna help people find your item, okay? Cause these are very similar, but they're a different year and the base is different and they go with a different set. So the person that's looking to complete their set is gonna look for the date, okay? So this one's going out $10 and 54 cents plus tax and shipping on that one. Oh my goodness, I can't. Baby's first Christmas. Look, it's Bambi and it's musical. Walt Disney Company. Oh, it's so cute. Wow, that one is adorable. I love it. That is so cute. We have a Mickey Mouse with bunny ears. So this is an Easter Mickey. This one is Dream International Limited. So I don't know. So far, nothing I've ever sold before. So we will be looking things up. This guy's funny. He has no tush tags. So I will use Google Lens to see. Right this next item, we're going over to tote 11 which I have no idea where that is. Is it over here? It's over here. Okay. It is a Lala Lopsie doll or something. I also got her at the bins. So all of these items are coming from the Goodwill Bins. She's a little bit heavier. So I probably had close to $3 and maybe 50 cents in her. $1.89 a pound. I don't know what she weighs. But I sold her for $21.70 plus shipping. But this one's coming out of tote 27. I'm looking at my totes. Where is tote 27? Ooh, I think it's over here. I have a feeling it's over here. Oh my goodness. You know, I kind of have them in order, but then they get moved around. And anyway, this one I've actually had for quite a while. I've had a lot of interest in it. I feel like I've had offers in the past. But I got it out of a thrift store mystery box. And it is a Hallmark ornament. And it is right here. Oops, I just popped her out. So it's this girl. Again, I think she's from Disney. The hunchback of Notre Dame or Notre Dame. Notre Dame, I think, I don't know, Esmeralda. And I think she sold for $9.92 plus shipping. So she's going out. All right, we're going back over to this tote that is still open. I didn't even put the lid on. And I'm looking for a couple of piggies. Flush piggies, where are they? I'm wondering if they're in Flush tote 11 and not, or 7th, no, that doesn't make sense. Where are they? Oh man, did I mess up. Where's the little piggies at? Love it when this happens. Gotta love it. Okay, I gotta find the pigs. If I can figure it out, does anybody know what kind of dog that is? It's got sparkly eyes. I wonder if it's a T-Y. We know what that is. It's a Dalmatian patch. Disney store exclusive 101 patch, 101 Dalmatians. Looking for a date, don't see anything. He doesn't look too bad really. Sometimes when they're white, they start to yellow, but look at his black ear and his white ear. How cute is that? Cute. So I looked in tote 17 because they sold for 17. They're actually in tote 11. So let's go over to tote 11 and find these pigs. Just in tote 11. It's just here. Here, piggy, piggy, piggy. Where are you guys hiding at? Oh, where are you? I think I found them. Get in there. So we've got the painter and the ballerina. And I believe these came from the Goodwill Benz Outlet as well. So I decided to put these together to maybe get somebody to bite on my item because they were getting two for one and the shipping was together, you know? So I do that sometimes with my plush. I will create a lot. So those are going out. Okay, we are back in this room and we are going to find tote G here.  This would be an example of that. So this one is another item that I pick up all the time and I'm guilty of hoarding it. What I do is I buy it in bulk whenever I see it and I throw it in a tote. Super guilty. This one, this piece in particular, came from a mystery box from Auctions for You. So I listed it and it's sold for $10. But I gotta find it. Why do I feel like it should be in here? All right, I'm gonna look for it with two hands. I'll be back. Okay, I found her. I was working in tote G, but I knew that it was a Fisher Price item and I keep all of my Fisher Price items in the same spot. So I was able to find her pretty quickly because I had put her where she was supposed to go. Unfortunately, I spent a little bit of time looking in tote G. So you guys are probably thinking, she is so unorganized. Every time she does this, she can't find something. I feel like I'm pretty organized. I've got like 7,000 things listed. So I mean, if I find it, that's good, right? We all make mistakes. All right, this is the Petting Zoo. And it looks like the American flag. Um, 1994 on that one. That's pretty cool. Brass button bear. Original tag, 1996. It's got his little tush tag here. His name is Cody. Cody. There he is. So this is craft tote too. It's always a joy. I am looking for, it is called Summer Sun, but in the title, I put Summer Run. And it was just a typo. I think it'll be okay. It has kind of a pink border. I think this might be it. Nope, that's spring roses, but it's a sampler like this. It's called Needles and Hoops. These came from an estate clean out and I've got so many listed. That one's Autumn Harvest. So I just really just have to look through here until I find the one I'm looking for. Here it is, Summer Sun. I think it was originally a big lots, like odd lots sticker. And this sold for, did I say 2470 plus shipping? And it's a sampler. And that's what it looks like. So nice little bolo right there, bread and butter bolos. Some of these can go for huge, big money. And a lot of them are just bread and butter, but 24 bucks and my cost of goods was probably less than a dollar. I'm really, really happy with that sale. Okay, so I told you this little girl came out of a mystery toy box from Auctions for You. I've talked about their Donatella Bottolino in Auctions for You all the time. I talk about them all the time. The next item I'm about to pull is a major bread and butter. It was in, I think it was a mystery toy box. I don't recall exactly which box. If you watch my channel, I have a bunch of unboxings from both of these ladies. Whenever I buy from their YouTube auctions, I do videos. And I show you guys, I unbox it and I show you how I list everything. So, and then you see me talk about them in my what sold videos. But this item was kind of one of those like, I don't know what to do with it. I know it's a bread and butter, but I think I'm gonna list it and just see if it will sell. And this one came out of a mystery box from Donatella Bottolino. Again, it's a mixture of all kinds of items. So you never know what you're gonna get, but I will tell you, if you get a box from them, list everything. Because even if it's bread and butter, it's going to add up, okay?  But if you get something, it's going to add up. Most of the items, you're gonna find lots of value in. You're gonna be like, this is nice. This is awesome. But sometimes there'll be something in there and you'll look it up and it's like, I don't know if I really wanna list this. But here is an example of one of those items that I almost didn't list. And I'm glad I did, because even though I didn't make big money on it, it still is money, okay? And it took me no time at all to list. So let me show you that item. Okay, so if this item would have had clothing, it definitely would have been worth more money. But she didn't have any clothes and that's kind of why I was like, should I list her? Should I not list her? And she is like a LOL, OMG doll, I think they're called. I used Google Lens to figure it out. I've gotta find her here for you guys. And she's not very big. She's a smaller one. And I sold her for $5.50. And this is what she looks like. So her hair is fantastic. And I'm like, you know, somebody might want this. So I went ahead and listed her. $5.50, buyer paid tax and shipping. So I'm gonna make some money on this. The box, the boxes that I get from them, I always make a ton of money. So definitely check out those unboxing videos and you'll see what I mean about listing, you know, some items like I just sold an item out of Miss Auctions for Use box for 80 bucks. And then another item might only go for 10 bucks. But if you pay 100 or 150 for the entire box, all of those smalls and bread and butter to add up, they're gonna add up. So list everything, you know, the more, if you, if you're like, eh, I don't feel like listing this. It's not worth, you know, listing. Then you're not gonna make as much. These items still sell. People are still looking for them. So yeah, $5.50, the buyer paid tax and shipping on that. Okay, this tote used to be like to the top. I've probably had more than one of these totes because I used to buy these and sell these all the time. Some of them I would source directly from Lane Bryant. These are Casique swimsuits. This one is a size 28. It's kind of like a leopard print. Retails for what, $50, $493? I don't know what I have it listed for. So I still have that one. This one here also came from Lane Bryant. So I used to do a lot of retail arbitrage. I would go on to websites and I would source from them. And Lane Bryant was one of them. This one came from Discount Fashion Warehouse. These are items that were discontinued that didn't sell in the store. So they're new old stock and they go to a discount store. They are then marked through like this. And that is so you cannot return them to the original store. I ended up taking a best offer on this little cute skirt for $11 plus tax and shipping. So again, it's just a new old stock item. The thing with these, I bought tons of them.  They sold quicker at first, but I think now they're even more discontinued that the right person has to come along that needs this pattern and this print or has something to match it. So it becomes more of a long tail item at this point and I'm just wanting to move them. So $11 plus tax and shipping. My cost of goods was approximately $3 for that.  So let's do the small ones first. Look at this. This is a happy little sheep. Huggable Irish friends. 2015, Aliyo. I don't know what that means. Never seen that brand. We have a penguin. GP, or GPI. GPI AST, I'm not sure. Peppa, Peppa. Or Peepee, Peppa. I don't know what it is. Peepee, Peepee the penguin. Oh, it's a Kroger. That's interesting. It's a Kroger mascot. This one is cute. It is the Barrington Collection. Little teddy bear. Hey, you ready for the big guy? I may just have to sit my phone down. Oh my goodness, he's huge. All right, here he is. He's a, is he a tiger? He's a tiger. Love's travel pals. No tush tag that I see. He's really soft and he is really big. So there he is. There he is. And you know what? There's one more in here.   He is a prayer buddy. My job has been to sit in church and listen to all scriptures, sermons and prayers. So now that I'm coming to you, I am immersed in the holiness of God for your benefit. Whenever you look at me, touch me or hold me, God's holiness touches you. And you're reminded that people of Chesterville United Methodist Church are lifting you up in their love. My hope is that you will love me since God loves you enough to send me to you that we might love each other. The God I serve bless you. Amen. Cute. So they must have done this as like a little project where they got stuffed animals and then they attached that and they gave them out as little gifts. It's all Greek to me. That's the brand of the dog. I'm guessing probably somebody just donated stuffed animals and then they added those on. Cute idea, right? All right, you guys, that's that bag. So let's talk a little more about retail arbitrage. These are all David's Bridal website dresses. I would not do this again ever. Have I made money? Yes. Have I sold a ton? Yes, but this is what I have left. And I have sold a ton. I got them anywhere from $5, probably up to around $20, $25 a piece. I tried to source them as cheaply as possible, but I just sold one. I sold it, I believe, for $22 plus tax and shipping at this point. I just want to get these out of my house and out of my life. They are Bridesmaid dresses. Now the good thing about it is David's Bridal's, they have colors on the tags. So this is Malibu. So somebody may be having a wedding and they're like, I don't care what dress you get, just get the color Malibu. And that's when my dresses sell because these are all now discontinued styles and they take longer to sell. These can also be worn for like prom and homecoming and stuff like that. But whenever I sell one, I'm super happy. The bad thing about these types of dresses, the return rate is kind of high because people don't check measurements. And a lot of these dresses, the sizes run a little wonky. They either run a little small or a little big or they don't fit in the bust or they're too tight in the waist. There's always something. But I have sold a ton of them. I have definitely made my money back plus money. So I kind of feel like from here on out anything that I sell, it's all profit from here. But again, look how much space these are taking up. Like I just want them gone. So anyway, I sold one, super happy about that. Hopefully it fits and they don't return it. This one right here is plum and it is the crumb catcher style dress. So all new with tags, original tags. I mean, they're great dresses, but man, just talk about long tail. And again, this was like a retail arbitrage thing that I did just like the bra, I did brawls and I also did swimwear on Casique Lane Bryant. So it is possible to source retail arbitrage online. I used to do it all the time, but I would not recommend David's Bridal. It's tough, it's tough. This next item, I picked up at a garage sale for two bucks. I just thought they were super cute and I sold them for $15 plus tax and shipping. And we're gonna head over to the tote and see if we can find them quickly. Hey, they're in tote 26, which makes sense because I have a lot of Christmas ornaments in here. That's not it. I may or may not have bubble wrap them. So we're just gonna look. It's all we can do, right? That's all we can do. Would you rather me cut the looking part out of the video? Or do you like to hear me ruffle through stuff and look for my stuff? And does it make you feel better about how long it takes you to find your stuff sometimes? Where is it? They might be in one of these bags. I'm gonna take a peek. Found them. I'll have to pop up a screenshot because I already have them bubble wrap. I will add some more bubble wrap, but they are like dough kids or something like that. I think somebody probably made them out of dough. I've been looking at this little lady for days and she popped out of one of the totes. So I just left her out because I wasn't sure which one she went in. And somebody's been showing some interest on Mercari and I've just, I missed Mercari alerts. I will say Mercari has a tough time with alerts, but I wanna show you this. She is Disney store. So usually you can find the tag on the inside of the leg. And I sold her for $20 free shipping on Mercari. And I'll just put her in a polymailer, probably fold her in half like this, polymail her up and get her shipped out first class. And I do pick up most Disney store plush because they have a tendency to do well. Some of them are long tail, but some of them sell quickly. And the Disney princesses, somebody is usually looking for that. All right, you guys, if you're not on Mercari, you can join with my link down below and you will get $10 off your first purchase. Now, I personally love shopping on Mercari. Sometimes I go to Mercari before I go to eBay.  I feel like it's a lot of hobby sellers and sometimes you can find a really good deal over there and it's sometimes easier to just search on Mercari. So even if you don't wanna sell on Mercari, definitely consider clicking on that link. It's a referral link, so you will get $10 free to shop. And now it's the perfect time to buy, right? Anytime is the perfect time to buy. You can maybe even source something to buy and then resell. But if you wanna sell on Mercari, once you sell $100 worth of stuff, you will get another $20 to shop. So that's $30 total you'll get from Mercari if you use my link down below. So definitely check that out.  I am looking for Y6 and I think it was B5. So a state buyout this summer and they had all of this yarn and I still have a ton to list. But my friend came over, the girl, her name's Erin. She went to the Goodwill Benz with me and she's in a couple of my videos but we like sorted this yarn and packed it up. And I just sold two of them on a bundle on Poshmark and I will pop it up over here so you can see what it looks like because I've already got these bagged and ready to ship. Okay, so I'm looking for B5 and Y6. Now, you can see right here, it's kind of a mess. I just have to find it, right? Inventory B5, there it is, B5. Now we're looking for Y6. Y6. So I know that these two are the ones that are going out. Again, that is a Poshmark sale. I took an offer, I believe, let me check. I took an offer on those for $24 for, I don't know how many it's an inch. It looks like five and one and maybe six and the other. I'll pop up screenshots for you guys to see. But I only put the yarn on Poshmark because I had all the pictures. Sometimes when I'm laying in bed and I just want to list quick and easy, Poshmark is really great. And yarn can, it's kind of heavy. So you've got that cheaper shipping over on Poshmark, especially when people bundle. So this will be under five pounds and it should ship for whatever, like the almost $8 price range, price point. And if you're not on Poshmark and you're thinking about selling on Poshmark, I do have a link down below. It's a referral link. So you will get $10 to shop when you use that link. And then when you buy something, they will give me $10 to shop. So I appreciate it when you use my referral links down below that really helps me out and I appreciate it. So we're gonna get that yarn shipped out and wait until you see the next item that's sold on Poshmark. And this is another item I think I only put on Poshmark. I felt like there wasn't gonna be really a market on eBay for this item. And I also felt like on Poshmark, things get pushed through the feed, like when you share and somebody might see this and think it's cute and maybe bundle it with something else. Well, they didn't bundle it with something else and I'm actually shocked that it sold because they paid quite a bit for the shipping because this is a first class item. And that's the one thing about Poshmark is even if the item is first class, it still ships out priority mail. Like if it's first class wait, it'll still ship out in a priority mail package. So they're still paying like almost $8 for shipping. So before with the yarn, it's like they're only paying almost $8 in shipping with the first class item. I'm like, they're paying almost $8 in shipping for this item. So it's kind of funny. That's what's great is like, you just gotta pick out which platform works best for you. And that's why I say you should cross post. So I started on eBay, I cross-posed to McCarty and Poshmark. I use this perfectly. There is a video down in the description that shows you how I use this perfectly to cross post. And if you watch it and you think that it's a good fit for you, you can get 30% off your first month with my referral code, Bolo Buddies, all one word, when you buy your subscription. And there's three different plans right now. There's the basic, the business and the pro, and I talk about all of those in that video. So you can check that out if you're interested in getting on multiple platforms. All right, so this is a major bread and butter item. Like I said, I cannot believe that this even sold on Poshmark because they paid $7 for it, but they also paid all the shipping costs. Let's see if I can find it here. It's a towel, it's a bath towel. I mean, it's a cute bath towel, but here it is. It's a snowman. And it looks like that. It's just a white towel with, it kind of looks like Olaf. But yeah, they paid $7 plus shipping. Now, I did send them an offer with discounted shipping. So they didn't pay the full amount of shipping, but I'm only gonna make like $2.33 on this, but this came out of a thrift store mystery box. So I have very little invested in this and a sale is a sale, right? So I wanted to share that with you guys. Thank you guys so much for being here. Thank you for watching. Be sure to like, comment, subscribe and share. Let me know which item was your favorite. Let me know if you're a fan of the reseller vlogs or if you prefer the screen sharing. I'm gonna continue to do both but would love your feedback. Thanks so much for watching. Just taken and sometimes it, let's try that again. Ah, ah, ah.


I pledged to finally get my leaky toilet fixed because I'm tired of flushing money down the drain. I pledged to turn off the tap because water is a precious resource we should not waste. We pledged to only wash full loads of dishes and laundry to avoid wasting water and time. I pledged to use energy efficient appliances because energy production is one of the largest water users and water polluters. I pledged to power down at night because when I save energy I save water and I also save money too. Join us and make your pledge for the National Mayor's Challenge Water Conservation by visiting www.mywaterpledge.com. Let's show our friends across the country that in Columbia, South Carolina, we stand together in our commitment to clean water.


Hi, hello, and welcome, everyone. This will be a talk about increasing GPU utilization in Kubernetes clusters. And I wanted to start from thanking you for still being here after this exhausting week. I hope you are not as tired as I am, and will stay with me through the talk. So my name is Maciek Mazur. I'm the principal AI engineer in Canonical. That's the company behind Ubuntu. And I'm typically working on the other side of the fence. I know that you guys are mostly business admins, DevOps engineers, and people working on Kubernetes clusters. I'm the user of the great stuff that you guys build and do. And I wanted to tell you a little bit about how we are actually using it, what are our goals, and how we built a couple of projects last year with infrastructures, which varied from 7,000 to 12,000 GPUs in one project setup or a cluster. So why we are even interested in Kubernetes for AIML from the ML engineer perspective. Like PowerLife is typically superstructured. There is a process that we are following called MLOps. And basically, there are a lot of small steps that needs to happen using various types of compute. And from our experience, we saw that Kubernetes gives us a lot of repeatability, gives us all the pipelines, portability, scaling. So all the standard stuff that you guys realized already 10 years ago, it took us a little bit more time, but we now started using it properly as well, I think. But why this talk is about GPUs, and why we do even need them in the cluster. So the actual things that we are doing is mostly matrix multiplication. A lot of things will show ML as something very complicated, but mathematically, these are sort of a standard computations. And why GPUs? Because it's very similar to this. This would be your screen with RGB, and it's also a matrix. So all the silicon being built by NVIDIA, by AMD, by Intel is basically the tool that was used to do a lot of similar computations. And that's why in machine learning, GPUs become super popular. And that's why we are using it very heavily during our processes. So typically, what you would do in any Kubernetes cluster, you need your GPU operator, something that will enable the device to be visible. If the cluster is bigger, you would also go with a network operator to connect the GPUs together. So that's like a fairly standard example of a stack from NVIDIA. But OK, so we want to use GPUs, which ones? And is there really any difference? And with GPUs, you can go with the setup from NVIDIA. That's the most popular one. That's something that most of the people are using, is how the clusters are being built these days. But you also have contenders. You have AMD with the ROKM kits and basically also working a lot on the optimization of the frameworks. The same goes for Intel with OpenVINO and all the extensions to TensorFlow and PyTorch like IPEX and IPEX. So these are multiple options. But what we are, as users, mostly looking into is actually the pure statistics and power. So if you look at the set of GPUs from NVIDIA, there will be obviously some performance metrics and benchmarks being made by people. But a very important metric from my perspective as a user and a solution architect is this, basically performance per the price of the GPU. And if you look at this list, this does not look exactly the same. So there is a lot of caveats in terms of which GPUs you are actually choosing to the computation that you want to do and how the mix should look like. So the price performance is a very important aspect, especially that the most powerful GPUs, obviously it's a new technology. It needs to be priced a little higher because of the development cost. So that's something that we are looking into very much. And how we are actually using GPUs, there are a couple of main categories. So we have the training environment where we are basically taking the data, training the models, perform the fine tuning processes. And we have inference environments. And on the training side, what we are doing, we are starting our work with a notebook, typically, or like a VS Code or IntelliJ hooked up to a notebook server. We have our ML of spy plans on Kubernetes with Kubeflow. We are doing some processing tasks which are also GPU accelerated if you go with Spark and Rapids. And we are testing a lot on this type of environments. But we also need to do inference where we are actually serving the models to end users that's completely different requirements in terms of computational speed. We are doing also a lot of serving in the real time, especially if you do like IoT devices or analysis of signals from a telecom network, a lot of pre-processing and also GPUs as additional hardware type. But what is super important is sharing that infrastructure among multiple teams and multiple users because GPUs are that expensive. Price performance is important. We need to figure out how we are actually fitting the workload into the GPU in a proper way. And there are multiple ways how you can do it. You can time slice the GPU. So basically give everyone a time slot when they are using it. But the most popular way, especially in big clusters would be MIG or any other kind of GPU slicing. So basically, if you have 40 gig or 80 gig GPU, you slice it into smaller pieces to be able to efficiently place the workloads which does not require the full GPU. Like if you have a 40 gigs on the GPU and your training process requires one gig because it's something super simple. Reserving the whole GPU is very counterproductive. And if we have more GPUs, we need to start looking into networking. And this is typically a big bottleneck because GPUs are very fast by themselves. If they are in one server, they also have ways to link them in a fast manner. But if you have multiple data centers, especially in different locations or even different availability zones or fire compartments, you need fast networking. There are a lot of technologies between RDMA overconverged internet, InfiniBunt or presently SpectrumX that will allow you to basically have the networking that is fast enough to pull those GPUs together in bigger training processes. But what it leads to, it increases the deployment complexity because we need to have a network for all of our data and everything that we do normally. But we also need a dedicated network for any GPU related tasks and processing. So there are a couple of options. InfiniBunt and SpectrumX are coming from NVIDIA. ROC can be used in basically with any type of hardware. And these are the choices that you would look into in the networking space. So when you have those GPUs connected already, you have a big pool of them, then the next question is how to schedule the resources. And from our perspective as users, the ML job would look like in a way that you do some CPU-based computations, then some GPU-based stuff, then a bit of waiting time, idle process fetch the data from the disk and then a rinse and repeat that type of process. So basically what you would have is a queue of things that need to run on top of your hardware. So the problem is that if the queue, there's something that you want to put in a queue exceeds the available GPU space, that's when you would start having problems and needing to basically do more resource-aware scheduling. And if you have a bigger setup, the problem is even bigger because on one single GPU, it's easy to manage. You can use stains, tolerations, and basically describe your pods in a way that it makes sense. But if you have a full data center or multiple racks like this, this problem gets more and more complex.  That would be Kubeflow in that case because it gives you a nice way to generate pipelines, notebooks, the QR code will bring you to the way, how to install it in an easy way. And that's like the front-end part. But what's actually important and what the stock is about is what happens behind Kubeflow and what are the things that are enabling, using it in very big infrastructures. So the first project that we are utilizing very heavily is Paddle-Paddle. That's basically something that allows you to split the bigger task or a training job into smaller pieces because obviously in any scheduling, if you want to fit stuff in the queue, if they are smaller, that's easier, right? Like queuing 70 gigs of a workload on 80 gigs GPU, you don't have a lot of flexibility. You have a lot of small jobs which require five gigs of GPU memory. It's much easier to distribute them. And then the scheduler, the standard Kubernetes scheduler is not really great with various types of hardware. So what we are using typically on those bigger infrastructures would be volcano. That's a really great project that allows you to perform gang scheduling and more complicated assignments of things. So if you have a job that's actually comprised of multiple tasks because you manage to split it with Paddle-Paddle, then the next thing that you will be doing is basically calculating how much resources does the job need so that you can do a proper workload placement. So if you see that, okay, I need 20 gigs of GPU memory. I need five CPU cores and that amount of RAM. These are my options and that's how I can place the workload. So volcano works by itself on a single Kubernetes cluster as a scheduler really nicely, but in reality, whenever you have a bigger training environment and you need to run on top of multiple case clusters because obviously a single Kubernetes will not be really able to serve like 12,000 GPUs because you calculate it to nodes, four GPUs per node. It's still too big infrastructure to run it in one cluster. So you will have different ways of how to actually split it in terms of the resources and pulling it. And the actual scheduling is pretty much something that looks like this on the picture here and it allows you for an every job to either schedule it but you can also preempt some jobs or remove them from the queue or actually even evict the pods if the higher priority thing comes in. So if you have multiple teams, you can allocate priorities between them and define which job would take a precedent, for example, if you have an overflow of a queue or other problems like this. If you have more Kubernetes clusters, the Armada project is really great if you want to have one single point of entry and then distributed across multiple case clusters with its own schedulers. So with this long intro of the technologies that we use, let me bring you to the actual part of the real-world examples and the projects that I was working on before coming here and over the last six to nine months. So basically there was one common denominator in all of them. So public sector-related big clouds that give compute capacity to multiple tenants. So the size of the infrastructure was between 10 to 15 K GPUs which means around 3,000 nodes and typically something around a few thousand of users split it into various teams. So that's basically the main problem sharing the resources in an efficient way. They do everything on those clusters. It's not only running training and inference but also stores the data. And typically if it's like a sovereign cloud environment, people are looking into open source projects and want to build it fully full stack end-to-end open source. And there are a lot of strict requirements about security and SLAs. And the setup that's one of the projects was using was a couple different open source projects as databases, the new additions to the standard thing that you would see here would be the vector DB and what we saw that works really well especially with like internal LLMs with racks and trying to get like internal data and knowledge about it into the LLM would be open search or postgres with PG vector extension. These are the two most dominant things that we saw working well in those environments and the rest is like a fairly standard MLOps tool chain. We also work with them on like a full guided journey started with architecting that environment put into production and now expanding it to multiple nodes. But what are the learnings? So first thing that we learned is to listen to various people. The biggest pitfalls that we got were actually not coming from the data scientists or from the end users of the models but from places like compliance department, legal people and a lot of other problems in terms of how stuff is being shared and it's important in terms of the GPU sharing because some of the methods of how you can allocate the GPU to a workload will allow memory leakage. And obviously if you are running workloads in the golf space there are strict requirements about who can see what and what type of data so you need to really watch out especially if you slice GPU into smaller pieces so that you don't have any leakage between the workloads. The hardware choice that was really well working for us is a mixture of different things. So first of all, the bigger portion of things around 70% of the cluster would be something big, H100, H200, powerful GPUs that has a lot of memory. We can slice it into mixed slices and basically run the big jobs, the small jobs like very flexible here. The 20% would be mid-range GPUs that will allow us to basically make scheduling more efficient.  And also around 10% you will have the low-end GPU, something like L40, which is best in terms of price performance like dollars per flops. And it's for inference, it's for testing and also for putting even the smallest pieces of the job into the GPU that you don't require that kind of level of speed and that fast connectivity. Another thing is that actually people still use VMs, which is really bad, but in that particular space if you want to put them in the mix, but nevertheless, they're still important. They are utilizing them first for the performance on the databases and also a lot of legacy workloads that they use to pull the data from. Like especially in the federal sector, we have seen tons of people still using things like SAP from which they need to pull their data legacy deployments and we are solving this problem using the open source projects FlexD and basically on the same cluster with the same resources, but defining resource pools with metal as a service and flexibly scaling the amounts of things that are still running on VMs and shrinking it down every month whenever they start migrating to something newer. One additional thing is that whenever you design this type of architecture, you need to look to have multiple storage types. Obviously, standard object store S3, that's cool, but you also need some additional local storage and one thing that typically people forget about is a big file storage. There are a lot of places where either the LLM model itself is a file that will exceed your file system capacity or you have big files like from oil and gas industry, field maps might be something that will not fit on a standard storage. So that's another thing that can bite you. The scheduling complexity, so we tried various different schedulers, ways of sharing the GPUs and the combination of MIG, volcano and Armada was the best thing that allowed us to actually increase the GPU utilization and we are typically using Kubeflow as the UI part to control it because those projects by themselves and MIG as a technology is super complicated and it's not that easy to use by the end user. So that's the setup that works best for us over course of those projects. And networking as the main bottleneck, if you have an infra with any new GPUs, you have to go with Spectrum X, that's the only thing that will give you enough performance and enough scalability in those clusters in order to go with H100, H200 level. And especially with the recently few days ago announced Blackwell chip, that will be the networking that you will use. So if you want your cluster ready to be expanded by that, that's super important. Make sure that you have three dedicated networks, one for the GPU traffic, which will be one of those technologies that I mentioned. A dedicated storage network because that also is sometimes a huge bottleneck whenever your fractional piece of the job needs to fetch additional data from your S3 or any other place. And also whatever automation tooling you are using, it needs to be aware of the top of the rack switch and be able to reconfigure it. Actually, during the scheduling, when we have jobs that are doing rebalancing and changing the amount of nodes in each cluster or rules for scheduling, you will in a lot of cases end up in a moment where you have to reconfigure your networking. And if the automation tooling that you are using is not able to see it, that will be super difficult and will require a lot of manual stuff and hacks. So typical deployment in terms of hardware that is under the bare metal Kubernetes that you would want to run would split more or less like this. So on the left-hand side, you see the management rack. This will be no GPU compute where you have your control plane, you will have your CPU-based compute for stuff like UIs, your Olama Web UI or whatever project you are using to basically visualize the LLM to people, your nodes that are controlling the vector database, doing some data ingestion stuff and so on. On the right-hand side, you have your GPU nodes and storage nodes. From the networking perspective, it's really good to place them in the same rack. And then you can see here two different mentions of observability stack. That's also something that we learned the hardware not to mix those two. So the observability for the ML pipelines and the tools running there versus the observability for the underlying hardware. It was much better for us to separate those. And then if you look at the tooling perspective, so having those nodes and those racks placed like on the previous screen, you put Metal SS Service to control them and add them to the plane. And then you can run a Kubernetes cluster with Volcano GPU Network Operator MIG configured and your tools on top of those case clusters. But what's nice, if you have any cloud management platform on top and multiple of those clusters shared across multiple teams, you have your job queues and then the resource pools that are in mass allow you the flexibility. So whenever your observability system sees that team A is typically utilizing additional resources, they schedule a lot of job on an idle resources that are on the cluster of team B, you can rebalance it. You can add nodes from the resource pool B to resource pool A and put them in one Kubernetes cluster. So basically that's an exercise that we are doing weekly over the weekends to rebalance all the resources and keep them confined into each and one cluster because whatever you do on the initial stage, on the workshop asking the customer for requirements, yes, this team A needs 200 GPUs, team B needs 400. That's an assumption and typically in reality it changes a lot, even like completely opposite. And the whole observability and evolution while super important, we are monitoring the utilization of the GPUs and also of the network and all the other components of the system and then do a full reconfiguration. So besides the resource pools, we are also changing the priority access between the teams in terms of the priority on the scheduling queue, changing the slicing of the GPUs because if we see that all the jobs that we run are utilizing at least two or at least four slices, it means that it doesn't make sense to slice this in that granular way and lose some performance on that. And also there is a lot of optimizations that you can do on the framework level itself try it on KSER for however you handle the request. That's also additional place where you can look into. And if you want to ask some questions about those projects or you have something similar, this is the way how you can reach me and contact me and I'm happy to talk more with you after this. Thank you for listening to that.


Well I wish you could test out your career before launching into the field well that's exactly what manufacturing students at Brainerd High School got the opportunity to do today. Sarah Winkelman has more. The high school, college and local industry workers have all come together through bridges to give students a glimpse into the manufacturing process. They live in a community up in northern Minnesota that's a nice great community but there's so much more out there that they really don't know even exists. Local manufacturers gave demonstrations in CNC machining the computer behind the machine tools. The Haas CNC mill is a smaller version of a manufacturing tool giving students experience into the real world of manufacturing. I find it very interesting like how you can make you can start from nothing and create some a big difference in the world or for somebody else. However today's project didn't involve making any medical or industrial equipment instead it was something made just for fun. I thought the machine was actually going to be a little bit bigger but it is yeah it was really fun to come and watch it work and make these fun little yoyos. For many of the students in the room today this was an opportunity to explore a possible career choice.  A future to college and beyond right here in the Brainerd Lakes area. So rewarding and it makes me so hopeful for our future because I believe our future is bright. We have young talented people. I think manufacturing presents so many opportunities here. Opportunities that spark those with interests of math, computer skills and problem solving. I'm really glad they actually had this happen here because it was quite a bit of a bit of like a showing of what actually goes on in like the fields today when it comes to engineering and manufacturing. I feel very hopeful and confident that we're going to have a great workforce. Students learn how much design, planning and creativity goes into making just one small part. Reporting in Brainerd Sarah Winkleman, Lakeland News. The Haas ANC Mill will travel to eight other local schools this year. If you've enjoyed this segment of Lakeland News please consider making a tax deductible contribution to Lakeland Public Television.