Hey everybody, today we have a jerry on the channel. So jerry has been following me for a long time and then basically came into the program stuff. Right. And I just want to say something about jerry. It's like the difference between the people who are successful and people who are not. And I see this pattern like very, very evidently is the people who are successful. It's like, you have the mentality where you're going to make it work.  That's all.  Okay. And jerry's like, when I met him, he's like super hardcore. And I know it's like, Hey, this guy, if he comes in that he just does the work you tell him what to do. He's going to do it. And that's why he's successful.   Alright, so my name is jerry. So it's been about 22 and a half days since we launched. And if we include distributorship we already we've crossed six figures really. So how I got started is for myself, I'm doing medical device I create braces for kids with scoliosis. I have this idea of having a recovery gear that anybody could use that's so easy and simple, but that was four or five years back. It was on the shelf for the longest time then when the global pandemic hit. My wife told me to, you know, bring that project back and see the market will react to it. So what I did was I did a round of Kickstarter, and we were successfully funded for 100%. In fact, we're funded over $4 more. Right. So for the pre order we were four to five months waiting time we raise another $4,000. Then, then after we were, I was, when I say this is just me. All the five months of loud period. During this time preparing the shipping the production and all. So when it's time to commercialize which was back in 25th of April. It's kind of month right from here. It's, you know, I was thinking, hey, if there is a blueprint that I could follow to accelerate my path to a stable and sustainable business. Right. And success is a byproduct of this. It was like, you know, the many, many student interviews that John has conducted, which is they found John to be a no nonsense straight up in your face results driven performance proven with a series of track record right so so I reach out to him. The first time we spoke, he kind of managed my expectation by saying he too doesn't have the confidence in making this work. So, so it was. But that was back the last year late last year, then coming to this year, it's. I saw his program was available. In fact, I've been stalking him on his Facebook group, his Facebook page on his LinkedIn on his website, and on his YouTube videos. What I did before coming on to this program is I would post every frame by frame, just to replicate the exact flow of the content that he's producing in terms of Facebook ads which is the sexy part right and all the way down to the landing page and the final structure. So, so I, so what we saw was a very positive result. Now granted, I, I'm just in short, an engineer who create products with my hands right so marketing wasn't my, my forte, but I managed to essentially plug and play his, his formula just by visual. So, so when his program comes in and say I give you everything. So it does not give you any reason not to be successful right. I, I waited and I wanted to sign up. Don't get me wrong, the price was intensive right it was holy moly. So, so when he finally offered an installment plan. I jumped, I followed. It's good. It's good. Yeah, I executed it. So I bought the Kool-Aid. And you know, whatever he say right in terms of like, okay, so the more important part I think you what you viewers are understanding in probably his face. So I had no fear going in knowing that this is a plan right so in my mind is that the cost of running a business or the startup cost, it's John's program right plus all the other apps and all. If that fail, I lose, you know, like what 10 or 15 K, but if I gain, I have acquired a skill, and also I have basically create a business and a brand right. So, so, oh, luckily it's option two. Yeah, definitely I'll be showing you the dashboard, you know, giving you an insider view of the numbers that we have actually pushed out. I think said that like what we realized was, I am the bottleneck of the business. So which is I'm doing the everything. Everything you can take off. It's done and I think moving towards the interview, I will tell you like, you know, the apps and the downs, but overall, how we should see it. It's part of the game right it's business as usual. It's firefighting all the time. So I think with more hands on that, you're able to seriously like bulletproof your your business moving forward or moving into words Q4. So that's the thing right.  So you're literally spending like, have you ever spent like five K a day. A high value asset right so you're spending so much but you're so confident that you just wish you could find a way to spend even more. That is something that is one of the greatest takeaway I think from this video of the other students video so you know let me just jump to dashboard to show you. Okay, so let me give you permission already.  I think he went out he's on pace actually to do 100k per month. I think now is like 5060 K per month right now. But yeah, Facebook ads, you know, we discussed this too. Okay, I just that I think PayPal held him up. So you can see for yourself. It's like super super aggressive. It's like zero to like 5k per day in the matter of what I think two weeks or so. Am I right. I think you go into analytics. So today is the thing right. Yesterday I was on the phone with John so a little bit background right is the phone we try to discuss like okay what are the current problems and we should slow down the the expand. So, so by single low and expand of course the money revenue for the first time. But I think it is best for the long term of the business right so but when I did that. Oh my God the sales are still coming in. Okay, that's good. That's good. Okay, good but there's also a bet so so don't get me wrong. I don't sell humble pie right so this is an indication that I should continue wrapping up my spend. I guess the sales are coming in anyway so you know, let me just hit it up. Right. Okay, let's go. So as you can see, let's see this April 1st to 24th. You see this. So so you launched, which day. I launched on the 25th. So I showed them like three before like literally. Yeah. So, okay, wait, this it. Yeah, it's literally zero. So let's go. So, the info all the way to today is the 17. Oh, no, no. Today is the May 19. You need to press from the start. So we are at 4745,000. Right. Yeah, so one thing that's there are different ways to further optimize this, including email marketing, including SMS marketing, including launching more creative right. So right now I only have one creative for my code traffic. But then I have five different copies. At all my five different copies to be honest, it's a John sampling that I created to my own. So when I say plug and play, it's literally guys don't have to worry.  Just play like you're playing on play.  You know, so okay, so what can I show you. I guess. Go ahead. So average other value is at 9692. Wait, wait, wait, give me a second. I need to do this right. So I've seen a lot of the videos which is let me refresh a screen to show you that there's no gimmicks behind the thing. It's my first interview right. I need to show you, but hey, this is real man how it's just yeah, people I know people are skeptical so Okay, so here's the thing right. So yeah, 9692 on average. So yeah, then you multiply by 495 so you probably get this number over here on my stock conversion rate is at 2.1. So, as you can see from the start we launched on the 25th. This, when we started launching is I engaged an external media buyer who whom I thought that has the necessary experience, given that, you know, his role as was like by on on the decent at spend so but what I did not consider was that he does not have the relevant experience in the countries. Right, so, so the testing was we did have a 3x RoS but our 3x RoS was the inverse right so we spend about 850 to get 244 This is not a business, it's worse than being a nonprofit organization if you know what I mean. Yeah, don't need the money but I'm literally giving away money to Facebook for people to buy my product. Yeah. What they don't want change right so what yeah, go ahead sorry. So right from the get go when he test his first campaign, I look at the way he structure it's, you know, it's just he doesn't do dynamic creatives right to optimize what what ads are being delivered and at the same time he's testing. Six different assets, but three assets and three assets are the same is just each asset individual copy. Okay. So, you know, it's, it's a very slow way of testing and yeah. To me, it's I miss money. This is money. Yeah, exactly. And he's testing like it was $65 per asset per day. Whoo. Yeah, yeah. We need to give about three days at least three to seven days so I calculated right. Yeah, 65 times. Yeah, correct. That is why most people are so afraid of working with media agencies because like they spend a certain amount of money and they're not getting results but if I can totally relate right so if you never met someone like john who put his money where his money is, you're probably going to have someone burn through like, so think about it right two and a half days 815 so let's round it up to 800 two and a half days one week. You multiply by 2.5 times, I would probably burn through close to 1.8 K to get back to this. I'll be like, turn off completely and not want to touch ads. But I think if you, you met a coach who can show you the correct way, a more suitable way, a more appropriate way to test marketing channels to and and your spend right. I think you are in very safe hands right and in this pair of hands is john right so yeah. I don't have to talk so highly about me this is for you man this story is for you. I still haven't talked highly about you right I can give you the flowery words and all but no this is a serious thing right right the number speaks for itself. So a few things that you want to consider apart from this this graph, I love this graph is just so sexy looking. But more importantly is the average other value that John highly emphasized so to combat rising cost acquisition right and of course the online store conversion rate that he also emphasized which is this is this is one of the things that you can manage right. So, to make your store profitable. So this is the thing I want to move into a way, do you need a day by day sales so let's let's go over the Let's go over the. No, no, no, don't do it. Don't do it. I just want to know, like at the high, right? For example, did you hit 6K yet, right? All day? No, I got a feeling like this week I could have ramp it up to six and push all the way to seven, honestly, honestly. Last week's call is, I haven't touched 3K, right? You were asking me, no, no, no, not last week. The first one. One of the Saturdays you texted me, like I think the previous Saturday, it's that, hey, Jerry, have you consistently maintained that 3 to 4K? Yeah, yeah, yeah. Because that's my next level, because I'm always hovering around 2.5K plus minus 200. After that conversation and everything just falls in place naturally, like I start getting more purchases, so I'm able to create more lookalike audiences. And I'm able to do middle of the funnel, bottom of the funnel. All this, if you do not know, by the goddamn cost. So the thing is, all this fall into place very nicely, right? So instead of like, have I hit 3 to 4K? Boom, I went up to like 4.7K. And I could have maintained beyond 4K, but earlier this week, we started scaling down our ad spend. So we do not have new traffic coming into our store. This is the, for instance, 13th of May, what's that? 2K, OK. 12th of May, I'm going backwards, but 2.7K. So you get an idea of moving into Friday. The more matters are coming in, just push, push, push. Yeah, it's 2.8K. So I have not crossed a 3K yet. Now come Saturday, we go fall. Yep. All right. Oh, bad days in mind. As we go fall, our ROAS across the blended ROAS started improving as well. So wow, it was a very, everything was good, right? Then we hit 4.7K. Oh yeah, man. It's where I told my wife, dude. Yeah, it's a business now. Going back to working. Yeah, but my wife brought down my expectation, right? Yeah, yeah, yeah. Again, 4.6K, very good. Yeah, you see consistency, then you are confident in the campaign, confident in the performance, et cetera. So every Tuesday, John has a call for Australia, Asia, or Southeast Asia. So we had that call. AdSpan ran down 3.2K. So I was like, I had this thought.  I'm spending less, but it seems like I cannot come down below 2K, but I say this with the greatest intention. Because, yeah, John, understand my situation and why we need to bring down that spam. Yeah, it's just PayPal, yeah. Yeah, exactly. So today, it looks like we're going to cross three by today's close. And we are at a 2.7% conversion rate. Oh my God. So yeah, everything looks fine. Everything was good. Yeah, everything looks good, but we had some problems and issues that we need to address. So moving into the scaling part, then we have absolutely no fear, right? We can focus on new problems, because each chapter has its new fear.  So PayPal hole about 8,000 US for the past two weeks. And there's an upcoming about 8,000 US that is still waiting for us. It's like 16,000 US, you know? But then again, it's cash flow, right? So it's part of the business. So we have to be able to do that. So one of the ways to combat is to find multiple different merchants like you've seen on other YouTube videos, right? But more importantly is to understand that the business is just fashion in this way, right? So you have to see it as a way of what you could do to work closely with PayPal, with Stripe, with Shopify. Instead of, hey, man, I'm going to find other things. Granted, you have to find other things, right? Not a problem, but your mindset from the beginning has to be like, I understand this. I accept it like I'm reading this. Just move on, right?   Completing is not a strategy, basically. John has drilled this countless of times, right? Not personally into my head, but throughout his whole video, it is just put your head down. Don't talk, get the work done. The results will come, you know? Yeah, if you're spending way too much time watching, particularly, you know, e-com drop-ship videos, and if you're not implementing or executing it, right? It's essentially wasting your time, right? So at the end of the day, watching videos can mean one of two things. One, you're making money for yourself. Two, you're helping others to make money, right? So I think it's to spend your time wisely, right? So every second comes, right? And for my case, I have a day job and I have a kid, right? 18 months. So in the evening, my time is dedicated to him.  That thing can provide for your kids. You probably don't have a kid yet. Yeah. Yeah, yeah. Yeah, that's true, yeah. Because once you miss his growing face, you, no matter how much money you have, you can't reverse that chapter in life, right? So of course, you want to spend time with him. You want to impart imbalance. At the same time, you have a business. So he, I think like any business owner who has like far-fetched dreams or fairy tale-like stories, go and have a kid so you can be more practical and realistic, right? Don't worry about launching off ads at home.  So don't worry about that, right? Just go and have a goddamn kid. Oh, yeah, I love my kid, right? So okay, I want to show you like the conversion matrix. So we can see, I think this is very important in part. I think the sales figures not so important for my case, right? Because everybody is doing like what? Six figure, yeah, 500. Because like the norm is like, if you're not doing six figures, I don't know why you're doing it, honestly. Okay, so you see from the start, right? 26, okay, so we actually started on the 26. So what is about, yeah, two days, right? One percent, two per one percent, one percent, right? So you see, we were like hovering. So as you can tell, one, one, one, these are abnormalities right here. So it's non-consistent, right? So this is just pure luck, like seeing whatever sticks to the wall. Since we're still one percent, I changed something and I forgot what I changed because it's fifth May and I probably can't remember what I had for lunch yesterday as well. We came in, wow, this is good, man, right? Right? So this, look, look, wait, six of May. Wait, six of May is a Thursday, right? Six of May is a Thursday, okay, I'll show you. So I had this call with John, like I thought I'm moving in. Oh my God, I'm highly confident it's gonna work. So I start adding a bunch of stuff and make it. Oh yeah, I remember. Let me tell you about the result. 3.2.1, whoa, that's a 30% decrease. 2.1, whoa, that's a 50% decrease. That, whoa, that's another 40% decrease. So from here, four days, look, 3.1 to 0.9. We are experiencing at least a 200% decrease in conversion rate. Okay, so I basically told you, like, stop it, right? You were like doing too much, right? You just completely, like, sorry, thought you were smarter than the market. Am I correct? No, I didn't thought I was smarter than the market but I thought like, in my mind was long-form copy works, right? So I literally go full-flagged, long-ass form. The Sparrow Johnson essays on risky. Show into their face, you know? And it doesn't work, right? And it costs me money to understand like it doesn't work. So John's advice is if it's working, please don't touch it. And instead, you should be doing the reverse, right? Don't touch it, increase the ad spend and slide into the weekend. Yeah, so yeah, so that's what I did. So luckily, for my case, like, I do remember things quite well. So I reverted back from here, we were back 100% increase drop. Then I understood something was wrong at the cut to check out series, all right? So I tweaked that part. So now we are very consistently 2.32, 2.3, 2.2, 2.2, 2.2, 2.2. So we know from here, it's profitable already and this is stable already. It's no longer an anomaly, it's a pattern really down here. So I think that's really decent. But in summary, look, look in summary, even though I have shitty performance and I have itchy fingers, overall my conversion rate is 2.1 months. So my campaign overall, despite all my mistakes, it's slightly profitable, slightly profitable. But the good thing is that it's the first month, right? So I haven't actually gone through like a very aggressive skill yet. I'm spending a small amount of money to learn mistakes, which I could have avoided if I just follow the Oxygen Media Program, which is John's program, right? But I did not, I choose to deviate and... Ta-da! Yeah, what is the, like a lot of the times people, if you think about it like extended business, you open a restaurant, like you're 30K off the bat, like negative in the rate, but then somehow like on edge, people cannot stomach, you're spending money to acquire data, right? So that's a very different mindset change per se. How did you become comfortable with that after you understood, like, hey, it's just numbers, right? Why are you so emotional about the, yeah. That's the main sense. Yeah, I think it does. So, okay, a few things, a few different levels to think about, right? One is the value of money, right?  Data, don't put bread to the table, right? So, but if you see like, you know, money is not a currency, right? But it is a value for something, right? In exchange for that thing that you want, it has to be in line with your business goals or your personal agenda. Only then, then you can justify the value for money money. Otherwise it's just impulse buy. Yep. So that is one mindset that you should be seeing. So that acquiring of data, like moving by executing your program is that, I understood that the value of money literally means nothing to me, right? By buying a Rolex watch or by renting a Lamborghini for the weekend. But it means to the world to me, if I got a set of data and also, you know, if all this fails, I could have said like, I spent this amount of money, I learned the ins and outs, right? So I had hands-on experience. So I think the value for money to me was kind of, I understood that, you know, the data acquisition was mine, was that value to me. I understood. Yeah. And the thing is, you were saying, is why people can spend very high K-packs, right? Capital expenditure, right from the get-go, depreciate their value year on year. I think it's a very common business thing, right? Yeah, because I mean, industrial estate and all of us from level one to level seven, 15 units, we're talking about 139 units in one building. We all have this same concept of business, which is high K-packs, non-asset-like, you know, a small amount of digital. And we think this is business, right? So in Hawaii, I believe it has ingredient in us is that, you know, it's not easy to start a business. The capital outlay, the cash you need to cough up is high. True in one sense, but having said that, like with the pandemic and all, we understood the fact that, you know, there is a global shift in consumer behavior or there's a shift in business model already, right? So if we fail to adapt this, then, you know, it will be very tough unless we have like a couple of millions of dollars inherited from our parents insurance, right? That we can burn it through. Otherwise, it's every dollar counts and it's to be as asset-like as possible. So everything from K-packs to O-packs, which is capital expenditure to operational expenditure, right? So the revenue and your expenses will just tie together instead of a high K-packs. But then your revenue is like, COVID hit, boom, you know, right? Yeah. So I think... It's quite natural what you're saying, right? Is that what you're saying? All right, yeah, the natural shift is in this, you see, you need to be very comfortable in buying data, right? Buying data and trust me, like everybody is talking about data scientists, tech, this, that, VC, all this. We know the key words or the most fashionable words is data, right? Your data. But why are we not willing to spend money to acquire those data, right? Data is the most valuable. It's gold, it's gold, basically. It is, it is data. It's the number one commodity right now. I truly believe, right? Yeah. Yeah, that's it. Okay, my follow-up question. Can you go back to your analytics? The one, basically that month to date, whatever, right? So like your growth curve is very aggressive. You basically took a business from zero, basically. Like 100K per month, brand-name. You hit 50, congrats, but then the PayPal thing happened. But I guess my question is, for me, for example, I know how to do it, right? So I've confident, then my positive feedback loops telling me, I know how to do this. I'm confident in myself, right? And I have no fear launching, right? But for you, for example, it's like the people who run their own e-com source, they only have experience in that one field, right? So all they know is that account, for example. So I want to understand like, how, I have no fear. I have no limiting beliefs, I'll per se, right? So what is that? How do you get that? How are you so confident in that when you start launching? Because normal people, like, you know, other people, I coach, for example, right? They're not like that. But for you, it's like, hey, just go, go, go, go. Where does that come from? Okay, so the thing is, you know, one thing is, again, like, let me go back to your video, right? So I watched all your videos, literally, like, I'm a stalker. If you can see, I'm wearing spectacles. People do know how powerful of a creeper I am, right? So, but back to all serious know-this-that, when I noticed your video step-by-step, like, all the way when you were, like, when you were talking slowly in the initial video to gain this, and over time, you know, that confidence increased, and you just, 1.5x play speed, you know, all the way. Yeah, so I took all that free tutorials you offer on YouTube, right? So it gives me an understanding already. So what I did is, I checked through the Facebook ad library of the current e-com stores that, you know, that's been targeting me, right? And I'm looking at their ads. Next thing is I go to their website. I see all of them have a similar flow, right? So if the, if what the big boys are doing correlates with what your video is showing, yeah, there is a pattern, a repeatable pattern, which is why you have to move here, right? Of executing it, because now instead of one percent of the business are focusing on net revenue, you know, I think your most recent video is, like, focused on top-line revenue. Granted, your product for the get-go has to be profitable revenue, right? So profit is a buy-go of top-line revenue.  Like, for instance, new startup that says, my burn rate is this to acquire new customer, but then I think we're going in a very wrong approach, right? So I don't want to know should I be elaborating this or no. Yes, no, yes. I understand your point. I understand your point. Yeah, okay. Do you want to be elaborate for your viewers or? I think it's fine. I understand it. It's fine, like, you're quite clear. I understand, I understand. Okay, four. Yeah. Once you notice this pattern with John's teaching, right, then you realize, like, okay, this makes sense. Everything starts clicking, right? So it's like, how do I get access to this blueprint that this architect has really created? It's a working thing. So if you want to talk about the facade of Facebook ads, okay, this is the one side of the thing. If you want to talk about CRO on your landing page, okay, this is the thing. You want to talk about increase your AOV through a proper funnel. Okay, here's the thing. How do I get access to John's three facades of that thing? Now, John not only offered that, but also the complimenting or the supplementary side of the business, which is hiring. The market research is very service-level. If you're not doing it already, you probably... That's good, ah. Yeah, you probably want to do so, you know. Otherwise, you know, you end up blaming that hey, John is not working, but then, you know, maybe you could ask yourself, like, hey, I'm not doing the work. It doesn't work that way, right? So you probably want to do that. But more importantly, it's like, what John's introduction, or not introduction, but his blueprint encompassed the paid acquisition side of the full spectrum of the paid acquisition side of the business. That's one, which is your paid search, which is on Google or Yahoo YouTube. Then paid social, right? And then he starts bringing in the hiring side of things People don't talk about this. So, okay. So I think like he gives a very holistic approach to how your business should be. Yeah. So I think like, does that answer your question? Like, why do I think this works?  You have confidence in the ability to deliver and you just execute that, right? I think a lot of people overthink it. It's not, it's not, it's like, because if you have never done it before, you never seen something that you don't understand. You don't trust it, right? It's quite simple, right? And people put claims of 50K per 100K per one in 30 days, like, that's crazy. Because I think, I don't know about you, like the mentality of people, it's just I cannot get rich fast. And get rich quick is like debut, right? But if you know how to operate business, if you just know cash flow, if you understand that, that it's actually quite simple.  Actually, if you don't make money, just reinvest in the business. You'll be in a much, much stronger position. And not only that, you're building generational wealth here because there's an email list, there's everything involved. And I don't know about you, but like a lot of people cannot think beyond three months, which is crazy to me, because you're gonna be living life for like how many years, right? But they cannot somehow just not suffer a lot.  I think so, it's, you know, what you say is very true is that, okay, for those people who can't think that long term, see it in this way, right? Big businesses are aggressively gaining market share, right? We're talking about the lights of what? Uber, Airbnb, right? They might be operating at a loss to acquire that much. Yeah. But their database is huge, right? So when this database is huge, you can potentially extract more value out of them. So when we take this content and apply it to our smaller scale and towards our business model, then I think it would be a better clarity on why we need to build a database or why we need to build, right? Because we are able to provide more value in exchange. We are able to profit from it, right? By allowing people to benefit from our product or services, we can profit as well, right? So I think that is the thing that people should see in the long term, yeah. So... Yeah, I understood. For you, for example, your family situation, I don't know about you, this is my perception when you committed programs stuff like that. It's like, you've got a child, right? You've got a wife that's essentially like, alpha female, she's pressuring you, right? There's a ton of pressure and a ton of expectation on you. So with the expectation you perform, I don't know about you, but a lot of times when people, they just don't have that willpower to work or they don't have that drive.  So here, I'll pin it on that, not sure whether you agree. Okay, so I think... Oh, by the way, Jerry, you can stop showing your screen if you want, by the way. Nah, we can keep this as a nice backdrop because if I show you this, see my background is like full of crap. So, yeah, this looks prettier, you know, as though I'm having a very clean office. So, okay, as to your... Some sure. Yeah, to your point of view is that, okay, it's very simple like, are you able to operate under pressure, right? So a lot of people say that no problem, I will get my hands down and dirty, but when punch time comes, can you literally operate under pressure, right? And most of the time that, you know, those who say they could, I would say 70% or 70, 80% wouldn't be able to handle the true pressure when it hits them. Yep. And then it's perfectly fine. That's nothing wrong. At least you've tried to get your feet wet in entrepreneurship and understand the hardship it brings and it's not all the bloods and glory that, you know, you seem, oh, how much people they raised, right? But you need to build that tenacity or you need to build that very strong will to push it through. You have to do it and by... And how you should do it is use your own money. Don't raise investment. No excuses, yeah. John, it's probably not in his thirties who's working out of his room.  He's at the back with a big band, right? Sticker. So, and he has 10 millions of ad spend with a net positive result, right? Myself, it's, okay, you know my backstory, right? So, which is, you know, a family guy with a day job and I'm doing everything with regards to this new econ business, right? No way you could give yourself an excuse because if you give your chance to come up with an excuse you will most likely do the same and say, hey, John, seriously, what the hell, man? It is true, right? I think it's about reflecting and understanding, right? And being honest with yourself, like, have you put into work before, you know, pushing the blame to everyone else? It's very easy to push the blame to everyone else, right? So, I think that is, that is the, you have to be honest with yourself, right?  Yeah. Yeah, yeah. I just, Frank, if you like, I don't know, maybe because like from Amir, right?  Like, yeah, so I don't know about you, but like, maybe I'm just too harsh on people. There's, you know, not cut, not come from the same cloth or like, do I have the same style of like, heart-nose mentality, but if you can't get it done, just say, you're not fit for it. You're not, it's not good enough. And it's okay, right? Like, if you don't want to run a business, a business is hard. If you don't want to run a business, just say, you're just not good enough, right? And just, yeah. Yeah, there is absolutely no wrong in admitting that being a startup founder is not your cup of tea, right? You can actually- Hey, jobs are cool, man. Nine to five's are cool. Exactly, exactly. Like, seriously, my friend works in Facebook. Yeah. Okay, close to, close to what? For 500K? No, no, no, no, no. She's young, y'all. She's getting paid close to five figures a month. Then she switched over to bike dance, TikTok. I don't know. Yeah, so now it's five figures per month already, right? So there is a stability and you still can get paid well, right? But then it doesn't come with all the problems that when you are home alone, thinking- Emotional baggage. Correct, because if it's too much of a pressure for you before you go to sleep, you are thinking what it's gonna take, what you can improve on. And this is part of business. Everybody will face that, right? You will crack.  You know? But that, of course, you are not being responsible to your own brand, to your own business because you don't understand the business. How do you expect someone outside to understand your business? Seriously, yeah. Yeah, I'm sorry. I wanna ask you more about the product itself, like the brand. Okay, I won't say the brand now, but basically, your background is engineering, right? You said? Yeah. Yeah, so then you went into not physiotherapy. I don't know what to call it. Is it physiotherapy? No. No, it's orthotis. So I'm a brace maker. Okay, then how did that transition into building this product for e-commerce? Is it like you saw the gap in the marketplace that you just went to solve it, et cetera, et cetera? How did that play out? Yeah, it could be in the gap in the marketplace, but more importantly is that when I do my brace for the kids, don't get me wrong, I got them good at my job. People have seen very good results. I swear, I am good. And I have no shame in saying that. Seriously. Yeah, of course. I'm bad. I'll tell you I'm bad. I'll work to improve it, but if I'm good, hell man, I am good.  You know what I mean? So back then, I was an active fitness person, right? To the level that I competed competitively in Spartan races. I got like a little ambassadorship from Reebok, right? Okay.  That's for me, my product, right? So I went to create something. Literally, I go to the bakery store, buy whatever that's baking essential. So my mindset is, okay, Wiki House says this one has certain properties. So I buy this, I do that. I did a basic prototype. Okay, it works for me. So I find a manufacturer. Can it be done on a larger scale? Oh, it can be done, right? So if it works for me, can it work for my friend? So I try, oh, it works. It works for them and they find value in it. Okay, okay. Maybe it's just my friend. Let me just try, hit out to, let me just reach out to Instagram people who have like surgery people who are squad. Then don't get me wrong. There's a lot of groundwork. You need to like go from places to places just to get their point of view. But all these, aren't your market research, right? You understand what they're looking for and if they abuse it, are they aware of the problem or are they aware of the solution?  Oh my God. Everyone is happy. One, one did say like, maybe it's just in my mind, it's better, but I don't know if it works. I don't use it, but it is better, but it's just placebo. So, so we know, we know already, there's a consistency like this. Proof of concept. All right, there's a proof of concept and proof of concept can be done. Okay, I spend like less than 20 bucks to do my proof of concept. Nice. So, so, dude, it's not about 5,000, yeah? Yeah, it's no excuses, I basically, yeah. Yeah, seriously, if you can't get your prototype to work at whatever you can get your hands on, right?  All right, so think of ways you can get your proof of concept as soon as possible at the lowest amount of cost because what you're gonna be spending on most of the time and most of the money, it's acquiring those customers who are willing to pay for your product, not at the discount, but at the value that they feel is justifiable to solve their problem. Yeah, understood, understood. How does that, let's go, when you sit, okay, I'm gonna build this product now, for example, right, you go to your supplier, get whatever you need to get, right? How did your family react? Cause you're essentially going to a new business, right? You're basically like pivoting. How did that conversation go? Oh yeah, I did not have the conversation with them. Okay, so you're just like, you're just going in and just, hey guys, I'm actually doing this, then they're like, yeah, yeah, sure. Yeah, so you was thinking about Asian family, right? So the stigmatism is so true. My European friends who are watching this is that everything we do, we will fail from the get go. Do you get that, John, from your parents when you started Oxygen Media? Yeah, I didn't tell them.  Do you get that? I didn't tell them. I was just like, hey, I'm doing this, they're like, oh, okay, then, you know, then just results speak for yourself, right? You know, you just, you can't win, right? You can't be like, unless you get results. So results just like, hey, results, so yeah. Okay, so yeah, I did that without informing them. I only informed them when I started seeing positive results. Hey, look, I mean, like, hey, I mean, it's not, it's not about lying to them, but it's just withholding the honesty, right? You're just delaying information. Which is not telling them anything, not, yeah, yeah, yeah. Yeah, of course, because you need to consider other people, how they would react, right? Like jumping into a new venture, especially in this time, is- It's like, am I crazy? Yeah, it is, it is, right? So why are you giving up something that is stable for something into the unknown, right? And yeah, so in my case, I could have approached the matter better by sitting down to them and telling them this is what I'm gonna do, more responsible family member, right? So, but I did not, I choose to push forth and tell them later, if it backs fire, oh my God, you know the dragon lid.  All right, all right. Asian families ruled by the wooden cane. Have you got that in the past? Yeah, yeah. I think about 70 cents, exactly. Yeah, yeah, yeah. Right, so, so, so I got that, right? So I, in a way, it kind of grow me or kind of like build my, yeah, correct. Resilience package, yeah, yeah, yeah, yeah, yeah, yeah, yeah. Yeah, right? So I was told like, I never cut up for business, that's one. My product is not gonna sell, that's two.  Probably not good enough, right?  Right, correct. The Chinese saying is, Yi Nian Jian Hong, what it means is that the first year you will see red, right? It's not green, it's red. First year, do you have that money to sustain you? You only see profit in three years. That is the Chinese way of thinking of business, right? Yeah, because they do not run paid traffic. The mindset, right? That has been angry in the Asian countries, right? So, or maybe it's just me, but that's what, you will hear this. So Yi Nian Jian Hong, first year, negative, right? No profit, losses, losses, losses. So I was brought up in this, everything will fail, everything can fail, everything will, you know? So I really resent that because I love eating popcorn and, you know, watching wrestling and learning all the weapon things, you know? I want to lead the American dream. Yeah, so that you guys truly believe in dreaming, right? Going for it, pursuing your goals, right? Mistake is just a stepping stone to your next chapter in life. That's it, I really love this approach, right? So that thing sticks with me, right? So yeah, that's how I have this mental game to push. I understood, I understood. I think especially if you build a product from scratch, then you validate it. And then people see success with your product, right? Then that's, I think, the best feeling ever, right? As a founder, I'm pretty sure, right? So because you build from selling from scratch, how did that, I guess, what validation or what, how do you reach out to people in the first place? That's the first question. And the second question is, how did you take that validation and then transition move on? Okay, so first question is, how do I reach out to people on Instagram? Yeah, let's just DM, let's just... Yo, man, like, I saw you have this thing on your most recent post. I'm so sorry, but hey, dude, I have something. I just want your feedback on the prototype that I'm working on. Nothing else, I don't need your post. I just want your almost opinion, right? If it's good, give me a shout out. If it's bad, if you hate it so much, tell me I'll work to improve it and I'll give you your 2.0, right? So it's valuable. Like John is creating that positive feedback look, right? But in this case, it's like having a feedback, positive, negative, but then you're giving yourself another way out, which is I will create a 2.0 cliffhanger for you, you know? So that's one. So it doesn't come across super aggressive, like you're pushing them into something, but you just want your opinion and you have to get an opinion because you are nobody in this market, in this... So that's how I reach out to them, right? And in part two to your question, it's how did I transition just to the validation? How are you so confident to like move in the next thing and what's the next step, yeah? So I moved over to Kickstarter. Okay. Yeah, correct. So at Kickstarter, you know, I was... What did I do? Yeah, I put it up on Kickstarter. Yeah, so I was told Kickstarter, if you're clients who does Kickstarter, they know like there's a pre-launch email buildup phase of, I don't know, at least minimum a month, but ideally it's two to three months. I did it in two weeks with two days of ad account ban, so 12 days. But hey, I still get funded. Yeah, nice, nice. Yeah, so what this means to me in my small head is that there is an actual, all the market is actually resonating to the product, right? So my product validation is there already. Now when I can come into e-commerce like last month, like together with John's program, is that now I need to find a price point. Are they willing to give out price point? Correct. And it's not about the discount one. It's, you know, it's not about like what, it's not about having the great fire sale, one day flash sale 70% off and do it like, don't guide your brand, right? You don't see iPhone giving up 50% off, so. Yep. But granted, you can say iPhone is a big brand. Back then when iPhone just, Apple just started, they also don't give 50% off. Of course. You're gonna try the wrong type of buyer. Correct, correct. So attracting the right type of buyer is very important, right? So that is one thing that you really need to protect your brand equity. One, two, it's understand who are your potential buyers, who are willing to pay the money that you are wanting to get from them in exchange for the value that you will be providing them through your product or service. Yeah, if you can't find those people, honestly. That's good, you're essentially good. Yeah, yeah, correct, yeah. Yeah, you go talk to any venture capital or whatever. All they say is, do you have distribution? If you do have distribution, right, you are screwed, you just get out of the office basically. Yeah, so yeah. Oh dude, I jump into kind of like a pitch deck for startup, right? Oh, seriously, when was this? When was this? My very first Thursday call with you. That's why I have to jump in and out of calls, right? So this is BC or just startup contest? I think BC, BC, okay, so it's kind of like a sharp thing. Okay, Singapore? Yeah, Singapore, it's very good. They're providing all the investors to the panel are providing a lot of value. But startup founders have this mentality like their business is worth 500K, 1.5 million. I have no idea where, yeah, seriously. Dude, seriously, right? My maths, I got like C6 in GCEO level British examination, right? What this means to your US viewers is that I get a barely passed in my mathematical exam when I was a kid, but zero multiplied by 1.5 million is still zero. It's either that or I probably missed out on curriculum. I probably have slept in class, that's why. But I have no, the fact that, you know, they come out with the idea like there's burn rate. Burn rate is equivalent to guys, I'm asking for money. This is the amount of money I will be losing every month. You are literally from the get go. It's kind of like finding the right customer, right? You are having the wrong mindset already.  Dude, like. I think the most toxic or pollutive thing about VC is like, the founder is like, I can spend money here. If you have the mindset that you cannot spend money, I think you just be a lot more resourceful and just make it work, which is what you did, right? But people don't get that. Like they always think they go to VC route, but then they also don't understand that your valuation is based on EBITDA. It's not even based on revenue. It's not based on top line. People value, buy companies based on free cash flow, not anything else. So like, if you're not even profitable, just get all off, please. I mean like, at the end of the day, like you hear from the side of investor is to have skin in the game, right? They want skin in the game. They want money coming. But so if you take that idle skin in the game to your own self, to your own startup ideas, is you need to have skin in the game to feel the burn man. You need to put in the money, the time, the work, right? And all these, I mean, I'm gonna do a little bit of marketing like Koyote, right? For you. So all these, they kind of grease the approach, right? What I did was I just bought into John's program. Oh, I did not question this thing. All right, so it proved my business model. So that's the thing. So I think like, you know, you don't have to raise funds outside to get a successful business model. I'm sure you see a lot of drop shippers like doing that. But then of course there are business kaput within the first year, right? A second year. Hey guys. That's not second year, it's a few months. Yeah, the second year is I'm, if you give me 30 seconds, I will share with you. This one, you take this one, you too. Oh, that's something. Oh, I was, I had this dream. I had this dream, you know. Yeah, so. How are you? Yeah, yeah. Okay, okay, okay. I didn't know it's a real thing. Okay, okay. I don't want to, yeah. So, I think like it's, you need to have skin in the game. But of course, like you want to be very confident in moving in. So if you're planning to ask for a VC role, before you even do that, please, please, please, just work with John in any capacity, right? With him or with you. Okay, this is all you, this is all you, not me. Oh yeah, I'm joking. We're talking about the startup pitch. Oh yeah, the startup pitch, right? Is, does it add any value to it? Okay, okay. So. I'll ask you how you transition. So you said you're talking about the startup pitch, remember? Yeah. Yeah, I think like startup founders have a very... Illusional. Yeah, correct. They, here's the thing, like they seem a lot of shark tank. According to a friend who is part of the automotive industry. They, they seem a lot of shark tank. So they come in valuation, but one thing they don't realize is one, market size, US have like what? 3 million pop. Singapore have like 5.5 million pop, right? Moving towards 6.2. Next thing, those shark. Because has actually revenue, right? This, this, this people who are pitching is like, dude, it's 1.2 million. There was no anything. Oh yeah, 1, 2, whoa. Hahaha. You're just throwing data science and AI and money. Hahaha. Like, you know, it's like, even if you, if you know, think about it this way, like if you go and ask your dad for money, that, I have an idea that's worth 2 million. Your dad will probably take the latest magazine from toilet and slap your face twice. Hahaha. Yeah, exactly, exactly. But yeah, but now it's common, now it's common, right? Yeah, like, I don't know, people, people don't like making profit. Like just run a profitable business. Why, why, why do you need extra money? Just do it yourself, right? So, yeah. Okay. Okay, so I think we've gone for quite a while already, Jerry. But I guess we transition more to finishing up. I guess, do you have any advice for people who haven't seen Breakthrough? Okay, don't, don't say buy my cosplays. I want you to speak for your heart, really. Just, yeah, your experience as a founder, yeah. For my heart, right, I think it was like, for people who are thinking of doing it, please buy Ormin's Cloak, okay? But understand your, if you are financially able to explore this new project, right? It be it Ecom or high ticket service, you want to provide consultancy, right? If you have the savings for you to last through the next at least one year, right? Please go and explore new things, right? That's one thing. For people who haven't explored entrepreneurship, right? Or start their own business, they want to do that. Please buy Ormin's Cloak and do that. It's a very fun experience that unless you put your feet wet, you won't be able to share this experience or have the same group of people, like, you know, mingle about the same things. The next thing is for people who have already started their own business or are in senior management and are thinking of starting their own business, are already started, right? Is that, you know, do not fall into the trap of, like, because of my past experience, this way it's, you know, it's not working. It probably won't work. Like, don't, like, you know, like, be like me, right? I, no, seriously. Yeah, I agree with you. I agree with you. I agree with you. Absolutely no knowledge. Do it, right? I have one motto, it's called just do it. Wait, isn't it? Yeah, it's probably a... Profing. Oh, okay. So it's not my motto. Sorry, sorry, sorry. Yeah, but literally, like, just take action, right? And don't, don't come in with a... Close mindset.  Yeah, that's the thing for people who are experienced, right? People who are non-experienced, oh, you are the best man, you are like a sponge waiting to absorb all those knowledge. But please get your information source from, yeah, please get all your information from the correct source, right? So that, I think is the biggest takeaway. It's to go in with an open mind and to absorb everything and just please execute the thing. Yeah, I think so. Okay, Ken, is that it? You had anything else before? Do you have anything else or? Yeah, I think me, that's all right. Okay, Ken. So I'd like to thank you guys. Thank you, Jared, for your time, spending like an hour plus with me. Well, it's been an hour. That's why it's been an hour. I know you're busy or so, that's why I was like, holy shit, okay. Yeah, so I mean, thank you guys for your time. And just, Jared, if you wanna reach out to him, I'm sure he's on Facebook somewhere, that you can reach out to him. I'll maybe put a link in the description or something. Yep, and I'll see you guys in the next video, okay?


Hello, everyone. Thanks for joining the call. So the presentation. So today me, Shashank and my colleague will present how to deploy a stateful broker, which can be deployed in multi-availability zone, have NZDM qualities, as well as it can be working on multiple cloud. So I made this a statement. Stateless design is the crystal meth for brokers. So why is that so? So majority of the times we are already into the world where people talk all stateless design. But do you really think that everything can be stateless? I don't think so. So in the words of the famous distributed computing guru, Jonas Boner, the creator of a car, he says, isn't state, you can never remove a state, you can just shift the state, right, from one component to the other. So you are making somebody else's problem. So the idea here is if you have a broker, the cloud foundry broker or any OSB API spec specific broker, can it be totally made stateless? Or there is specific requirement where you need to maintain state. So this talk centers around the concept where we say how to write a stateful broker. And then what are the challenges you face in terms of providing NZDM qualities, how to deploy it in multiple availability zones, and then how do you get good fault tolerance out of it. So as I said, there will always be requirements where we need a request spanning across or multiple requests going across or maintaining certain state. So you can never say that I will be totally stateless in all kinds of system design I do. So there can be requirements like you have to maintain cash in one of the nodes. So what happens if the request then goes to the other node? There is no cash. You have to recreate the cash there as well. There can be issues like where you want to coordinate between two nodes. As an example, if you have a scenario where you want to deploy certain things, then you make a request to one node and if that node goes down or if you want to concurrently deploy something, it should not happen that both nodes pick up certain IDs which you want to coordinate on or which you want to attain a lock on. So it's basically the problem boils down to doing a coordination between distributed nodes. So how do you solve such problems? Then how do you maintain a state? This is very specific to the polling API which Cloud Foundry provides. So how do you maintain a state to serve across the polling APIs? So want to make a stateless broker. So if you have a stateless broker, there are very well-defined principles where you say I will add a load balancer in front and do a scale out. This is easy to do. Increase the number of nodes and you can do a horizontal scale. But this is not true when you have stateful or state being maintained within the broker. So think about this. As I said, you have to coordinate between nodes. So take a classic example of what we do with Service Fabric today or I'll come to what Service Fabric is, but in principle it's deploying, doing a Bosch deployment where we need certain state to be maintained. Reason being that we don't want two nodes to take the same idea of the deployment and try to deploy the same thing. So there are problems of doing a lock contention, solving how the locks can be maintained, how they can be coordinated, how nodes talk to each other in a seamless way, which is also scalable. So go back again. The other problems which we see now, we are talking just saying that we solve a problem of having an HA and we have, let's say, a coordination mechanism solved. But what happens from the consumer perspective? You register the broker with the Cloud Foundry, Cloud Controller, but then you only provide a single IP to it. Now, as an example, you say that I want to serve requests from a different node because the first node has gone down, maybe the availability zone has gone down. How do you switch the IP? You cannot go to the Cloud Controller and register it with a different IP. So you want a stable endpoint exposed to the Cloud Controller from your broker. Then you need support for zero downtime maintenance. So in case of Bosch, if you are re-selecting a virtual machine, you might take few minutes, but that is not the downtime people will accept. So you want to develop a system where you can get NZDM qualities, where you can provide a stable IP and you can also be fault tolerant across availability zones. So now we talk about what is the example of such a stateful broker, where I hand over to Keith to talk about service fabric in the context of a stateful broker and how we have mitigated or addressed certain concerns. Thanks, Shashank, for setting up the context on what is stateful broker, what are the challenges there of stateful broker? An example of a stateful broker, no surprises there. It's service fabric that we have been talking about. So let's see. What is the service fabric architecture? So this is a diagram of service fabric architecture. This is the laser? Yes. Yeah, okay. So any user using service fabric services can go via various platforms. It can be Cloud Foundry, it can be KITS. Using CFCLI plugin or other UI or any other CLI, user can create a request on a platform that I want to create a service instance, which is provisioned by a service fabric broker here. So the request comes to service fabric broker. So there is a plus plus here. If you notice, there is a reason for that. So service fabric broker is an OSBA API compliant service broker. But apart from that, service fabric also maintains and operates thousands of instances in SAP production. So we need some more extra capabilities for this broker to handle, like backup and restore, scheduled updates, scheduled backups. And these are some of the extension APIs that service fabric broker supports, and hence the sign plus plus. So you can think of these extension APIs being fulfilled by, let's say, actions that is coming up in OSBA API spec. So when service fabric broker gets this request, it uses API server as a resource manager, where this is API server from Kubernetes, and it is backed up by HCD. So this resource is created in API server. Now there are various operators that are listening to this resources. So take an example of Bosch operator. So whenever there is a service instance creation request for Bosch based deployment, the resource created here is notified to operator. It gets that, okay, I'm the operator. I am supposed to solve this request. So it in turn gets the lock on the resource and then processes it via with the help of Bosch and it creates a Bosch deployment. So as you can see here, I have here one KITS operator, which is a dotted line. So with this kind of event-driven architecture with the help of API server and operators, we can bring in any kind of operators here. So be it KITS operator, be it, let's say, you want to provision Ali Cloud's native Upser RDB RDS. So you can easily add another box here saying Upser RDB operator, and it can serve the purpose that you want to provision. So I hope you understood the overall architecture of service fabric and now where the challenge comes in. Because now service fabric stores the state in API server and this is currently one VM and if we add one more VM, how does this consuming platform talk to the service fabric broker? How do these nodes coordinate between each other? Then there are multiple operators. If we are running multiple instances of these operators, how do they coordinate between each other? How do they manage the locking of resources? So various challenges come in and you can make out from this slide here. So let's move on. Let's see how we have solved this for service fabric where the setup is multi-AZ. It is highly available and it supports near-zero downtime maintenance. Okay. So I told you that there is currently one VM in our setup. So this is, let's say, in Availability Zone Z1 with some IP. Now, let's say we want to make it HA. So we add one more VM. So it is in separate different Availability Zone Z2. Now there are two VMs. Okay, fine. But how do they coordinate between each other now? So we have Keep Alive D process running on each of these VMs and Keep Alive D is nothing but a mechanism which decides who is the master, who is the backup node via VRRP protocol. And it basically sends each other these hard bits to decide who is the master and backup. So if one node goes down, the other Keep Alive D knows that, okay, I didn't get hard bit. So I'm now supposed to promote myself to master because master is down and it promotes itself to master. So this is basically a mechanism and this Keep Alive D process is running on each of these VMs. Okay, so now what are you solving with this setup? Just by adding one more VM and Keep Alive D. So we are solving multi-AZ issue of service fabric, making it more fault-tolerant and we are solving highly available functionality of service fabric broker. But there is one problem. So how do we, if some consuming platform wants to talk to service fabric broker, how does it talk? Which IP does it use? So that is why we need another virtual IP here, okay? So we are currently using 10, 11, 250, 210 and this virtual IP is a concept. I'm currently explaining the generic concept for this setup. We'll go into details of IaaS but let's assume this is a concept where we provide a virtual IP which is a stable IP. So this virtual IP then now points to master. So that whenever a request comes to this virtual IP, it is redirected to master and served via master. And hence, whenever service fabric registers with Cloud Controller or any other platform, it registers with this virtual IP because it's stable and we don't need to worry or consumer, consuming platform user doesn't need to worry about what is going in the background, how is the failure happening, who is the master, who is the backup and which IP I should point to. So it can just connect to this IP and rest is taken care by the setup itself. Okay. So now let's see, this is the ideal situation we want to be in. That master is always alive and it is getting served very peacefully. All the requests are getting served but we live in real world so it doesn't happen and hence we need all this setup. So let's see what can go wrong. So let's say this VM process, service fabric broker process goes down on this VM. So Keep AliveD notices that there is this process which is crash, it's not running. So what it does is it says, okay, now I cannot be a master, I will be backup. When it turns to backup, this Keep AliveD knows, gets notification and it promotes itself to master. So when this is promoted to master, the virtual IP is shifted to this newly promoted master. So now this virtual IP here is now pointing to this newly elected master and as you can see, this consumer doesn't have to bother about it if there was a failure, failure. So this is the typical failure situation that can happen in production or any environment and with this kind of a setup, what we are achieving is nearly zero downtime maintenance and stable IP. Why nearly zero downtime maintenance? Because let's say this was a legitimate go down of crashing of process because the VM is getting updated, the broker process is getting updated. So this can be a legitimate situation and in case there was only one VM, it would have been total downtime for service fabric broker but now since we have this HA setup and multi-AZ setup, now it can as well be served by this master while this node is getting updated and later this node can get updated and this can become master. That is why the NZDM concept is also achieved here. Okay, so now we have seen a generic concept or architecture diagram that I just explained. Now as I told you, this is just a concept that we want to have a stable IP. Now service fabric runs on various ISS, currently it is running thousands of deployments on AWS, Azure, GCP, OpenStack, Alibaba Cloud is also on the cards. So how do we actually implement this architecture in various ISS? There can be multiple approaches that we can take on different ISS. So let's take an example. Okay, let's take GCP and Azure. So if you see, this is a similar diagram that I just showed you and explain you. So just to reiterate, there are two VMs. They sit on different AZs. There is a keep alive D on both of the nodes. One is master, one is backup. Now this is the box that is changed in the slide. So instead of virtual IP, there is internal load balancer. So on GCP and Azure, we use internal load balancer to achieve the stable IP. Now what is internal load balancer? It is just attached to multiple VMs here. In case of this particular architecture, we have VM, broker VM 1 and 2 attached to this load balancer. Any request that comes into load balancer is forwarded to a healthy node via TCP port forwarding. Now what is called healthy node? What internal load balancer does is it has this HTTP health check. So it sends HTTP get request to a health check process that is running on each of the VMs. And whoever sends HTTP 200 okay is the healthy node for load balancer. And all the requests are then load balance. So if there are multiple nodes and some of the nodes are unhealthy, all the requests are transferred forwarded to healthy nodes. In this case, now let's see how we utilize this kind of load balancer in our HA concept. So what we do is when HTTP health check of load balancer sends HTTP get request to master node, it sends HTTP 200 okay. But whereas when it sends to backup node, it sends back HTTP 500 error. So from load balancer point of view, the master node is always the healthy node and backup node is always the unhealthy node. Though it is not exactly an unhealthy state, but since we want to route all the traffic here on master, we make this backup look like it is unhealthy node. So all the traffic that comes from a platform gets routed to master. Now in case of failure state, what happens is now this VM2 is promoted to master and this is a backup. So when HTTP health check asks for the health check, it sends HTTP 500 error because now this is a backup node. And this newly promoted master will now start sending HTTP 200 okay. So like I said, this internal load balancer then always sends request to master node because it is healthy node. It is posed as healthy node and backup node is there sitting for the backup. So this is the approach that we have taken on GCP and Azure. So I'll hand it over to Shashank to explain it on OpenStack. So I think as Ketki said, different IS have different requirements and different ways of addressing a stable IP. So as an example on Azure, we cannot have an IP which can work across availability zones. So for case of OpenStack, we use a mechanism called an allowed address pair. I don't know how much of that concept you are aware of, but the basic principle is that on a neutron port, you are able to attach one more IP, which is the floating IP or the virtual IP. But when I say floating IP in OpenStack, it is a different connotation because it becomes public. But we use an allowed address pair to make that IP private, which means that in case of a failover, I can just update the neutron port and switch the allowed address pair based IP to a different port. So here if you see the same scenario, what Ketki also talked about Azure and GCP, same concept applies via VRRP. The nodes are determining who is the master, right? And once the election happens, we do a port update to switch the IP to the master node. And all traffic over the virtual IP then it gets redirected to the actual master in this case. Again, so now if you understand the architecture of service fabric, we have a concept of a broker, then there's an API server, and then you have operators. So you might be wondering that we talked of only the broker part being made HA. But what about the operators which are running behind the scenes, which take the job and then do the actual deployment? So we have taken care of this, but there's a different mechanism because they don't need a stable IP because they are not the front end components which are interfacing with Cloud Controller. So what we do here is as an example, you see three operators, but just take an example of multiple Bosch operators running in a scale out mode. So whenever the request comes, so this is what you see. So when a request comes to the API server, both the nodes are watching for the resource change or the CRD change. What changes are happening? Is there a new deployment request? Is there a demand made on the system? So once both the nodes see, both get the events and both try to do a deployment for you. But before doing the deployment, what they do is they attain a lock because we already have an API server and behind the scenes, it's an LCD being there. So what they do, they attain a lock on the API server and only one node picks up the deployment and does it. So basically what we are trying to do is not enter into a race condition, but only one node picks up the deployment and always the system is in a consistent state. So this is the whole flow where I mean you can then scale out multiple Bosch operators and then do the deployment. Maybe then we go over the demo quickly and showcase. So let me just go over the demo quickly. Okay, so what we'll show is from the GCP. So where we have a load balancer and it is attached to both of the VMs. So I'll quickly explain how this pains look like. Okay, so on the left hand upside, we have a CF pain. So I'll do some CF operations there. Then we have Bosch pain here. So I'll do some Bosch operations here. And here if you see these are the two broker VMs that I'm logged into. So we have two node set up and I have logged into these two VMs. I'll explain in a moment what each node stands for. So let's first see what are all the services in the marketplace. So we have this blueprint service, which is a demo service for us. And there are various Docker and Bosch based plans there. And currently there are no services created, right? Then let's see how our service fabric broker deployment looks like. So I'm not sure it's really visible, but if you can see there are two broker nodes here, which are in Z1 and Z2. So two different availability zones and they have different IPs, 250, 211, 253, 11. These are the private IPs. Yes. And if you see now that these below pains basically represent these two nodes. So this 9737 is the left hand side bottom pain and CC8E is the right hand side bottom pain. Now let's see what these nodes have. So what is the state of these nodes? So as I told you, KeepLivD maintains this state. One node is master, one node is backup. And this is maintained in a file system by KeepLivD. So you can see that this bottom left is master. And similarly, we can see the other node should be the backup node. Yeah, so this is a backup node. So now what we will do is we'll try to induce a failover and see if the node request gets served via the other node. So we'll log in into the master node. We just saw that this first node is the master node. We'll log in into it and let's see what all processes are running in the VM. So you can see this service fabric broker process along with some other processes are running. So this is the process basically who tackles with all the lifecycle operations of any instance. So what we'll do, we'll also tail the logs in both of the VMs so that we know which request goes to which VM. So the master node should always get all the requests. So it will be evident from the logs that we see here. And similarly, we can tail here also. So you can already see there are some info or API endpoints getting hit on master node, right? So okay. Now let's do one thing. So let's create a service instance. So you know, service instance creation in service fabric is a sink call. So it will get created and the operation will be in progress. Now currently this is the master. So you can see that there are the requests are coming to this master node. And you can see last operation call from CF, which is getting by served by the master node, right? Let's see if the service instance ID matches. So if you can see 154 f4 and here also the ID is 154. So this is the instance that we are getting requests for. Now what we'll do is we'll stop this service fabric broker process. So we are currently in master node here. We'll stop this process and let's see if the failure happens now. So we will check that this, the process is stopped. Yeah. So you can see it is not monitored now. And let's see now this, this was the master VM. Now what is the state of this VM? So with keep alive, we can see the state of the VM and currently now it has gone to path. Yes. So the failure has happened. Now the master has become backup and you can already see here that the request has started coming to this, this other node which was backup node. Now it is getting, it has promoted to master. So all the requests are coming to the backup node now. So in a bit, in a moment we can see that this create in progress is also turned to create succeeded. So in the previous case what would have happened is the creation would not have succeeded because the node was down. But now there is another node which can very much take care of the creation in progress. And you can already see that the creation is succeeded. So what we have done is reduce the downtime of minutes to few seconds by this mechanism here. Okay. So then that's it from our side. If you have any questions for us, feel free to ask. Thank you. Great talk. I have a question about state. So the state is basically in at CDN, in the API servers, right? Yes. And you use Bosch to manage and scale at CDN, is that correct? So the machines for the service broker fabric are Bosch deployed and there is a Bosch job. That is one of the flavors. Yes. Okay. Thanks. Which we are using as a coordination mechanism. Right. So only one node at a time should attain a lock on the system. So we use at CDN to persist or to have a lock locking mechanism there. Okay. Well, thank you Kitaki and Shaka. So you can also reach out to us on our email addresses. Service fabric is already incubator project. So you can also join us on Slack and also reach out to us at any time. Thank you very much. Thank you.


We are one day away from the March 18th anti-war peace rally here in the belly of the beast Washington DC. CodePank will be joining over 200 organizations as well as tons of individuals out on the streets and calling for an end to all wars to fund people not the war machine. This is coming at the 20th anniversary of the U.S. is legal evasion of Iraq and we want to see as many of you join us on the streets for peace. We'll be meeting a little bit before the rally starts which is 1 p.m. in front of the White House. Wear your pink. We will have lots of banners and signs and get ready for the rally as well as the March. We will be marching and then we will also be having a teach-in at the New York Presbyterian Church. So find out more on our website and get involved. We can't wait to see you tomorrow.


USS Somerset, conducts training evolutions with Boxer Amphibious Readiness Group. USS Somerset, LPD-25, with the Boxer Amphibious Readiness Group underway, in the US 3rd Fleet Area of Operations, conducting advanced tactical training that increases war fighting capability and tactical proficiency across all domains.