Yeah, I can hear you now. Oh perfect. What did you have to click? Did you click on something? No, I just plugged in my headset. That's what it was. Gotcha. I think the volume is going to my headset. Good, good. Where are you? Where are you calling from? What country are you in? This is Mississauga, Toronto. Oh, you're in Toronto? Canada. Canada, right. Oh, good, good, good. Great to connect with you. Thank you for reaching out. Oh, I've been following your videos and I really love them. Nobody kind of makes videos so simple like you make them. I thank you so much. And how do I pronounce your beautiful name? How do I say it? It's Raman. R-A-M-A-N. Raman. Beautiful, beautiful, beautiful. Yes, thank you. Thank you so much. I started putting videos on YouTube back in 2017 just to put them up. I didn't expect anything out of it. I just started helping a lot of business owners and helping a lot of real estate agents with social media. So I just started talking about it, you know, on video and before you know it, you know, people started finding me on YouTube and more people started subscribing. And then I finally, you know, learned how to upload videos the right way and how to be more consistent.  So I'm just very humbled by it, Raman. So thank you so much for reaching out. Yeah, really good, you know. Thank you. Oh, you're most welcome. And I'm watching videos from other people, you know, right at the point where I need information, they kind of cut out that information, you know, but your video takes you to the end. Oh, thank you so much. And how can I help you? What are you working on now? That's why I'm liking your videos. Good. Thank you so much. I'm humbled and I appreciate you. I'm working. I'm not, I'm not in real estate. Okay, yeah, anything. You're welcome. So I'm not in real estate. David, I'm working on Facebook ads because I am connected to an adjunct. If you have heard of them, the can can get water. It's like an affiliate marketing. Oh, yeah. We were trained of all. Yeah, so, so, but I feel my ads, they are not going anywhere, you know, like my team members, they get like a good response, or I've tried video ads, I've tried image ads, I have started kind of warming up my page and engagement But they're not going anywhere. I feel that I'm just spending money, you know, and the team members that I have, they will not like to share what they are doing, you know, they just kind of give the general information. Okay, you can do this. You can do this. You can do this. So I started following your videos and I like the way how you teach to duplicate the campaign or duplicate the ad set or duplicate the ad level. So I started following that. But when I duplicate the ad set, I'm charged three or four times like my, my budget is not distributed. I start getting charged if I my budget is $20, then I start getting charged 20 times for Yes. Yes, you don't want to do that. So as you as you go and set up the first ad, rather than rather than doing $20 per day. If you're planning on doing four ad sets, then the first ad you're going to set up at $5 a day. And so you're going to do $5 a day for the first ad, and then you're going to, you're going to duplicate that. And so now, now we have four different ad sets spending $20 a day, total. You get it $5 each. Okay, okay. I thought I should. Yeah. Okay, okay, okay. I'll get you now. Okay. So what I was doing it, I was doing $20 and then I wanted to split $20 into four locations. Yes.  Yeah, I don't want that. I kind of thought. Yeah, yeah, yeah, that's that's what you want to do is make sure that the first ad set is is $5 and then duplicate that as many times as you want, you know, for three more times to make a total of four. And there goes your, your $20. But what I what I also, you know, can help you with as far as just the the objective. A lot of people, you know, they want, they, for example, they want to lead. Right. They're trying to, they're trying to capture lead. But I truly believe that a lot of people start with the wrong objective. So are you, are you choosing the correct objective? Like what do you, what do you, what objective are you choosing over here? So I did the traffic ad for 15 days and then I did the engagement ad for 15 days. And I also did the like campaign for 15 days. That was what was in a training. And then I did, now I am at the conversion ads. Perfect. Perfect. So what you, what you could have done if, if, if capturing leads is the goal. And if you have a landing page, what you could have done is rather than traffic, you do. Okay. Perfect. So then rather than traffic and rather than engagement, you could have went straight into conversions. So that would have saved you 15 days. That would have saved you a lot of money. You would have started capturing leads right away. If you want to, if you would have just went straight to conversions. Right. Yes. Now, now with conversions, your custom conversion has to be set up for sure. Like, are you, did you set up the custom conversion? No. Okay, so that's the reason. So the first step when it comes to, well, first of all, the first step is, is obviously installing the pixel, right? The pixel has to be on both pages. So for example, let's say, let's say you have your landing page, right? Let's just say, let's use this one as an example. We're running a Facebook ad and we're taking them to a landing page. This is our landing page. So the Facebook pixel, obviously, you know, is on the landing page, which you're, which you know about. The Facebook pixel also has to be on the. No, my pixel is on the welcome page. Yeah, so my pixel is only on the thank you page, not on the landing page. Okay, you, you definitely want it here. So when they click onto the landing page, they are. Okay. Go ahead, tell me, tell me. I'll be quiet. Thank you. No, no, no, no. I want to know what you're thinking. Okay, so when, when through my ad, when the custom, when people, they click on learn more, they are taking to the landing page. Yes, when they click onto the landing page, they enter their name and the email and then they are taken to the thank you page. Yes. So the thank you page has a pixel, but the landing page doesn't have a pixel. Okay, you want them on, on, on both. So you want the pixel on the landing page as well. Okay.  And let's say, for example, it's this one right here and they click on learn more. When they click on learn more, and they come to your landing page. Because there's a very high percentage of people who are not opting in. And that's normal for everyone. So for me, for example, if I send, if I send 10 people to this landing page, most likely anywhere, anywhere from, from, from seven, anywhere from five to seven people are not going to opt in. So, so, and let's even say worst case, let's say out of out of 10 people that click learn more. 10 people come over here. Eight of them are not going to opt in. Two of them will. And that's good. That's fine. Two of them opt in. But what about the eight who didn't. We need to track that. Because those eight people who landed here, they obviously saw some interest. They read your texts. They saw the image. They saw the video. They clicked learn more. There was definitely some interest there. And so if I'm bringing people here and there's eight people who did not opt in, I want my Facebook pixel tracking them. And then I also want to have set up a custom audience. So when I go and run Facebook ad number two, I can retarget those eight people. I can retarget the custom audience. I could do. Okay, so you're leaving a lot of that's fine. My ads. No, no, no, that's not one that's not that's not a reason, but that's that's a very, very, very required and important step. We have to have a pixel on this landing page. We have to create a custom audience of everyone who is landing here. Right. And then, and then we have to retarget the people who land here, which is going to happen from creating a custom audience. So we're going to create a custom audience of everyone who lands here, but does not land on the thank you page. So, so we're going to create an audience around people who land on the landing page. We're also going to create a custom audience of everyone who lands on the thank you page. Because as I go and set up ad number two, I'm going to retarget all the people who land on the landing page, who did not land on the thank you page. I get you. Okay. Okay, now that's not that's not one that's not a reason why your ads are not working. I'm going to I'm going to answer that question. But this what I'm showing you is simply required. And it's the most important. I get you. I get you. I was going to say that this is this could be one of the reasons why my ad is not coming out of the learning stage. It's always learning or learning limited. Oh yeah, for sure, for sure, because Facebook, you have the ad, you have learned more. But Facebook doesn't really know you gave them the link you gave them the URL, right, you gave them the link to the to the to the landing page, but because the pixel is not on it. They're not tracking it and they don't they don't even know it. They don't even know what to what to who to send, because their pixels not on it. So the pixel has to be on this landing page for sure that's that's the most important. So write that down Facebook pixel on the landing page, create a custom audience of everyone who lands on the landing page, who does not land on the thank you page and in order to figure out you create a custom audience of everyone who actually lands on the landing page. Got it. So so I need to have a custom audience for everyone who lands here I need to have a custom audience who every of everyone who lands on the thank you page. And then, and then when I when I set up the custom audience of everyone who lands here. I'm also going to include people that land here, but did not land on the thank you on the landing page and I'll show you how to do that I'll show you how to do that. But, but that that's the most important that's number one so see, as you can add and here's what you want to do also which I don't you may already have this plugin, but you may not. You want to install this it's it's a free plugin. It's Facebook, Facebook pixel helper. You can install this just go in Google that Facebook pixel helper, and it's this first one right here. And all this is is a free plugin, it sits right here at the top right can you see up here in the top right. Yes, yeah. Okay, so this little shaded gray box. Right now, it's shaded gray. Facebook, this little, this little plugin. It's letting me know that this page that I'm on is not tracking me. Now Google's tracking me, but Facebook, there's no pixel on it. So, so, so Facebook is not tracking anyone who lands here. Now, this great, it's great. So when it's gray, there's no pixel on the page. And that's why when I look at other marketers, or I look at affiliate marketers, or network marketers, or I look at anyone building a business. I go to their website, or their landing page or whatever they're whatever they have. And I and and because I have the Facebook pixel plugin installed. When I go to their page, and it's gray. I know for a fact. Number one, they're not running Facebook ads, or number two, they don't have their pixel installed correctly. And that's the most important. So if I go to your page, this is going to be great. Now, if I go to my page, look at one of my landing pages, look what happens to the great. So right now it's great. And again, it's not just my landing page. If I go to any website on the internet, if they have a Facebook pixel, and they're tracking me, it's going to light up. So look, right now it's great. If I go to my website, fbadsforbeginners.com, look what happens to the to the to the little square. It turns blue. And now you see the number three. So that tells the person and it's public. So any anybody can see that, but not everybody installs the Facebook pixel helper, you know what I mean? It's only it's a marketing, it's a marketing, you know, thing. So people like you and I that are actually marketing.  I don't care about that at all. The reason why I have it is just so I can test my pages. Whenever I create a new landing page and whenever I, I, I install my pixel, I get to go to my page and this is going to light up. And if I click on it, there's my pixel. So it just allows me to look at it and say, okay, great, I, I installed my pixel correctly. Let's move on. Right. So so it gives me that information. And so that right there is my landing page. And so there's my pixel. That's, that's the most important. That's number one. Now here's, here's, here's number two. Here's the reason why your ad is not working. Now remember, I'm encouraging you, if you're looking to capture leads, which I'm assuming you are, you have a landing page, it's conversions. It's conversions. Now, now, now what you also what is what is really, really good as well is 99% of the time it's conversions, but the 1% can be can be really anything else. It could be engagement. It could be video views. That's really what it would be. It would be either engagement or video views. The 1%. I wouldn't really do messages. I wouldn't even do lead generation if you have your own landing page. The lead generation is only if you don't have your own landing page and you want to use Facebook's landing page. So I don't do lead generation. So, so, so 99% of the time I'm running a conversions to a, to a landing page. But the 1% I can either be running engagement, or I can be running video views. Now the only reason why I would do that, the 1%. The only reason why I would do video views or engagement is so I can create a custom audience. Everything is about custom audiences. I need you to understand that everything is about custom audiences. I need to create custom audiences of everything that's going on. So, so what I explained in the beginning is I need to create a custom audience of everyone who is landing here because I know for a fact, everyone who is landing here is not opting in. So I need to know that I need to know that. And so, and so the reason why I need to know that is so I can have a custom audience and I can retarget them in the future because if they landed here. There was some interest. They did not opt in for whatever reason maybe they were busy maybe they were driving. Maybe they had to go to dinner, whatever, but they landed here. And that's a good start. So if I can know who lands here and I can retarget them in the future with another ad, maybe, maybe my next ad will will entice them even more to get them back over here to finally opt in. So the custom audience of everyone landing here but the 1% the 1% is either engagement or video views. So what I do with that is engagement is this over here. So as we come across this ad, Facebook is tracking everything. They know that I stopped. They know that I'm reading it. They know that I click like they know that I share it. They know that I comment. All that is engagement. If I click on it. And I go over to the page. Look, I click on it. He's taking me to a landing page. Look at my pixel up here at the top right. It's gray. It's gray. It's gray. Now it turns blue. And now I see the number five. Right. So look, look, perfect example. I land here. He's tracking me with his Facebook pixel. In fact, he's tracking me with two pixels. So he's really not messing around. He's tracking me with pixels. He knows what he's doing because here's the reality. Yeah, here's the reality. I clicked on it. I came over here, but I did not opt in. Now what he has is data. He knows now he doesn't know me by name and phone number because that's private information. He doesn't have a lease, but he has the data. He knows that these amount of people landed. Now, if I scroll through my newsfeed tomorrow, if he's a smart marketer, which I'll assume he is. When I'm scrolling through my newsfeed tomorrow, or when I'm scrolling through my newsfeed next week, I should see another ad from him. Because he has my data. He knows I landed. Right. So here's my point on the on the 1%. Here's my point on the engagement and the video views. The reason why I would run that is so I can create a custom audience around everyone who engages or I can create a custom audience of everyone who watches my video. Now, it wasn't a lead capture. They didn't go to my landing page. They did not opt in. But that was data that I was willing to purchase. I spent $5 a day. I ran it for five days to capture the data. Now, why do I want the data? Because the data equals a warmer audience. Because if he can create an audience of those who click and come to his landing page, that is a warm audience. Now, what's going to happen is if I see his ad again, I'm going to be even more warmed up to him.  So maybe three weeks, four weeks down the line. He keeps creating custom audiences of everyone who's engaging. He creates custom audiences of everyone who's landing on his page. He creates custom audiences of everyone who's watching his videos. Because he has this data, guess what? I'm going to keep seeing his ad and eventually he's going to get me to opt in. Right. Got it. Does that make sense? Yeah, it does. So here's the next step. Here's why the conversions ad is not converting for you. Because you have to set up a custom conversion first. Now, the custom audience, as I bring somebody to the landing page, they click on the opt in and they come over here and they opt in. Okay. Then teach me Facebook ads. Then that person clicks the button and they come to the thank you page. This thank you page needs to be a custom audience because we need to know everyone who lands here. Now, this is what we call a bridge page. That's why you don't see a Facebook pixel on this because this is not my thank you page. It's actually a bridge page. Okay. So what I created is a double opt in. So when someone opts in, when they opt in, I don't want to take them straight to my thank you page. I want them to double opt in. So what I created is a bridge page. Thank you for subscribing. Just a few more steps to confirm your subscription. So what I, what I set up through cartridge is that lead will get an email and they have to confirm that subscription when they confirm the subscription, then I set up to the thank you page. I see.  I want solid leads. And that's why I created double opt in on some of my pages, not all of my pages, but some of them. And this one I created double opt in because because I rather have 100 serious leads that actually took a second step and double opted in versus 500 people who are just spamming me. Got it. That's the, that's the reason for a double opt in, but not everyone. I mean, you don't have to do a double opt in, but, but I did it on this one because this one for me, for example, was getting a lot of people to opt in and a lot of them were spam. I created a double opt in. But, but my point is, is that let's say, for example, this is the thank you page. Okay. Number one, the pixel has to be on it. Number one. And then number two, we have to take this and create a custom audience. So the link, we take the link we copy the link we don't need to copy the HTTPS. It's just the URL. Okay, it's just the URL. Copy the link. I'm going to copy that.  I'm going to click on events. Man, actually, I'm going to click on audiences. And I'm going to create the custom audience first. Okay, I'm going to create the custom audience. And here's here's how I'm going to do it. I click on audiences. And then I come over here. And this is going to be, and so all I'm going to do is I'm going to click on the blue button. And yours, I mean, yours might look a little different, but it should be pretty similar. Right. Does yours look like this? Or is it completely different? I have never come to audiences. I don't know. Okay, so, so in your, in your ads manager, does your ads manager look similar to mine? I will check on it right away. Because I know, I know in different countries, it looks a little different. So we went to audiences. Okay. Yes, so if yours looks like mine, then, then you'll, yeah, it says create a custom audience, create a look like audience and create a saved audience. Okay, perfect. So there are three options on that. Okay, perfect. So come back to my screen. Can you see my screen? Yeah, I can see your screen here. So what I did is I create, I copied the URL from the thank you page. Okay.  I click, I click on custom audience, not look alike, not saved. I need custom audience. I click on that one. Okay. And then what I'm going to do is it's going to be website. Now, that's why I showed you the 1%. 99% of the time it's conversion 1% could be either engagement or video or video views. And the reason for it is so I can then come in here and create a custom audience around all those who watch the video. Or, or I can create a custom audience of everyone of anyone who engaged in any post or maybe engaged in my Facebook page. So, so it's all about creating the custom audience. And in this case, I'm creating the audience around my website, whether it's the thank you page, whether it's the landing page, whether it's the sales funnel, whatever, it's the website. So, so, so now I click on website. Now I'm right here. So, I'm going to create a custom audience. I'm going to click this drop down. And it's going to be people who visited specific web pages. And all I'm going to do is paste in my URL. So now I'm creating a custom audience, and I'm going to, I'm going to name the audience. This is going to be leads leads. Let's, let's, let's name it, let's name it exactly what it is. Leads leads from FB ads for beginners. So this is my length, this is so this is my funnel, right, they go to the landing page, they opt in, they go to the thank you page. Whoever lands on this thank you page is a lead. So I'm telling that to Facebook. Okay. And I'm going to, I'm going to name it. So leads from FB ads for beginners. And so let me just, I don't get confused. I'll just put test. And then I create audience. That's number one. Okay, that right there is number one. That's done. So, so now we have an audience. And then also, I want an audience of everyone who lands on the landing page. So that, I'm going to come back over here.  I'm going to go people who visited a specific page. Now over here in the past 30 days, I can go up as high as 180. That's the max. Okay, Facebook can track back the last 180 days of everyone who landed on this specific URL. So that right there, and I'm going to call this one. FB ads for beginners. So David, this happens only when the pixel lights up on the landing page. Yeah, no, no, this is a custom audience. The pixel needs to be installed on the landing page as well. And that's what that's what's going to make the little square up here light up. But but but but it has nothing to do with the custom audience that that's going to light up because your pixel is installed. So that's step one. Step one is the pixel install the pixel step one step two is the custom audiences. Right. Okay. And then and then step three. And then that's all I got for you because that's that's the only thing you're missing. Step three is then the custom conversion, which will get to. So this right here is a custom audience of everyone who lands on the page FB ads for beginners, and I'll just name it landing, landing will go landing page views, because that's all it is. Here's what I'm going to do so here's so pay attention this is this is very very important right here and I'm recording this so you'll see this recording come out on YouTube. But here's what's so so important. Now, these are going to be people who land on the land on the landing page, but what but what we're going to create is everyone lands here, even the people who opt in. They everyone lands here. But what I'm going to set up in my custom audience is I'm also going to tell Facebook. Hey, this specific audience are people who only land here. And have not landed over here. Okay, so so here's what I'm going to do. So this is how we set it up. This is FB ads for beginners, landing page views only. And then I'm going to hit create audio actually know what here's what I'm going to do also. And then I'm going to exclude so right here, you see exclude people. So I'm going to exclude, I'm going to exclude so if I if I come right here to exclude now we're in the exclude. I'm going to exclude people who landed on the thank you page. Okay, so now I'm going to grab that URL again. I'm going to copy that. Now, I'm going to paste it in. So we have, we have a custom audience of everyone who lands here, but hasn't landed here. Right, those are people, those are the eight out of 10. You got you got 10 people landing on your learn more, you got 10 people clicking on learn more, but eight of them are not opting in. Right, this is how we track them. Right. So, so, so we have the link for the for the landing page. We titled it landing page views only because we exclude the thank you page URL. Right. So now we create audience, we create audience, we just click done don't worry about a lookalike audience you don't need that right now. The most important. So now we have two different audiences. Right. So here's what I can do now. As I go back to the ads manager. Now, as I go, go and set up my next ad. Here's what I can do. So if I come to the ads manager, let's click on, let's go to ads manager. And let's click on the green button, and we'll just go in and set up the next ad. And here's how you would do it. So we come here, we click on the green button, we click on create. Let's say, for example, I'm running, let's say I'm running a conversions ad. So, so I, I, yeah, let's just say we're running a conversions ad. So I'll give you that example. We come over here, we click conversions. Yeah. And then we scroll down. And then what I would do is I would just shade this to great campaign budget optimization, create a B test. You don't need to worry about that right now. Okay. And now here's where the magic happens. So as I'm in the second step, the ad set optimization for ad delivery, there's conversions, there's website. So now here's what I'm going to do if I scroll down. And now I come down to audiences. Here's where I'm going to include my custom audience. So now this is going to be, I click, I click inside the box, I go to custom audience. And now, see now I'm in a different, actually, let me go back to, I'm in a different ad account, but, but you would see it right here. So if I click on custom audience, actually, you know what, let me show you the real one, because let me go back. I think it was in this account that I created it in. That way you can really, really see it. Let's go conversions. I created it in this one, created the custom audiences in this one. Let me see. Okay, so I click over here, click continue. And then we come down to the audiences. And let me see if it's in this ad account. So here's audiences. And we want to look at. Okay, here we go. So scroll down audiences, custom audiences, I click. You hover over custom audiences. Actually, it wasn't this account. I forgot what account I created it in. Actually, was it in FBS for beginners? But, but either way, it would be the same thing. Right there you click and you hover over custom audiences and you would see that custom audience that you just set up. I think it might have been, it has been this, let me see if it's in this one. Okay, so audiences, click in there. And then let's see if it's in this ad account. Okay, so now what I can do is I'm going to target this audience. I want to target the people who landed on my page and viewed it only because those are people that are warm. Those are people that know of me. There was some interest there, but they haven't opted in yet. Right. Okay. Okay. Now here's what I also want to do. I want to target them for sure. And I want to exclude, I want to exclude my custom audience of leads. I got it. Okay. Because I don't want to keep running ads to the people who already opted in already opted in. Yeah. Does that make sense? It does. Yeah. Okay. That's the importance of custom audiences. Now here's the last one. Here's why your conversion ads have not been working. Now, as we come in here to the, we click on the three lines and we go to events manager. Here's the last step. Yeah. You have your landing page and you have your thank you page. Okay. The, the, the custom conversion is set up around the thank you page. So just like we did for the, for the, for the custom audience. We, we took the URL and we created a custom audience on the thank you page. We're going to take the URL. Everything after the forward slash. Okay. You don't need the HTTPS. You don't need the forward slash. It's everything after that. Okay. We're going to copy that. And we did it for the custom audience. We're going to also do it for custom conversions. Okay. And you can always type that word in up up here at the top, like custom conversions, pixel, custom audience. If you cannot find it in the dropdown and there's too much going on, you can always search it up here and you'll find it. Okay. So right here, it's custom conversions. Yeah. So I'm going to click on that.  This is the most important. Okay. So now I'm going to paste in that URL. And I'm going to give it a name. This is going to be FB ads. FB ads for beginners. Leads. Okay. Because this is, this is the thank you page. So, so, so what I'm telling Facebook is I need people and let's just name this FB ads for beginners. Thank you. Let's just name it. Thank you page. Okay. So remember that FB ads for beginners. Thank you page. And then if you're not selling of an actual product, you don't need to put in a price of value. You can if you want, if you're selling a physical product or, or you're selling any kind of product, if it's, if it's $10, $20, $100, that's where that value would go. But if you're not selling anything and we're only looking to capture leads, even if you are selling something, but if you're not selling something through this actual ad, you know what I mean? We're only capturing leads. So you can leave that part blank. So here's what we did. Custom conversion. Now we click create. And here's what we do next. So this, we click on done. Now. Here's what we're going to do. Now we have that custom conversion right down here. Okay. Yeah. So what I'm going to do is I'm going to now go to the actual setup. So I'm going to come over here and also make sure your Facebook pixel is on that. Thank you page, which it is. That's the most important. Because Facebook needs to have that pixel on their first. And that's why this right here is showing no activity. Because the pixel, even though I set up the custom conversion, the pixel is not tracking it because in this case for this example, I don't have a pixel on this page. So right here, we're going to see no activity, but it should be lit up green. And it should be ready to go because your pixel is on there first. Okay. Pixel goes on there first. So I'm going to click on the green button. And I'm going to click on the green button. Custom audience. Number three. Custom conversion. So now as I come over here. And set up my next conversions. Add. This is what I'm going to do with Facebook. So I'm going to come over here. I'm going to click on the green button. And I'm going to go to conversions. Because with a conversions. And a conversions add. Here's the difference. With traffic. You're, you're telling Facebook. Get people. To my landing page. I'm looking for traffic. Get people here. That's traffic. With conversions. Because I gave Facebook. The URL. Not only for a custom audience, but because also I gave Facebook the URL for my quote, unquote. Custom conversions. Now I'm telling Facebook. No, I do not just want traffic to my landing page. I want conversions to my thank you page. There's a difference there. Right. Yeah. Right. So now I come over here. I set up conversions. I click on conversions. I scroll down. And I click on the blue button. Make sure both of those are shaded gray. I click on the blue button. Now this is where it really happens. I can come over here and I can include or I can exclude. Any custom audiences. And what I would for sure do on every ad from this day forward is make sure that you exclude.  Right. And they come to the thank you page. Every ad from this point on what you're going to do is you're going to make sure that you exclude that, that audience from any, from seeing any future ads. Right. Because we don't, we don't want to show them ads. And it was this one right here was leads from FBS for beginners. So, so on this one, if this is ad number three, ad number four, I'm going to make sure that they don't see my ads no more because they've already opted in. They don't need to keep seeing my ads. Right. So, so here's, here's the custom conversion. As you come to this page, a lot of times you'll see this part read. Click in there. We scroll down. We scroll down. And there goes our custom conversion. FB ads for beginners. Thank you page. I'm going to click on that. And now I'm ready to go. So now I just continue and set up the ad however I want. And now I'm telling Facebook, Hey, I don't want people just to engage in my post. I do not want traffic to just come to my landing page. I want conversions. I am looking for people to get to my thank you page. Right. Yeah. That right there is the answer to your, to your. I got it. I got it. That makes sense. It makes a lot of sense now. Yes. Perfect. So what else you got for me? So, uh, uh, now, would you be able to have a look at my Facebook page, maybe on the next call, if you can do it with me one more time? Yes. You know what I have, I have 10 minutes right now. So if you want to, do you want to show it to me right now? Yeah, sure. Yeah. Go ahead and go ahead and share screen. Share screen. Do I have it? Start video. No, share screen. Yeah. Share screen. Host disabled participant and screen sharing. It says it's disabled screen sharing. Actually, you know what? Let me give me one sec. Let me check the settings. Okay. Try now. Yeah. Okay. Can you see my screen? Um, not yet. No, I cannot. Can you now? Um, no. Did you click on share screen at the bottom? Yeah, I did. Uh, yeah. Select a window application. Uh, that you want to share. I think you can see it now. It says screen to share. There we go. There we go. Perfect. Okay. So, um, this is my page. Perfect. Okay. So, uh, See, so look, even, even on your video, on your videos. What I would do with those videos is run video views. Okay. For maybe, maybe five days. Okay. Maybe, and it doesn't have to be a lot of money. Like maybe you do $3 a day. Okay. And do it for five days, maybe do it six, seven days, whatever, whatever budget you want, but you don't have to put a lot. But the, but the whole purpose of that, that's the 1%. So 1% of the time I'm running video view ads. So I can capture the audience. So I can create a custom audience around those who watch it. Right. And then next ad, I'm going to run conversions and I'm going to retarget everyone in that custom audience, everyone who watched the video. Okay. So just like we set up a custom audience for the website, the URL, you would set up a custom audience under video. Okay. I got it. So I'm just going to create a new video today. You know, this was not kind of giving me good results. I'm just going to create a new one today. And then I will start right from the scratch. So I just wanted to show you my landing page, how it goes. Okay. Perfect. You can click learn more on your, on your post. Okay. Visit our page. Yeah. So that was, was that a lead generation ad? That was a lead generation ad. Okay. So that, that's the Facebook form. So do you have a separate landing page? I do have a separate landing page here. And also is this, is this the landing page? Are you able to put a pickle on this landing page? Or is this the landing page your company gives you? This is just a landing page that my company gives me, but I think I will contact them today and see if they can have a pixel on here. So see, there's no pixel here, right? But if I enter my name here, it's going to take me to the thank you page and that will have a green pixel on it. Okay. I can't, I can't see it anymore. Oh, you can. No. Okay. Here we go. Perfect. Okay. Okay. So here's, here's what I would do because a lot of the times, because I'm very, I'm very familiar with direct sales and network marketing. Like I've been around it forever. And a lot of the times the websites that the company gives us, we cannot alter. We cannot edit. We cannot add our pixel. So if that's the case with your company, which I'm pretty sure it is, if you email them and say, Hey, I got these pixels. Can you add them to my pages? I don't think they are. Okay. That's number one. Now you could try, but, but I don't think they are. I don't think they're going to add your pixel. I think if anything, they have their pixel. Okay. Cause this is their webpage. So what I would easily do, but again, you would have to have another expense. But if it was, if it's really that important and you want to start capturing leads and you want to start running Facebook ads, which is 1000% how I would be building a network marketing business today. If I was back in network marketing 1000% I'd be running Facebook ads. I'd be running it to a landing page bottom line. I wouldn't even care to talk to anybody that I knew knowing what I know today with online marketing and Facebook advertising. I would for sure be running Facebook ads to a landing page, but that landing page is not the companies. The landing page is mine. Okay. And that's most important because I need to be able to control that page. I need to be able to add my pixel to that page. I'm the one running Facebook ads. Not there. Yeah. So this page can easily, easily be duplicated. Like I could, I could duplicate that page in literally three minutes with landing page software. Okay. So, so that's what I would do. Like I would for sure duplicate that. And, and, and as a person opts in, I'm sure they receive emails that follow, right? All those emails that the company probably already has set up for you. Like if I go to your page right now and I opt in, am I going to receive emails? You're going to receive emails. Yes. That's the company that the company has set up, right? Yes. Okay. Perfect. I would grab all those emails and put them into your own system. So for example, with me, I use Cartra. So I would have Cartra for my landing page. I would have Cartra for my email automation, but the company there to have all these emails. So I would simply copy every email that the company sends out and I'd paste it into my own email auto responder. And, and, and it can go out that way. And the only reason for it, the only reason for it, the absolute number one only reason why I would need to do that. It's so I can control the pixel on these pages. That's the only reason. So if I have to spend a hundred bucks a month for landing page software, I'm going to have to do that. Okay. So that's something you're going to have to consider for sure, because I'm willing to bet. You're not going to allow you to add your pixel. I will work with them today. I'm going to call them today. Let's see what response they give me and then I will contact you. Perfect. Maybe we'll get, maybe we'll get it going then from you then. Yes, for sure. For sure. I have, I mean, in any of my, my YouTube videos, there's my link in the description profit with Cartra.online. And that's the landing page software that I use. There's a 14 day trial. So you can just try it out for 14 days. But that's the software that I use for, for landing pages and for my email automation, but, but it's only, I tell people all the time, landing page software. Yes, it's $99 a month. So it's not exactly cheap, but it's only for people who are running Facebook ads or Instagram ads or YouTube ads. It landing page software in my opinion, it's only for people who are running paid traffic. Okay. If you're not running paid traffic, you don't need landing pages. Okay. If you are going to run paid traffic, which I highly encourage to do to build your business. Right. You need, you need your own landing page. Okay. And, and, and something like this. Oh my gosh, I could, I could help you build that in two minutes. Okay. No worries. Let me contact them today. I'm going to send them an email or call them today. As soon as I hear from them, you know, I'm sure the answer is going to be no. So, but then we'll get it going from you. Yes. Sounds good. Message me, message me. And then here's what you can do. If the answer is no, you can go to any of my videos, click on the link, profitwithculture.online and just set up a 14 day trial. You don't need to commit to it right now. Just set up a 14 day trial and then message me. And then our next call will be going over the, the landing pages and I can help you duplicate this in one minute. Sure. Sure. We'll do that. Okay. Yeah. Thank you. Thank you so much. God bless you. Thank you so much. Reach out to me soon. Okay. I'll talk to you soon. Thank you. Thank you so much. Thank you. Bye bye. Thank you.


Cool all right, so welcome to our talk everyone we we know it's in the last section of the day So we're between you and the booth crawl and Yeah, I know appreciate you everyone coming by so yes welcome to whose packet is it anyways? Awesome. Thanks Kevin. My name is Doug Jordan, and I work at Airbnb on our Istio based service mesh called air mesh I focus on extending the mesh to virtual machine based workloads TCP and other ingress use cases Previously at Microsoft I worked on our bare metal control point where we adopted Winkardy My handle on github Twitter and other socials is usually DWJ 300 and as you can see from the photo on the right. I enjoy cycling Climbing mountains and here's a photo of me doing both those Yeah, so my name is Kevin. I'm cooler. I am a software engineer at buoyant the creators of Winkardy I've been working there four years now and have worked on all the control plane components as well as the proxy My social handle on github and Twitter is K line cooler You can also reach me on the linkerd slack for any questions you may have after the talk or in watching this recording Thanks Kevin. Here's a quick agenda of what we'll be talking about today First Kevin will walk us through the how Specifically how a TCP packet gets routed through the mesh Then I'll discuss TCP debugging and walk you through a real-world example breaking out TCP dump and wire shark Cool So we'd like to start out with an overview on how a packet is routed in a service mesh So the things that we cover here will help lay the foundation for understanding some of the debugging steps Doug will take us through later. I'd also like to call out that we're going to try to keep this as service mesh generic as possible While both Doug and I have had a lot of experience with SDO and linkerd the concepts We talked about today are generally shared between the two as well as some other service meshes So this talk is ultimately about debugging traffic in a cluster with a service mesh So just want to make sure that we're on the same page about what a service mesh is and the common architecture of one that we Or you may be debugging So a mesh provides key properties today Those tend to be the four listed here observability for things like logs and metrics Routing things like traffic splitting and endpoint discovery security think mtls and authorization policies and reliability for example transparent retries of HTTP requests Circuit breaking things like that In order to provide these features the service mesh needs to intercept traffic into and out of the pod So how do we achieve this? So this leads us to the service mesh architecture here we see the sidecar proxy model This is the model that most meshes follow these days. It's worth noting because Some messes some meshes including the previous linkerd version ran a proxy per node In the sidecar proxy model each pod gets its own proxy container Inbound and outbound traffic can be redirected through this container, which is how some of those features I just discussed are implemented Multiple pods may be injected across multiple nodes and the grouping of all injected pods makes up what we call the data plane The second part to the sidecar proxy model is the control plane in linkerd This consists of the components that inform each of the proxies in the data plane Probably has a destination component used for routing decisions an identity component used for assigning TLS identities to the proxies and a policy component for determining who can talk to who Depending on the mesh there may be different components So the sidecar proxy model injects a proxy container into each pod Somehow inbound and outbound traffic from the other containers end up going through that proxy So in order to understand how it's happening. Let's take a look at what a container actually is So the first thing to know is that Linux doesn't actually have containers. It has namespaces namespaces partition the kernel resources such that different sets of processes see different sets of resources This means that each process or group of processes is associated with a namespace and can only see the resources within it the Isolated resources depend on the namespace and we'll see what some of those are next So here we see that by using namespaces. We can create a pod with multiple containers The blue boxes represent containers within the red bordered box pod The red bordered box represents a single network namespace There can be multiple network namespaces on a host which is how we end up with multiple pods on a node Looking at the blue boxes each process ID is associated with a network namespace With a network namespace and can be a container Each of those process IDs can be associated with for example a separate mount namespace So that they see their own file system so now that we have a Idea about like a higher level representation of a container and a pod in Linux We're going to build one up from scratch and focus on the parts Important for routing a packet. So first we start off with a host which We are referring to as a node and Next we create a network namespace on the host You'll recognize the red border box from the previous slides Again, this virtualizes the network resources so that processes using this namespace only see those network resources Not the ones on the host or other network namespaces Upon creation, we have the loopback interface for local traffic as well as a virtualized ETH0 interface For traffic into and out of the pod Additionally, we have a private set of IP addresses routing table socket listing connection tracking table firewall all the network related resources Then each container we create is a process that shares this view of the network resources You'll recognize the blue boxes from the previous slides And finally we return to the fact that we can create multiple network namespaces on a single host and end up with multiple pods on the node. I want to reinforce the idea that each network namespace has its own view of network resources This is highlighted by the fact that here we have four containers running Two of which bind to port 8080 and two of which bind to port 3306 These bindings don't conflict because they take place on separate pods Which means that they're in separate network namespaces Each pod is then also given an IP address that is reachable from other pods So in the last few sides, we've seen that within each network namespace I've been highlighting two specific resources with green boxes IP tables and sockets So why are our IP tables important for the stock? So well since IP tables are unique to the network namespace We know that all containers within that namespace observe the same IP tables configuration So they are responsible for the redirection of certain packets to the sidecar proxy Before arriving at an application container or leaving the pod So if a packet matches any of the configured rules They reroute that packet so that it is redirected to the proxy container But how are they actually doing this? So here we get into the fact that every IP packet has a header and we can see what that looks like here A packet header is the part of a packet that precedes its body Each row that we're looking at is a 32-bit word that encodes all sorts of addressing information for that packet This can include things like the total length of the header and data packet identifier and Most importantly for this talk the source and the destination I've highlighted the destination address with the red box Because this is the field in the packet header that IP tables is rewriting IP tables has determined based off some piece of information Say in this case the destination that this packet matches a rule that it is responsible for rewriting The destination to a different one The proxy in this case So returning back to this picture we can see now that instead of going directly to the original destination Inbound and outbound traffic first passes through the proxy container So inbound ends outbound traffic We have to consider the fact that the proxy will have different behavior Depending on the direction of traffic So therefore the proxy will actually bind a separate port for each direction So here the proxy container is going to bind port 4143 For inbound traffic and port 4140 for outbound traffic IP tables will have separate rules that rewrite the destination address fields To these ports depending on the direction of the packets Also note that these ports are not arbitrarily chosen. These are the actual ports that we use in think or D for example So this is great IP tables have done their job and the proxy container is now receiving traffic We have another issue though. Remember that the packets destination addresses were rewritten So even though the proxy is now receiving this traffic It still needs to ensure that it all ends up at the originally intended destinations This is where the other network resource I've been highlighting in previous slides comes into play socket tables So when a connection is opened we can examine the listing in socket tables that corresponds to that connection of The things tracked in each listing the one that we care about here is the original destination for that connection the proxy calls into libc and That gets sock up function and gets the original destination for that socket Using that we ensure that the traffic is going to the pre IP tables destination I've been talking about IP tables having rules that match traffic The mesh cares about and ensuring that destinations are rewritten The presence of these rules are also a responsibility of the mesh and there are few ways that these can be added to the pod The most common way to handle this is using an internet container an internet container Runs before any of the pods application containers start The mesh is responsible for injecting this in that container similar to how it's Responsible for injecting the proxy container The internet container will run to completion and add the necessary rules to IP tables for that pod If we're just adding rules though, why do we need a separate container? So rewriting IP tables requires elevated permissions a problem that Doug will Cover in his side of the talk And we don't want to give the long-running proxy elevated permissions for something that really is only going to happen when the container starts so an internet container helps solve this by separating the need for permissions between containers So the internet container is run and the application containers have started and now they observe the IP tables configuration required for the service mesh So therefore traffic that we care about is redirected to the proxy container So another way to solve this issue is to use a CNI plug-in Without getting into the details on this it ensures that each pod has properly configured IP tables One of my co-workers Alex gave a talk yesterday at service mesh con about how Lincordies CNI plug-in is implemented If you're interested in that I would definitely recommend checking out the recording on that awesome Thanks Kevin Now that we understand a bit more about how packets actually flow through a service mesh I'm going to walk through how to debug a real TCP stream. I Encountered the need to capture TCP packets when debugging an issue with our Istio based service mesh at Airbnb called air mesh specifically within our custom metadata exchange For this talk, we're going to rely on a few key assumptions Note that they aren't requirements per se, but they will make talking about and visualizing the underlying networking model much simpler First is that you're using a CNI container networking interface such that each pod has a unique IP address from the directly from the VPC that's routable from other pods on the network for instance the AWS VPC CNI Second we use it will be using the cryo container runtime, but all the commands shown commands shown today could easily be modified for container D Last and probably most importantly having SSH specifically route access to the nodes We'll discuss how to avoid this requirement at the end of the talk The use case that brought about this debugging involved talking to patchy Kafka So I'd like to start with some quick context about what Kafka is and a little bit about how it works for those who may be unfamiliar Kafka is a distributed messaging queue at a very basic level a client can either produce a message to a topic or consume one Internally Kafka consists of brokers, which are just instances of the service topics, which are like categories or feeds and Partitions which are well, you know partitions of the data within a topic Kafka uses zookeeper to run leader election at the partition level So every partition will only have one leader shown here on the slide in green When a producer wants to send a message to a topic they'll first compute the partition ID Often using a consistent hash function Then it will use its internal metadata of the cluster state to write that message to a broker that contains the leader of that partition When a consumer consumes that message It's actually pulling the Kafka broker for recent messages on that particular partition Now that we know a bit more about Kafka. We need to explain where our service mesh fits into the picture to help us trace some packets In order to send producer consume requests Kafka client Kafka's client first needs to discover the initial state of the world ie all the brokers topics and partitions To do this it uses a special request called metadata request This request happens on client initialization and can be routed to any broker in the cluster as they all share the same view of the world Thanks to zookeeper Once the clients have this information it will then route requests to specific brokers on based on topics In our case We'll use our service mesh for that initial metadata request because we don't actually care which broker responds to it And we just want to route it to a healthy node something that the service mesh is really good at After that all subsequent publish or consume requests will go directly to the specific brokers thus skipping the service mesh entirely This is because Kafka's client wants to be in control of exactly which broker it's talking to as it knows the internal mapping of partitions to topics The issue we encountered was during this initial metadata request. So in order to reproduce the issue. Let's use Kafka cat Kafka cat is a CLI used to talk to Kafka and is incredibly useful when you don't want to spin up a heavyweight JVM Just to check the state of the cluster Here we use to control exact to exactly to run a command in a pod in this case a pod called pod test against our app container We'll specify the following arguments to Kafka cat minus L for metadata listing and then minus B to specify the broker We want to talk to in this case. It'll be the address of our Istio service or our service mesh service and called Kafka dot service on port 1992 which is Kafka's default TCP listen port And we get this error right fatal error at some line of metadata list Broker transport failure what does that mean well after a bunch of googling and lots of code spulking through Kafka's code base All I can find out is that the request is malformed But what does that tell me? Isn't our service mesh supposed to give us all this rich observability? Well, yes and no For HTTP the telemetry emitted by service mesh is incredibly useful. We have response codes Logs and even response flags in the case of envoy As shown in this slide, there's 25 unique flags for HTTP requests that are emitted in the logs when things go wrong But for TCP connections, we only have seven flags that are extremely generic Without response codes or detailed response flags. It can be really difficult to find the exact cause of some of these issues So what do we do? We break out our favorite tool TCP dump and Then look at packages in our shark But how do we do this in the context of Kubernetes specifically Kubernetes with the service mesh? We're going to take a look at two different packet captures The first one is what the client Kafka cat is actually seeing and the second one is what the proxy is seeing As Kevin explained earlier all the containers in a pod share a same single network namespace so They use IP tables to rewrite packets destined for meshed services to the proxies inbound port on the loopback interface So to capture what Kafka cat is actually seeing we can just TCP dump on that loopback interface Once we have that capture we can take a second one This time to see what the sidecar proxy sees by running TCP dump against the host on the virtual interface for the pod We'll talk more about that later Let's start out simple. We'll just run TCP dump in the main container We'll run kube control exact against our pod same one pod test and we'll run it against our application container case app And we get another executable file not found in path. Well, that's obvious, right? We don't include TCP dump in our container images We do this for variety of reasons. Hopefully these are kind of obvious, but things like we run distro as images We'd want a minimal security footprint and in general. We don't include debugging utilities in our images So that we can reduce our security surface So let's just install it, right? If it's a VM just the app update app install and call it a day But it's not that simple doing this in Kubernetes at runtime in the container would require both root and a package manager So it's probably just easier to bake it into the image of build time In order to bring in extra dependencies to our main container image We've gone ahead and modified our deployments back to run an additional sidecar container We'll use the net shoot image which already has TCP dump installed As a reminder all containers share that single network native space So we can just run TCP dump in any of those sidecar containers to capture traffic on the loop back interface and observe everything that's going on in the pod But again, it still doesn't work You don't have permission to capture on that device. What gives? Hmm turns out it's because we're running our containers as non-rate users Using non privileged containers is becoming more and more common and to be clear. This is a good thing But how do we capture on an interface without it? We have two options We could add net admin and net raw security capabilities to our pod security context But that would be less secure and doesn't really help with the just-in-time access model If we only need temporary access, this would be too heavy-handed Alternatively if we have or can get pseudo sh access to the nodes we can take a package capture directly from there So let's go do that Okay, but how do we actually do that? Many of you folks know how to do this But as a reminder we first are going to get the node name from the pod object We'll run coup control get pod And specify that this JSON path to get the node name in this case the pod that we're trying to look at is running on node one Okay Now that we know the nodes name. Let's get its IP address We use coup control get node with node one and then this JSON path to get its internal IP address and Note that in our situation We want to get the internal IP address because it's accessible on our corporate VPN as opposed to say maybe a public IP address Which may not be While we're here. Oh, sorry, and the the node is running internally on our 192.168 1.9 network So let's use that While we're here. We're also going to grab the container ID and I'll explain why we need that in a second This can be any container in the pod, but since we know we have our sidecar proxy injected We'll just use that one So we'll run coup control get pod pass this to jq Look at the container statuses and finally select the one whose name is proxy and then get its container ID kind of long command And we see our container ID is cryo colon slash flash 9 4 a d dot dot dot which makes sense right we're using cryo Then lastly, what's a stage to that internal IP? awesome as Kevin mentioned earlier each pod has unique net NS So in order to run tcp dump within it, we need to first find it Now that we're on the node and have the container ID We can get the network namespace by using cri CTL, which is the COI used to talk to the cryo container runtime so we'll run cryo inspect on the container ID and specifically Select the namespace of type networks path using jq Note this namespaces slice has all the namespaces not just network right the mount the pit the user etc But we'll get the network namespace in this case. It looks like var run NS and then 1 4 9 some on good Now that we have this network namespace. Let's run a quick sanity check If we run if config from inside this containers network namespace It should see the same IP as the pods IP right because ultimately it is the pod So our pods running on 100 dot 116 dot 95 dot 102 so We'll see if we can see that same IP address To run a process inside a particular namespace of any type not just network. We can use the NS enter command So here we'll run NS enter with the minus minus net argument and Specify our network namespace that var run NS and then some on good And look if we look at the output we see the I net address of 100 dot 116 dot 95 dot 102 Which is our pods IP address So we know we're on the right path now that we found and verified the network namespace We can finally run TCP dump the moment. We've always been waiting for Again, we'll use NS enter to enter the network namespace and run TCP dump with the following options Minus I for loop back because again, we want to capture a traffic between the application container Kafka cut and the proxy. We'll use s0 to capture all packet sizes We'll use NN To not resolve port or host names And lastly, we'll use minus w capture dot pcap to write it to a file pretty standard With this TCP dump we can actually open a wire shark. So here it is And we're going to look for something very specific. We want to make sure we observe this metadata exchange. I Don't have time to go into the full wire protocol of Kafka nor do I fully understand it But if you we look at the first packet after the standard TCP handshake number 137 We can see the client ID in the body's point text Rd Kafka This is the default client ID said by LibRD Kafka, which is the underlying library that Kafka cat is using Note that the reason we can actually see this body in plain text is because the capture is taken between the proxy and the application Next we'll look at capturing data from the external side of the proxy So if we want to observe the external traffic, which is encrypted between pods We can use We need to know which interface to listen on if we want to avoid getting hit by a fire hose of packets Ie we could capture everything on the host, but it will be pretty hard to find the signal in the noise Since we're using the VPC CNI There is a route on the host routing table for each pod IP to the virtual interface created for that pod So we can use IP route show to look at the host routing table and just grab for the pod IP as We can see the route for our pods IP address the one ending in dot one or two is using the virtual interface EMI blah blah blah ending in zero nine seven Using that We can finally run TCP dump on the host to capture traffic outside the proxy. This is again traffic. That's Transiting the pods boundary. We use the same TCP dump command as before except this time We'll listen on this interface instead of loop back and we won't won't run it inside the network namespace so we run Just vanilla TCP dump minus I with the ENI the virtual interface ending in zero nine seven and this other the rest of the options Are the same as before? As you can see we've captured five hundred seventy nine packets I won't open this PCAP up now as the traffic is encrypted using the proxies to a certificate and won't be very useful to us I'll leave it as an exercise for the attendee to figure out how to get the certificate needed to decrypt this traffic Spoiler you can use the debug port of your proxy After going through all these hoops and hurdles just to take a simple pack of capture There must be a better way and the good news is there is ephemeral containers are a relatively new feature of Kubernetes that allow launching a new container that shares some of the Linux namespaces of The running pod in our case. We want to share the network namespace, which is done by default There is one major limitation, however They currently do not support changing the security context of a running pod So if you launch a pod with a restrictive security policy that prevents net raw or net admin You still can't use ephemeral containers to run TCP dump But for argument's sake, let's say we don't have that Restrictive security policy. How would we use ephemeral containers assuming the security context allowed it? Well the coop control and debug command will attach a new container to the running pod using a provided image So we can just run coop control debug pod against our pod again pod test and Pass in a custom image in this case. We'll use net shoot again as we know it has TCP dump installed Same argument supplies before and we can capture packets Okay, so ephemeral containers are cool and all but it doesn't help us get around this core issue of not being able to Set a different security context within an ephemeral container I have good news. I was kind of lying before this issue is actually just recently been fixed upstream And we'll be coming to a coop control debug version coop control debug command in a future version Going forward. I hope that we can integrate ephemeral containers into other open source tools like case sniff a coop control plug-in That runs TCP dump and use a wire shark to capture remote to start remote captures on pods All right, so for quick recap Thanks. Yeah, so Linux doesn't have containers. It has namespaces These isolate certain resources from each other The network namespace Isolates the network resources So when multiple containers run in a pod, they all observe the same network resources a p-tables rewrite the packet headers In order for traffic to be redirected to the proxy IP tables will rewrite The destination part of the packet header and then the proxy looks at the socket table Since the destinations on the packets were rewritten by IP tables The proxy needs to ensure that the original address is used So we can look that up in the socket table, which has a corresponding entry for the connection And on the debugging side TCP observability is extremely limited outside of things like response flags To capture traffic within a pod you can use NSN to run the host and run TCP dump on with back To capture traffic leaving the pod you can run TCP on the host TCP dump on the host against this virtual interface for the pod and Lastly, ephemeral containers are here to save us from all this friction If this work is interesting to you or you're passionate about kubernetes or service meshes At Airbnb, we're actively hiring. Please go to Airbnb comm slash careers to learn more Cool. And yeah, if you enjoyed the presentation buoyant runs a service mesh academy You can sign up for these are monthly hands-on workshops for real-world production users and it is free and We also work on a service mesh as a service Product that manages linker D for you on your own clusters. You can book a demo or come find us at the buoyant booth Yeah, we'd love to talk Well, that was a great session if you


Let's get started. More talk this session on UX, UX and the Fire. Can everyone hear me? Okay, good. So the other room I was in the microphone was really bad, so I'm just joking. Right, so if you wait for the last stragglers, you have to show up. So what made before we start? So the topic here is UX and the Fire. It's essentially a lot of criticism that UX comes under due to a lot of mis-conceptions that exit to our UX. So my name is Jacob. You may know me as one of the people behind Node 1. I was one of its founders. Nowadays I'm actually working, I'm actually on my own again and I'm building a company called Strait, which is a distributed digital agency. And here are my complex details. So it's a strategic digital agency, a distributed agency. And we do things differently from other companies because it's something I'm actually working on right now. All right, so you can check out the website in case you're interested. So UX is a lot of bad rap from a lot of people, because they don't understand it. And there are a lot of myths. So I think it's time to bust it. And one of the things of a network UX is that it's just like this kind of dialogue. Like what engineer made, can you please let them color and make it pretty? I even have a friend that was called in to approach like this and he declined. They were like, yeah, we're working this, this looks good. So we just want to make it present to both of our CEOs. It's like, oh, that's not the way you should do it. So to understand why there's a problem, why this is wrong, let's go back to the roots of why UX was invented in the first place. So I'm going to take some famous examples of porting to UX. So in Europe, we have an airline called Ryanair, maybe you heard of it. And Ryanair, they have tried to, they're basically a bad cop of Southwest. So they, essentially, they're cheap, but they're not good in any way, including their usability. I don't fly Ryanair as a one principle, but that's mostly because they treat their cabin crew, you know, badly and I think that's the way it's suggested. But something to do is when you actually buy a ticket to the website, you get the option of choosing insurance. And there's no option that says, I don't want, I don't want any insurance. Instead you have to hit it in the list of countries you can live in. So there, don't cover me, it's a country apparently. It's a new member of the European Union, yeah, according to Ryanair. Here's an elevator somewhere in Mexico or Spain. And those of you who speak Spanish understand why this is very confusing. This is a very much a weird world example, but still. I would, I would have any idea, you know, which button to push if I was in the elevator. And then there's, I don't know how the subway menus look here, but these would look very sweet. And the first time I saw this menu, I'm like, what? Like, do they have a menu with just a, with just a sob and one with a, one with a soda and one with a chips? No. You're supposed to read them, like, you're supposed to read them like, this looks good on everyone and then just read these parts. But those of you that know, like Gestalt psychology, and other things that are lined up and grouped are seen as belonging together, that's what your brain sees. And here's a really famous example from XKCD. The Venn diagram of what people look for at a university website and the information that usually is there. And like I said, full name of school, check. But in e-commerce, poor usability is not just a nuisance. It's actually, it means loss of revenue. I like, Jacob Nielsen once said, like, increase sales by 100% by fixing usability, small fixes on the e-commerce side. So if your customers are vendors, online vendors, running from our sites, then they have a lot of good reasons to actually pay you to do some e-commerce work for them. So do you think these are problems that can be resolved just by some colorites? Well, of course not. So what is Ux? If it's not a Muslim colorite, it's not one thing. For me, Ux is a number of practices and methods. So in UX, you are to understand the user needs, translate the needs into requirements, design, integrate solutions, and evaluate those solutions intuitively. But I would like to see it as an umbrella expression. So in the user experience, we have a number of activities. We have information design, usability engineering, interaction design, information architecture, experience design, graph design, which we use it with these phones here, you know. They will come back later in presentation. So this is how I see what UX is about. So let's ask Google, you know, like, what do other people think? Apparently this guy, Patrick Marseille, UX includes HTML, Excel, slash JSON. So what are you doing? What are you doing? I want to raise... Well, according to the Japanese information architect, which I would honestly doubt the information architecture efficiency with all this stuff, like... Yeah, it's pretty, but, you know, it doesn't make me an advisor. And apparently at this company, you also did customer service. So if you're looking for a job somewhere, ask them to draw the kind of BIN diagram of the UX specificity that might evolve. And at this company, you're not the only one doing human computer direction. There's someone else having SA2. And chances are, this is what looks like your office. So there's clearly a lot of confusion here. So let's find this. For these things here, they are the activities. We do. These are the sets of tools that we do carry out in our work. And then there's the qualities that we achieve that we produce in the products. So how do these things relate? Well, at the core, we have the usability goals. Things like efficiency, you know, easy to learn, have good utility. And those in turn lead to probably having other things, like being supportive of creativity, meaning and helpful in motivating. So the one great use experience requires good usability. So if it's easy to remember how to use, which means it's probably motivating and satisfying at the same time. Can you think of a product that is not easy to use but still motivating? Well, so aren't. Games are hard. We play them because they're hard. But they're hard in the right way. If you struggle with the user interface or for a game to watch, you will use it. Now, you want to be beaten by the gorilla, like throwing logs at you, not by the button. You don't like doing what you think it should. So we have all these tools, and they allow us to do disability and great use experiences. So here comes that. We also implement solutions to solve this problem. We also give the design work, we actually build prototypes, and we do user interfaces. So, you know, like, what does it matter? It's going to work anyway. Well, use doesn't concern or something looks. It concerns how it works. And that's why in Sweden, we have the expression for an evening, which means a given form or a shape or something. People use it to change the way we design. What I like about design is that you design a solution. You just don't give something an appearance or something. You actually, you design how it works. And this is critical, that you can get enormous business value by making sure something works in a pleasing way. And one way to do this is to look at this pyramid here. So, ranging from tasks to experiences, you can look at different solutions. They were from functional to meaningful. So let's map some products here to see how they, this is just my subjective understanding of things. These here is where you start. And some companies actually cast by this line here, the cast in here. And some actually are meaningful. These are meaningful and just pleasurable to use. Things to use just because they're fun to use them. Like you can imagine the flicking emotion on your phone when you flick that less than your iPhone or Android and it keeps the momentum. It also keeps rolling. Or the way that I can roll over the screen that in fact it's entirely flawless. There's no slowness or anything. And I remember I said this to a friend. He had an Android phone, an early generation, he said, look how they look at stuttering, you know, when you're scrolling the screen. Look at my iPhone, it goes smoothly. That's just the same thing. It doesn't matter. But it does indeed. Because the way you perceive something to work is how you think it works. So if it seems to work well, you believe it works well and you're going to trust it more. So look at some companies here. I would say they end up somewhere here between useful and reliable products. Ubuntu, the books products, they're actually getting better and better. The latest version of Ubuntu is actually on as you cross the class, you might say. Microsoft, even better. Not really great, but they're really spending a lot of time doing UI. Google, I like Google. They are, I mean, look as fancy as some stuff, but it works. And of course, Apple. But that's the brand. They made the design, which was meaningful and useful, one for design to be their brand. So let's just apply these tools we have. So let's look at an example. Google Analytics. So when they built this product, they applied a lot of these tools in a way. So let's look at which of the tools actually had an impact on design. So information design is a skill and practice that prepares information people understand, like graphs and diagrams. All about traffic to the site. Experience design is how it feels, like what kind of emotions do you give the rights to the user in the usage? And it also has to do with using colors and things that engage the user. Information architecture, it's about how you stretch information, how you tag content, text, on-race, navigation, menus. Here's build engineering. It has to be the small details. And use build engineering is often about what we call user research, when you actually move with something pixel or you change the order of something and you look at what the impact has in this experience. Interaction design has to do with behavior. They probably thread a lot of different versions. These are dropped down or tapped before they decided to go with exactly that kind of solution. And graphic design, the overall look and feel, the gradient here and the colors and so on. But there are many that still, despite all these tools, don't make it. 82% of all the products are by their buyers, considered unsuccessful. So let's break those down and have a look at what it is that way. So 33% of the products are casted for completion. They're gone so out of the wrong that they give up trying to finish them. And 25% are delivered on time budget, but the result's not what's expected. I'm going to argue that results is not a matter of whether the project is exactly to scope or requirements where they're actually delivered to the business market. I'm going to do that in a bit. And 25% exceeded the budget. But they got everything they wanted. And the reasons for this won't, they may. Very often lack of user input involvement, it builds something to actually solve people's problems in a way that can help them. Requirements being incomplete. Requirements being clarified later on over the course of the project. I don't really see expectations. The customer, you know, like the customer having an idea of what they were buying, but it was not what you were able to deliver. So it's a massive waste. I mean, but we can fix this. You know, we have the tools, and we have the tools to identify problems. We did find the needs and design for them, but we also need to have the tools to understand. And I'm going to show you this tool. And this tool is called Effect Mapping. Effect Mapping relies on two different assumptions. First was that every site website is built for a reason, and to meet a goal. There are certain business values related to building this website or system. The goals can only be achieved if people are using the websites. If they're not being used, there's no effect coming out of it. It has the advantage of providing a map of very much a helicopter view of the different of the goals and of the stakeholders involved. It's called an Effect Mapping. It looks like this. So we're going to break down the map now and see where it consists in parts. But the core here is the actual effect. And finding that effect is the first thing you need to do. And it can be kind of hard because it needs to be a lot of thinking and the customer to think differently about like the products that you should do. And I call this the walk of why. So because it comes to you and say, hey, we need a better website. Or like, okay. But why? Well, it's really hard to find out who we are and there's no way to post comments and feel involved. Like, post a comment, okay, but why do you need that? Well, a big share of our customers want to feel involved, okay? Well, we need to reach those customers in order to challenge more sales to our site. Uh-huh. That's what they actually hope to achieve. They want to challenge more sales to their site. That was not what they told us. They wanted a better website. But what better meant? That's what you find out actually by asking why, why, why repeatedly. I'm not recommending you to ask why, why, why. I'm asking you to involve in a more meaningful dialogue with your customer. But the core is to understand the motivation and the person holding money, why they're actually giving you that money to build this site. So, okay. So we know what, what do we have to do to reach those customers in order to challenge more sales to our site? Well, we're going to write that as an effect. An effect needs to be precise. Should be more than a few sentences. Um, it needs to be measurable and it should be long-term. So if this is not something that's going to happen overnight, it's going to happen over a month or even longer. Alright, so we formulate the effect and then we write the number of ways to measure it. And as you can see, these are all highly quantified. It's not opinion. It's actually, if you see this effect in your patterns, in your boom and lift count, then we have succeeded. Um, and there's also a limiter and that allows you to basically look back of six months and see what's working, what's not. Let's revise, let's improve, let's change. And here it goes, in the blue box. And right and left of it, you have the goals specified. Alright, so we have the effect, we have the goals. We need to understand that users are. But these are the ones, if they're not using websites, just not going to get an effect out of it. Do you need to understand what motivates them and what do they expect from your faculty? So, first of all, who are they? Well, to find out who they are, you can use these for engineering. Um, again we start with the observation and interviews to establish personas. Are you familiar with personas? Can you use maths? Okay, good. So, alright, you're going to deal with personas. So personas are, personas are really good, not just for you, but for the designers and for developers too, because they can actually relate to what it says in the persona. They're not designing for nobody, they're actually designing for a person. You can give a person a name, you can call him James, because he's 20 years old, because his work is in the account, for example. You know that, you know, in spread time, we're two sports, we can bear with friends. Okay, kind of stereotypical, but still like, you're like, would James ever like this? Would he actually think this button was sort of like, would I help him? And you can start involving a dialogue with that person. I would even recommend you to print out these personas and tap into the walls in your office. And that allows us to answer these questions. What goals do they have? What tasks do they need to carry out? And what are these issues? And then we can melt you. I've done a very, sort of, a lot of research on how to divide people into these roles, because these roles are often not about where they work in the company, but what kind of behavior they have. What's it like, information seekers, information retrievers, information publishers, the curious ones, and so on. Try to think more about what kind of behavior, how they are as people, more than where they happen to work. So the next step here is to figure out how they conceptualize information they need. How will they be able to find the information? And here's where information architecture comes in place. And using card sorting and tree testing, you can actually, you can see, you can find the most logical way to divide your content, the way to present it, you know, in terms of menus, and doing interviews and sit-ups and so on. And that in turn lets us answer even more questions about these users now. Okay, so we know a bit about who they are and what kind of information they're looking for and how they expect that information to be provided and organized. So the next thing we need to know is, what are the goals? Like why do they need this information? And here's where experience design, experience design, interaction design, and graphic design come in. We can do wireframes, we can do stifle hard to catch a feel on the racial structure of the site. We can do mock-ups, storyboards, and journeymaps, build scenarios. Are you familiar with journeymaps? So something that's really exciting recently is a discipline of design called service design. So regularly in the business, you think in channels, someone does a print channel, someone does the web channel. When you service a sign, you go above that and you look at the customer's experience through all these different channels. You need to sign, regardless of channel, find solutions which are the best. You need to visualize the way a customer would interface with their company because of gold and journeymap. All right. So how do we measure this? We try the goals that the users have and then we put up a way to measure them. So we do the overall effect. But now we can look at each target group, what they expect, and how well we meet their expectations or sort of that need. And this is the most... Now, what you have now is that you have to find what the customer expects. You have to find who they need to engage and what they are expected from the website. There's just one more component now in order to actually have a full, complete... Basically, they basically have the foundation for a project, for the requirements list, and that is actually tasks. That's the next step. So, like you see it, now we'll put the goals out here and we also made them measurable. We also translated them into tasks. So in order to... I want to, for example, this user here wants to replace paper forms with online forms. What to do that needs to be able to create forms of communication requests to make it possible for users to digital sign forms. So you know what functions it's expecting. Now you can design the user interface, the mock-ups and everything to serve that need, given its personas, given this person's expectations. So, as you can see, using research, user research and design, we can actually convert business, we can actually create business value and make requirements that help deliver that business value. This is not at all about slapping color on something. It's actually about making sure what we're building actually serves its purpose. Without this work, we're shutting blind. So how can we help people understand this, that this is actually what you're just about? That criticism of UX is misplaced. But now I understand what you're just about to do. The pivotal role that UX has in modeling art. I dare say that many of the failures of IT projects can be avoided. And I think we have to do this. I mean, we can't actually get away from the consistent failure of actually building solutions that meet people's needs without actually thinking this way. The resistance to UX as a practice is a resistance to working or my dissolution generally. Companies are hiring people, companies are hiring more and more UX people. It's evident even the big ones that deliver all the systems that big corporations use in order to do all the internal stuff that you're doing, from accounting to all the modeling, all the process everything, can understand that poor usability is really stopping them from getting the results they expected. So it's better money.  Well, one way to do this also is usually called the UX value proposition. And this is a way to visualize the impact of some UX work. It's highly subjective though, so it's not objective anyway it means, but it gives you a visual idea of where you basically, where it's worthwhile to put your efforts. So it works such a way that you break down your project in the number of goals. You want to migrate users from big and modest stores to online self-service. These are customers that have big and modest store amounts to convert to billing. You can recite to reach more customers and maybe eventually shut down the physical store. Then when we release those, then we actually break them down into the UX attributes of influencers. So what influencers will use the ability to have, for example, on this goal, for example. What we then do is that we map them out in this list. This can be used in a regular spreadsheet. And then we rate the current state of it and then we rate the potential state of it. So when it may be suffered, what could experience potential be like? This should be conserved to guests in your park. What could it potentially help them? Great. So gain potential costs investment. Essentially gain potential costs investment. In the end, you'll have a sheet where you can pretty visually see where here's where I can actually make an impact video. So it's all good, right? But then I'll say, oh, yeah, this isn't going to pay, you know. But do you know in the real case where this really made a difference? And yes, I do. So here comes the $300 million button. I mean, yeah, that's the right amount of serial, yeah. And this is actually something Jerry Explode wrote for him. This guy was actually a keynote speaker at the last Rupert County in the U.S. I wasn't attending that one. His name is the primary right now. But he's written an entire book about web forms. And he wanted to start a shop where bad web forms can really have an impact. So it's a lot to the e- retailer. And they have a ticket form. It looks like this. You can buy Hodgepodge. You can say how much you want. And you can see the mantle and how fine. All right, nothing fancy. The problem is they try to be smart. So they add this function. You have to register before you can take out. And then I get the service, you know, like, wow, it takes so much time, you know. It's a big gift in a year, so it just takes a few minutes. All right, good, they think. So Susie, she wants some Hodgepodge. And she's inside, and she's ordered some Hodgepodge. And she's like, seriously, what do I have to fill this in? I don't know. I don't want to get involved in a relationship with these guys. And she's like, I might have ordered it before, you know. Let me go back and check. In fact, she has ordered tons of times. She's a big fan of iPodder, so she has all these iPodder in spite of email addresses. The thing is she has constantly forgotten her name and she has changed email address. So it has like four entries in the database. Eventually she's like, never mind. It's not worth it. Somehow they actually find out. The people run and say, oh, right, this is a problem. So like, maybe it wasn't so smart to put that registration form there anyway. Maybe it was so smart to force people to register. So let's try and remove it and see what happens. So they remove the register button and replace it with S. You can continue without, or you can log in. And results were pretty massive. And the 5% registry in sales by that simple change. But that was not a change in engineer realized. That was not a change that the designer saw was something that built a professional, analyzed the workflow, and talked to users, and analyzed the result. So there's massive games to be made in the UX. And UX isn't important. So a takeaway is, well, UX isn't just iMac and other Sonicay peers, it deals with how something works. And UX links business goals to requirements requirements in a way very few of these things can. It offers toolbox and methods to help us do these translation business goals to requirements, and it's proven, and there are 300,000 cases. Why does? So if you like the session, please say evaluated. I would appreciate your feedback, and thank you so much. All right. Sorry. Any questions? Do you have any hands here? Yeah. Hi. Sorry. So I work at post-secondary in Lethbridge, Alberta. And I am curious how you would deal with people who have gathered data, but you have a set of your own. But there might be conflicts because some of it may be biased, especially people who have worked with the audience for quite some time, but they may not have noticed some of the trends that you've been gathering on your own, which is an unfortunate case of miscommunication. But so I guess my question is, how would you deal with my data versus your data? And if there was bias there, how would you deal with that? Okay. So the customer has been doing user research on their own? Yeah. It's not a case of recruitment, but I find they're not asking the why. It's more like a therapy session, I guess. Well, I guess, sorry. I just find that they're not asking the why is behind some of the reasons for why people do things. They're just, it's more like a case of how do you feel, but they don't really explore that. And you get handed this information as some kind of something you could use as sort of like a... Like usually a survey or if they're in a focus room. I see. All right. And they handed you and like, yeah, look at this. You're going to help you make a better design, for example. Yeah. But I would go back to those users and maybe I'll talk to all them but talk to some that are representative and try and have interviews with them. You usually need to speak to so many people. And the interesting thing is that if you read Steve Krug's book, I think it's not the one... Don't make me think it's another one about useful testing. And it's like, most of the useful mistakes, the biggest ones can be discovered by anyone. You don't need to have someone who's a perfect, you know, target group you can find. And it's the same in this case. You don't need to talk all the way. You need to talk to a few that do think are representative of the group. And that will help you validate your ideas regarding what they are, actually, the motivations for feeling and thinking the way they do. Okay. That's great. Thanks. Any more questions? Hi. I was wondering how do you handle feature creep in projects when it comes to UX design? If you have clients that through the mock-up process or the wireframe process keep adding things, wouldn't it be nice if our users could save their favorite something, try and handle that in the mock-up phase and then say no more features? Could you comment on that? Well, feature creep usually was a problem. Later on, we're trying to build the project and fix the line and you actually had a fixed scope. I previously, just before this session, which I have a test on the PMBuff, which I recommend you guys to attend after this one, where we're going to discuss these things even more. So I think feature creep is the biggest problem is during the implementation process. That's when it's really scary. But even as a designer, when it comes to changing it, you have to, design is holistic. Like, you have to sign one thing in detail in isolation. And even though it's good to try and achieve modular design, it's not always realistic. I would try and discuss things with customers relevant to the customer. Basically, trying to get these things up is to change like in the wireframe stage and try to get all this feedback out in the wireframe stage and have them understand if they're going to change, make changes later, it's going to look good. It's going to take time. And try to have some meaningful dialogue and not just go from basically what they say to read the mock-up and try to increase the... You can actually go from a very rough wireframe that once you can make for the tool called mockingbird.com or things called GoMockingbird. It's called... So it's an online mock-up tool. It makes very rough wireframes. And then you can take one of those and maybe move it into Illustrator and make it even find the add fonts and the colors so you can collect feedback over several stages. And then maybe you can catch some of those requests much sooner when the change is much easier. Great, thank you. More questions? We have plenty of time, so... All right, well, if you have more questions, you can just ask them on Twitter or you can come look me up. I'll be around here tomorrow too. So thank you for attending. Very good questions, right?