You guys welcome back to today's module in this video you guys I'm gonna simply show you how I send out broadcast email This is like real day in the life In the trenches Send out emails, right? This is part of my email marketing. This is part of my overall marketing This is what I truly believe every single one of you guys need to be implementing Whether you're a real estate agent or an insurance agent or a local plumber, whatever my whole my my whole whole Passion really is just teaching local business owners how to really dominate right how to take over their local area, right? And even if you're building nationwide if you're building nationwide same exact thing as we're in this this the scale of social media, right? As as social media has really taken off and exploded 2023 and beyond It's obviously important that that us as business owners are on social media But the number one number one goal for every single business owner It doesn't from from small to to freakin extra large The number one goal is to is to eventually take that attention or take that individual take a piece of that audience Off of the platform get them to opt in to your world and that happens with a free download That happens with a free PDF a free e-book a free training a free case study a free video a free Anything right? That's where it starts.  They enter their name and email and opt in That should be the number one goal for every single business owner on Social media you have to be able to take that person off eventually, okay? And get them into your world and that happens with simple email marketing that happens with simple Email domination. This is the basics, right? So so as I take an individual off off the platform again Whether they came from a listing funnel an open house funnel a free download funnel Whatever what whatever may be right there they're often in Once I have that person on my list number one, they're gonna be on a list, right? That's number one. They opt in they enter their name and email into a form They're now on a quote-unquote list, right? That list nine out of ten times. In fact, I'll just show you Here's the list. So I use cartridge for everything right you guys So if you haven't activated a 14 a trial, I don't know what you're waiting for Because all these email software is out there you guys I've been using cartridge since 2018 It's what I've taught on this entire time. It's what I recommend still to this day 2023 and beyond I'm still recommending cartridge and it's really because of the price point really the price point and obviously the software the technology behind it Because I'm gonna fill yet. I'm on different conference calls with cartridge Executives with cart with the cartridge board members with the owners of cartridge And again, I've been promoting it hard course since 2018 all my clients. It really came First because of the tools but because of the price point in the very very beginning It was the price point that got me and I'm gonna tell you the truth I'm still on that price point, which is 99 bucks a month a hundred bucks a month That runs my entire products my digital products. So when I charge clients, right? So so if you're charging clients Some of you guys are getting this our real estate agents. So you're not you're not charging clients, right? But if you're a coach a consultant a trainer anything like that, right? Selling digital products selling an e-book selling a course whatever I run all my products all my transactions through cartridge I run my email automation through cartridge, which is my email autoresponder I run my pages all of my landing pages every page I could possibly need to build for anything Simple templates. I run all that through pages. The form is what they opt into The membership is what you guys are in right now. You guys are in my membership course I'm able to create this course provide value monetize it. It's all hosted on cartridge So I'm here using it for many many different reasons, right? But but but but in this case with email automation, this is the most important This is the backbone to every single business, right? So so as I get people to opt in what I want to be able to do is is yeah Number one put them on a list number two put them on a sequence. The sequence is the automation So once you're opted in you're on a list. That's one thing. The second thing is now the automation That's what what cartridge call is the sequence, right? So so as you opt in today tomorrow, you get an automated email two days later You get an automated email a day later. You get an automated email three days later I can set the days for whatever I want They'll get an email later and these are preset pre-written emails that I would have put together, right pre and then boom I just love the sequence and and and in a way Automation goes so so powerful you guys because again as you're building your brand on social media That's where people are seeing you right whether it's a YouTube channel that you're creating Whether you're just simply posting on Facebook. You're simply posting on Instagram That's really the only time your audience sees you they need to see you again They need to see you again That's why video form is so so powerful YouTube channel is so so powerful because okay if you found me on Instagram That's one thing you're seeing my feed you found me on Facebook. We're friends on Facebook. That's one thing You're you're you're seeing my feed um But but but but but but but but but but we're out we're out to my showing up, right? We're also my showing up. That's why video you'll be seeing me on YouTube right my video That's another touch point. So all of these different things in business are touch points to the audience I'm gonna touch you on Facebook.  They're on a list Then you want to tie it to a sequence so you have a handful of emails that are preset ready to go out automatically, right And then whenever you randomly just simply I don't know in your business You're following because you're you're you're you're the authority in your industry You're the one staying educated with everything, right? Because that that's the value you eventually start to start to give to your to your potential clients and And prospects, right? You're always staying educated. So if you're real estate, you're always staying educated You're probably going to conferences meetings. You're probably affiliated with with with nar You're probably affiliated with you're probably a realtor, right? So now you're going to to annual events through the through through the realtor association, right? So so you're always staying up on on on on game, right? Let's just put it that way. You're always staying educated, right? And and that's that gives you the the the leverage To to attract, you know more more business, right? So in this case, for example Something happens in your in your industry news flash headline news alert, whatever or or or or as as easy as as what i'm doing today What i'm doing today you guys as i'm training you at the same time is i'm setting up a broad a broadcast So now we're going to go into broadcast Broadcast is a one-off email. So whenever something randomly happens in some news. I want to share it with a certain list um Or maybe my whole database whatever I can come in here and send send a broadcast email So no matter what email platform you're using whatever crm you're using Um, you should have the capability of broadcast email sequences is your automation, right? Your preset emails But the broadcast is a one-off, right? So today Again day in the life. I'm training you. I'm preaching. Um, um, um, um, um, um, what does it go?  Because I just released a youtube video So what I want to do with that youtube video is I want to send out a broadcast I could for sure put it in a sequence and have it go out at the end of the sequence Let's say I have 17 emails that are already preset in that sequence. Okay today's youtube video could be Email number 18 in that sequence, right? So one email is going out every three days, for example, and I have 17 ready And and now i'm now I have this new youtube video So i'm gonna i'm gonna simply add it and it'll be number 18 in that sequence It's like dude that could be that that's good. That's good. Obviously that's super super good That's why you want to make your content evergreen because because if a lead ops in today There are 17 more emails that they're going to be getting every three days before they even get to this youtube video Right in this case with the youtube video So Yes, I want to add it to my sequence, but I want people to see it now Like I want people to see it today. I want this email to to reach certain people You know with within the next, you know a few minutes type of thing. That's a broadcast. That's a broadcast Okay, and so this this is what you're going to always want to send out broadcast like probably like at least maybe once a week Right if you're out there really working and grinding and creating content and just continually Continually sharp sharpening your sword and and getting better in your industry You're gonna always have news to share You're gonna always have have have more content and more value to bring to to your audience and again What do most people do they'll just take that and post on linkedin They'll take that and post it on facebook.   You got to really know know know the value and and and so the broadcast you guys the broadcast I'm gonna always continue to educate my audience and and I'm gonna always try to send out a broadcast At least at least once a week because something always new is evolving in in in any industry And and that's good content that's stuff you want to share So rather than just simply posting it like you guys are organically doing now Start building a list start building a list Run ads And if you're posting organically make sure you're you're posting links to your landing page where you can capture the lead It's all about the lead you guys. I can't stress that enough. Okay, so so here in today I just released a youtube video literally just just An hour ago really actually yeah not even yeah an hour ago And I want to promote this to it to a handful of people on on certain lists, right? So here's how I do it I click on communications to the left and this is how I do it in cartridge. So again, if you haven't activated your 14 a trial There's a button below Activate a 14 a trial because this software at the price point I'm teaching how to use the entire thing and if you're in the real estate space You've gotten all those funnels for free the listing funnel the open house funnel the seller guide funnel the buyer list funnel The cma funnel, right? So so that's a whole added value right there. I just saved you literally a few thousand By just giving you those funnels. So jump on the software with me activate it This is one major major important component within it. And that's the the communications. This is your email marketing Okay, so I'm going to click on communications to the left.  Okay, so now all over here. These are all just past broadcasts that I've done, right? So Over here, I'm going to click on the green one and I'm going to name this broadcast The name of this broadcast is simply what whatever I want to name it, right? Like like this is regarding a youtube video I'm going to be blasting out right now to an email getting people to my youtube video This is this is crush it. This is crush it crush it with facebook Video ads and no one sees this part. This is just internal I'm just I'm just giving it a name, right? So and then the category that I'm going to put this in is going to be the fb ads cheat sheet Yeah, I'll just throw it in that one. Um for now. That's fine and then create Okay, so so now I'm going to go through the the basic setup. This is step one send by email. That's the email. It's coming from Being sent from and then if that lead were to apply They're replying to that email and I can change these out too. I can change it I could put whatever but it has to be a business email It has to be a business email This cannot be your gmail or or yahoo or whatever like this has to be your your your your business email Okay, so I'm going to hit save and then save a next and then here is where I can I can I can This this is why I love cartridge you guys. That's why shout out to Russell Brunson Shout out to click funnels Shout out to all these other kajabi and and softwares, but but but cartridge is so simple It's so freaking powerful and at the price point. It's yet to be beat like it's yet to be beat So so that's why I I continue to use it. I'm an ab I'm an avid user of it. Um, I'm an advocate. I I promote it.  Click the button below Click the link below and and and activate it. Okay, and I'll and I'll help you with all this stuff So in this case with the recipients, this is where I really narrowed in now I base a lot off of tags. I'm always trying to tag a person So whatever funnel I create the moment they opt in they're tagged, right? So so that allows me to really filter future emails And and and for sure filter broadcast emails. Everything is based on a tag Or I can go everyone subscribe to a list It's there's all these different criterias all these different ones. So in this case, I'm going to simply put down Has tag I'm going to base everything off of my tag, right? So has tag has tag fb ads cheat sheet Okay, um fb ads cheat sheet. I'm going to put The people that are going to receive this email. They have that tag Or or or they are subscribed. I'll do actually I'll do another tag or has tag Clients I'll go with this other this other client list or they have that tag right or they have that tag um And and also when it yeah, or or they have that tag and then and then is there any other one? Um I'll leave that I'll leave that that this is and here's the thing to you guys You'll see some people that have lists that have thousands and thousands of people on the list Some people will brag that they got that they got 10,000 plus, you know emails on a list What what they're not saying is the fact that x amount and a good x amount a good percentage of that 10,000 lists, for example are unsubscribes and and and and undeliverable Where where the email was wrong, for example, right? And so people like that tend to not even Account for that and not even clean their list I clean my list like once so once a week like anyone that unsubscribed I take them off Take them out completely take them out of my system. So it doesn't go against my number, right? Um If they unsubscribe I take them out if it's a if it's a bounced email So they put in the wrong email. Um, I take it out. So that number doesn't reflect So my my number man my small list is freaking powerful. My open rate is powerful These are people who really want to hear from me, right? Um, they're engaging they're opening They're opening my emails, right? They click on my emails. So you always want to scrub your email list Get rid of all the bad emails in there. Um, and and only focus in on that small list You don't need tens of thousands. Yes for sure. You'll get there for sure and and and yeah for sure But as long as you clean them and and and stay engaged with this email That's why you want to broadcast at least once a week broadcast once a week This is my once a week right here that I'm doing with you guys and I'm just turning into a training video. Um But but this is what you want to do. So so I'm going to send it to this this highly engaged smaller list People who have downloaded this recently And in recent recent clients of mine. Um, not full-blown coaching clients That's one small list of just just random basic clients. I should say Not random, but basic clients. I have different levels in my in my coaching So that's my that's my level one client list and I'm going to go save a next That's it save a next and then I'm going to hit single email And then all I'm going to do is simply go to rich html I want to work out of rich html in here. I could choose all these different templates These are all different templates that carts already has available. So so I can literally just take any of these templates click Drag it over drop it there and then boom work off of this template. I can change everything about it the color That's not my yellow. Then boom. I want to change it to freaking pink. Whatever, right? I can save colors I can create palettes, right? I can link that button somewhere Right. So so if I want to link it somewhere or if I want to just completely take the button out I could completely take the button out, right? I can change out this image right here and put a completely different image I can link the image. Um, if I don't want the one two three, I could just delete it, right? Just delete it, right? If I don't need this this bottom part right here. I'll just delete it, right? So what I what I normally do is all come in here on the very very very first email and I'll and I'll completely optimize my template, right? I'm going to change out the copyrights. I'm going to change out the the the social icons If I don't need phone number, I'm going to I'm going to take that out, right? So so I'm going to optimize it and fully fully deck it out. I can create my own I can I can click drag it over and just simply create my own um Or I can yeah out of all of these different ones like I can add a countdown clock I can get super super creative with it you guys, right? I can I can I can Put a countdown clock and I can base that countdown clock off of um off of off of a fixed day Off of a date fixed day and time fixed date and time Based on a due date. So I can even base it off of a tag, right? If they have a tag Then then then then make this countdown clock countdown, right? So so I can get super super fancy with you guys But but but also keep it simple at the same time So once I really get my template, right? Once I get my template, then I would save once I create my template I'm going to do it one time change out the blue Maybe add maybe maybe create a different color whatever man, right? I can I can do whatever I want to this template and then what I would do is I would save it Save as template, right? That's what I would do. I would save as template now in this case what I'm going to do I'm going to just get rid of this real quick Because I already have a simple template and my my template is basic But but the reason why you want to simply create a template one time and then save it Is so it saves you all the time in the future This is my basic basic template Let me drag click drag it over. There's my tech spot There's my button to where if I want to link someone to and then maybe I'll put a picture down here Um, in fact, I have the youtube thumbnail picture. So all I'm going to do is I'm going to put that here That's the youtube thumbnail picture. So boom that goes right there And then what I'm going to do is I'm going to link that video. So I'm going to go to that video Um, let me switch out accounts real quick So let's go to this one And then right here. So this video and this is what I want you guys doing. So if you guys or if you guys are um Let me switch account again If you guys are building a youtube channel, this is what you want to broadcast often, right?  Let me let me grab today's video obviously in in the sequence, but The broadcast too, man get more attention To the the the video the moment it's released, right? That's a broadcast. So boom. I'm going to share I'm going to get that link. That's all I did there. I got the link and then I'm going to bring it over here And I'm going to link the image I'm going to link the image And then I'm also going to link this button right here and I'm going to change out the color I'm going to I'm going to go down to my youtube red And and all I did is I simply Chose the color and then I hit the three dots create palette. That's how you can create it, right?   That's it And then boom this button That right there is is the link That's the link and then I'm going to write out the text and then I'll even link my little logo down here I'll even link that To the youtube video. So now I'm going to just simply write out some text And we'll be on our way. Give me one second Okay, you guys just wrote out something super simple and basic and here's another thing too you guys when it comes to these emails Authenticity is the most important Communicating with leads as if they were your friends is the most important, right? No need to be super corporate. No need to sound too Company like no no need to to sound fake like keep it real like treat these people Like like you're texting your friend. That's that's just my professional opinion.  We we are attracted to to people, right? We are attracted to to to socializing, right? That's what we gravitate to we are social beings So that's that that's why social media is so freaking Just magical is because we're connecting with people. That's all it is We're connecting with people being human beings connect with human beings bottom line And and so as you're bringing people into your into your world, right?  Talk to them as as if they were your your your your pal, right? And and that's what I what I try to do and I try to keep it short, right? Not too long There's some marketers that will really write stories in this email, right? they'll they'll write freaking 17 paragraphs and it's and and and They work they they actually work they convert they work But but but in my opinion as a consumer I like shorter and go with based on what you kind of prefer, right? This is your business You want to tailor it towards the audience for sure, but also you're a consumer So how do you like things done, right? How do you how do you read your emails, right? For example, you know, so in my case when when I get emails that are super lengthy I don't read them like I don't read them and they could be coming from those credible Marketer in my case they could be coming from the most credible business person, right? But but sometimes I don't even read them And if I do it I'll maybe read it two weeks later when I get freaking five minutes to spare because it's going to take five minutes to read it, right? But but many emails come through me come come come to me from other marketers that that I Right like that. I really look up to and aspire to be right people that I'm learning from just like you guys Learned from me. I'm learning from others as well And and I have emails that come through that are short and sweet and to the point and and and the whole objective is to give me to click Right. That's the whole objective. So rather than getting me lost in your freaking 19 paragraph story Make it short and sweet and get me to click get me to click and what I'm encouraging all you guys to do 2023 and beyond I'm encouraging all of you to start putting videos on youtube start putting videos on youtube put videos on facebook for sure Um, if you're doing rills on ig. Okay, cool. But but for sure on youtube start building a youtube channel So in this case what I do in my a lot of my broadcast email is that's all I'm trying to do is get attention to that video Right away that and it keeps it that's and I keep it that simple. It is that basic When it comes to my my part of the back end of my marketing, right? So here quick little short to the point email It's all about getting them to click. It's all about getting them to watch here So i'm going to hyperlink this word watch here And i'm going to simply come down to Let me just click it again. Here goes this right here and i'm going to link There is the youtube video watch here, right? And that's it and I'll and I might vote. Yeah, it's already bolded. So we're good there And then And that is it you guys so we got this hyperlinked, right? We got the button linked We got the image linked. We even got the logo link, right? So so again Find find your template and these are just dynamic variables all use their first name a lot you guys these are dynamic variables So right here in here you got we got all the templates where we can build a template But up here we have the dynamic variables all use their first name, you know a lot I'll use it in the subject line. I'll use it in the body of the text the body of the email Let's see how I could put it up here in the subject line jack This is how you dominate This is how you dominate a local Area, okay. This is how you dominate a local area. That's it. That's it That's it. That's simple. So, okay. So now again, if it was my first template I would save as template and be able to use that over and over and over again, right? So I put their first name in the subject line. I put their first name down here. Hi jack Um, I put their first name down here start targeting your local area jack and dominate, right? Um, it can happen. It can it can all happen. It can It can it can all happen It can all happen With simple facebook video ads here's how to set it up. Watch here. Okay. That's it. That's it. That's it. Okay, cool And then we're going to go save and next save and next you guys and then We don't need any any automation right now. So we don't we don't need to do any of that schedule So I can I can save as a draft. I can schedule for later. I can send now I'm going to send this bad boy now save and next Yes, and that's it you guys Congratulations one two three fourteen now what I like what I what I really really love about carter as well is the fact that I can track All my leads I can track the activity. I can track the click the the engagement the open rate I can track all that in real time. So right here. I'm not positing pausing the video. I'm not not editing this part at all Um, we are literally in real time right now. This is live where I'm going to simply a few seconds later I don't even think it's been 60 yet, right?  I can go to broadcast And I can last 30 days. I can go today and boom look at that for opened it already, right out of 314 for opened it already Um, and that's right here. So show me right here. Oh, even show me who sent me to spam Right. Um, it'll show me who unsubscribes and people will unsubscribe. That's totally fine Um, but but four of them opened it that quick, right? So so I don't really micromanage this back end part like the unsubscribes for sure because and even the spam So if someone reports me to spam, um, I'll take them. I'll completely take them out of my database, right? That's just how I do it, right? Some people will leak will will will keep that list or keep that email But it's like no do that person reported to spam. They obviously don't like your stuff or or they unsubscribed They obviously don't like your stuff or they put in a bad email.  I need the most most most highest engaged most loyal followers To be receiving these emails, right? That's who I want and that's another thing you guys another misconception about social media And just marketing in general the fact that we think we need millions and millions of people following us No, we don't no, we don't like look at my youtube channel My youtube channel still not even in the tens of thousands of subscribers at this point 2023 I'm still cracking 5000, you know subscribers on youtube But do you know how do you know how blessed I've been with with just that amount? Like I've gotten so many clients shout out to every single one of you They're going through this training right now that actually found me from youtube, right? And that's all organic It's it's video content, which i'm telling you right now is magical You got to be creating video content on your subject right on your business um No matter how you look no matter what you sound like no matter your accent No matter where you're from none of that matters. None of that matters. There's a piece for everyone That's why you don't need millions of people. You don't need even tens of thousands of leads You need a few engaged one highly engaged ones, right? And I truly believe most of you guys that are getting this this video right now Most of you guys that are going through this training I I I I'm willing to bet that it doesn't take many many clients for you to make a good income Right for you to build a bit a build a big business So so don't worry about having tons over here focus on the ones that you do have treat every single one Like like like you're like they're they're your favorite one Right and and really just bring value bring value and people will stay on your list you guys people will engage even more And and you'll be you'll be closing deals and and building building relationships and in clients seven clicks Look at that dude. Look at that. Actually, that's the whole that's the whole 30 days Let me go back to today.  Again, I haven't paused the video here. You guys we're in real time 22 people. Look at that the open rate seven percent so far and it hasn't even been five minutes It has not even been five minutes. It's been like two Right. So so look at that one unsubscribed and that's totally fine. Dude. Here's what I do.  I would delete the lead and completely take it out and that's it. No big deal. All right So so you guys that's how you send out broadcast emails You want to be engaging you want to be communicating with your leads and this is how you're going to do it You're going to do it by by by by building the value on social media, right building the value putting out content and really educating your audience And then driving them to a click driving them to a landing page where you can get them to opt in That's where where it starts you guys. So I cannot encourage it enough Enough when you're running ads when you're running ads, it's all about capturing the lead You want to capture the lead because this is going to be another touch point that you'll be able to educate that person Build a relationship with that person Um, and eventually do business with that person. This is the back end. This is email marketing So if you haven't activated your 14 day trial, there's a button below and I'll see you guys in the next module


Hey, hey, this is Carlos. I'm the founder and CEO at product school and today I'm here with Caitlyn Quinlan, who is the general manager at Gainsight. Hey, Caitlyn. Hi, Carlos. Thanks for having me. Good to have you on the show. Love having people from your company like CEO Nick. I want to continue digging deeper into the conversation because you guys are building something really cool that has an interesting angle that connects customer experience, customer success with product. I haven't seen too much of that yet. So but before we get into that, I also want to learn more about you. General manager. That's a that's a big title. So how did you get started? Yeah, so I've been saying for years, like, let me give you the abridged version of my background. I feel like as the years go on, it just gets longer and longer. So I'll do my best to keep it short. But I will go back to the beginning. I started my career on the East Coast of financial services, doing something completely different and investment due diligence for private equity companies and VC clients. And I noticed in 2009, 2010, there was like a huge shift on the deals that we were working on and everything was software. So concurrently, my partner Matt and I moved back to California. He is going to get his MBA and I just started to get the foam of Silicon Valley startup life. So as I looked around, you know, at those clients, private equity companies, what are they investing in? And to me, like one factor emerged to be very prominent, which is if I'm going to go to a company, I really need to believe in the product. And the custom, the company's success is going to lie in that durable value of their product. So I made a jump to an operating role at it. What was then a startup of MongoDB and far and away, like the coolest thing for me was going from financial services where everyone had my same job and same role. We were all kind of doing the same work to working across all the functions with completely different disciplines. And it was my first time ever working in the weeds with product and engineering and sales leaders. So now in total, I've been at four software companies. MongoDB, Mixpanel, Nova, which is a spin out of DreamWorks and then arrived at Gainsight where I am now and had done throughout that time pushing to learn more about each part of it. Organizations function. So I've held various roles across the functions and leadership teams which have led me to my now current position, which is the GM for the PX product line at Gainsight. So for those of you that don't know, PX is what product and customer success leaders use to create the best possible experience for their customers in their software. And I'll just have to make a quick plug for on product school. We do have our micro certification for product like both using PX. So check it out. Yes. I was also looking at your LinkedIn profile. I saw you had no nine different roles at Gainsight. And that's really cool. Well, first of all, even before that, it's not a lot of investors turn operators. I've seen a lot of operators turn investors. So that's definitely like an interesting move. But then once you turn operator, you've been wearing so many different hats before you became a general manager. So maybe you can guide us a little more through that journey, especially at Gainsight. Yeah, totally. So I joined Gainsight focused on sales strategy and operations, which is a role I'd held at Mixpanel prior. And I think what drove the sort of like multiple roles and moving to different parts of the organization was one, of course, curiosity. But a lot of it was following the customer journey. So I started out really focused on just sales. And then as like the classic go to market story, there was so much of a relationship to marketing that it quickly bridged to be like, OK, it's not just sales analytic strategy. It's got to include marketing and marketing ops and that stuff. So it becomes revenue operations. And then, of course, Gainsight, you know, very focused on like customer and the customer journey. So, OK, well, you can't really just think about pre-sales. So then you've got to look at what happens after you close the deal and soon adding in CS analytics and strategy and putting this strategy together so that all of the functions are operating in the same way on the same cadence and executing on the same plan can really make or break the success of a product line. And so that's where I started to pick up other things. Like I started focusing on the strategy in particular for just the one product in PX because I was like, OK, I can follow decision making. The way that Nick phrased it to me was I need to know that when one team makes a decision in one function, there's somebody that cares about the results everywhere else in the organization. And so just sort of built up from there until I found myself here running this product line. The way I've seen your your career evolve within Gainsight is similar to the way I've seen product-led companies evolve. They typically started with a more of a sales-led approach. And at some point they started looking at, OK, but what drives the sale? And it's typically an excellent product experience. And then eventually it's like, OK, but what is driving an excellent product experience is actually engagement and usage. The users need to love it first, right? And that creating that motion obviously leads to sales. But it's really it's been interesting. And I haven't seen many sales leaders that are able to kind of branch out of sales to start getting a bigger scope for for an entire business unit. Yeah, I think you're so right. I, you know, for a long time, I think a lot of people thought as of PLG as and just sort of like a premium conversion from a free trial. And it was very new business focus and like very just on the the marketing or presale side. But it's been so true for me and especially like learning at Gainsight as the company has grown that a lot of PLG is focused on your post sale, what your users are using in your product and how they achieve their outcomes and grow with your product and find more parts of it. And then as your company grows and becomes multi product, that's what I call like the second funnel after the sale for cross-selling expansion. Your product is both the most effective and the most efficient version of selling that secondary product. And if you optimize the experience with good digital engagement, that is going to be the way that your customers interact and grow with you. And in a way, your product is very meta because one of the first movers are the evangelist in product-led growth. At the same time, you're building a product for product people. So so what is your your day today as a general manager? Yeah, and so sort of like zooming back out as the GM of PX, I'm responsible for setting the strategy of the product line. So this guides how our go to market and our teams go to market with PX. But it's, you know, that sounds like it's just from the sales and marketing perspective, but it's also about the roadmap for PX and how it's set and maintained with product and engineering. So a lot of my day-to-day can be unblocking decisions or managing, you know, sort of like escalated. How do we address this new thing that's come up? But basically, it's like I was sort of saying, keeping all the functions aligned around the same strategy so that we're operating towards our goals and the most effectively that we can. And, you know, do spend a fair amount of time with customers and prospects understand their first hand feedback and experiences. Which to your point on PX being very meta just sets the bar really high for how our own experience of in-app engagements and how people self-serve through a knowledge base, just how they interact with our product and how they make that journey easy. Because I like to say, no matter how hard you try and no matter how many human connections you make by default, your product is the most contact that your users and customers have with your company. So it's got to be a good experience. So as a GM, do you own profit and losses? So yeah, so there is a, I mostly focused on the growth of the product line, which is just the nature of the cycle that it's in. But I spend a fair amount of time looking at our full P&L, so profit and loss statement, for just that product line. So we definitely balance, like as we think about where we'll grow, we're balancing how the Cogs line up to that, for sure. It's a little bit, you'll have to tell me, it's my first sort of almost scratching the surface of what it's like to be a CEO running a mini company within a company. Well, I was just going to say, a lot of people used to define a Rova product manager as a mini CEO and kind of disagree. Actually, I think a GM is more like a CEO, mini CEO in a way, like you are the CEO of a specific business unit. I've seen a lot of product leaders become GMs. And usually the difference is that at some, at the GM level, you own P&L, product leaders don't typically own it. But I'm also curious to know, how is your relationship with your, with your product leader? How do you divide and conquer? Yeah, extremely close. And not just one product leader, but the whole like set of product leaders. It's one, I think it's just important to make sure that the strategy is aligned with the roadmap. So in this case, Maxime Oceanov is our product leader. And so his team owns like the full roadmap, right? But the way that we prioritize what gets built and what we work on is through like a really collaborative relationship with my, my role and the other general managers. So which is great because that allows us to really heavily influence the strategy, but always be operating in the same way. And there's continual sort of check-in. So based on this development that happened, how does that influence or not influence that they change in our, in our strategy? And I'll say that I think working with people who are product managers or product leaders is, is really helpful in this role just because the GM role tends to be extremely cross-functional. And you have a lot of stakeholders and there's probably not a lot of other roles in the organization that from the very first level, you're starting out with so many stakeholders across the business, like a PM has. I mean, like the PMs and audience can tell me, like, do you have loud stakeholders that all want different things? How do you reconcile what everybody wants? I feel like that's a very similar skill set. So it's always really great to be collaborating with that team. It sounds very similar. I always say it's, it's really a mindset more than a specific title. And in your case, like, just at a very high level, like, what is the status of the company?  How many products do you, do you have? Yeah. Um, so I lead the product experience product line, but overall, Gainsight is a platform with three primary products and growing. Um, so we have our customer success product, our digital hub, and then product experience, which all sit within a platform to have a cohesive customer experience. Um, uh, yeah, suite of products. Is that what you meant? Or did you have an additional pricing? So the follow up question is, okay, so you have a platform product that is supporting additional products. So I'm just thinking about your roadmaps and your interactions with a different product leaders. Like, as you think about growth, um, how do you ensure that there is some consistency, uh, to just to make sure that the way you are growing your different product lines, uh, it's, it's, uh, and their specific strategy. Yeah. Um, so I think we have some structural things in place. And then for me, I have some sort of like methodology or approach things that are in place. So structurally, um, each of the GMs, we work really closely together to make sure that we have an aligned prioritization. We understand based on how the business is doing, where the business is investing and growing, where do all of our, um, product roadmap investments fit relative to each other? So we are very aligned. So when we have those conversations with product leaders, we are very in lockstep with what the rank of prioritization is. So it's very collaborative in that way and, um, organized, but I think more than that for me, the approach and methodology is what are the use cases that our customers are buying for and more specifically, what are the outcomes they're trying to achieve and how do those relate to not only my product, but also the other products. And for that, you have to look at the full suite of products, not just your own, because a customer, when they come to you is not buying one product in a silo alone without thinking that everything else is disconnected. They think I have this goal and this problem. I think you gain sight, can solve it for me. And I don't care which product does it. I have just a problem and you just need to solve it. So when you think about the answer to that, it's more about the use case than it is the individual product. So that's sort of another way I try to stay aligned and make sure that I believe myself in the customer's shoes. So it's more about the use case than the individual product. I agree, because at the end of the day, that use case goes across different products in some situations. And one thing that I've seen in the product world as it evolves, like I started 10 years ago and when did gain sight start, by the way? Oh, that's going to test my knowledge. I believe it's 2012, maybe a little bit earlier. So over 10 years and we've had a couple lifetimes and had a different name before. So it's been a while. It's been a while. And, you know, but back in the day, there wasn't that much awareness around the role of product or customer success. And you guys have been at the intersection of it. I've seen different products that are also serving product managers that take different angles. Like that could be analytics, it could be road mapping, it could be prototyping. And I've seen gain sight of all kind of from the product, the customer success, but that now connecting with other functions such as products. So I'll be curious to know what are some of those key use cases that you've seen that can go across both customer success as well as product. Yeah. So maybe calling out the three big ones that are like top of mind a ton these days. So usage data or product telemetry in a health score to predict the performance of that customer and whether they're going to renew or not. So I think a lot of people right now, especially given the economy, would like, you know, a crystal ball of where are all their customers going to be? How happy are they? Are they going to renew with me and, you know, have 100% knowledge? Which is great. I think one key input to that is how are your end users using your individual features and the stickiest features? And are they completing the different paths in your product that make them realize value? So one big one is how you marry product usage with health scores and the prediction of like retained ARR and growth and net retention. And another one that has come up a lot is digital led onboarding. So you have this great product. This just been sold to a customer. You need them to onboard. A lot of that can be done manually in trainings, but that is not how we as users are trained on our iPhones today. It needs to be really intuitive in the product. And so I think that's something where CS has a huge investment in making sure that the customer on boards and has a good experience. But the product can bring a huge amount of that like guiding the user through sending them in app guides that say, hey, we've noticed that you did this or we know that your role is this. So we know you're going to use the software in this way. Let me give you a tip on how to, you know, move forward in learning this product. And then maybe coupled with that, the third one would be self-serve is really big right now. And so is you think about a user and I like to my example I like to use, which maybe will resonate with you. When you think about self-serve, it's not like I regular user have a help ticket that I need to log in, whatever it's like. Think about the executive using your product, which is the person you most want to be in there. It's like the CEO of the other company, but it's Thursday night at 8 30 p.m. And they have a question and they don't want to bother somebody on the team who probably know the answer. But you want to keep them in the software because they even just logged in. They're doing something great. You want to keep them in there. If you have like a knowledge tenor bot or you have a community that can answer their question right away. So they keep moving in your product. That's super valuable. So those are things that are company level strategies or their customer success strategies and well product strategies that those use cases really marry nicely together. And definitely to double click on customer success piece, because for the longest time it's been treated as the second class citizen, right? Like sales, marketing, engineering product would get usually more visibility and resources. And then you have this used to have this team in the corner where like really get a ton of value keeping the lights on trying to make sure that our clients are getting value from our product. And now they're definitely being elevated. They're really getting an AC at the table. We're seeing more companies that are appointing chief customer officers. That's that's huge. Like I was literally celebrating that a few years ago when I saw more and more companies appointing chief product officer. I think that's all news now. And I see the next wave of chief customer officer. So I'm curious for your perspective, like what are you seeing? And how is that relationship between a chief customer officer with that CPO to make sure that they are also getting enough partnership? Yeah, totally. You know, it's funny. I go back to my original work that I did in my very first job. And I think about because I'm very much like math science person. And I think about the logic of like why why why did the chief there was no chief customer officer? Why were CS like I had a smaller seat at the table or no seat at the table? And it didn't make sense to me because very quickly as you're growing as an organization, your customer base becomes way larger than your new business revenue that's being acquired. Like to the order of tens of millions or hundred millions of dollars, this number becomes their customer base because it's a lot larger. And I think that gets exacerbated in an economy right now where it's high customer acquisition costs to acquire new customers.  Oh, surprise, you know, 80% of our business is recurring customer base. And it really starts that conversation about where is your growth coming from? Well, it's actually coming from our customer base growing with our own products or cross selling to multiple products. And if you think of it from a product perspective, and you're thinking about where am I gonna invest my time and where am I gonna invest my roadmap? Well, one of your biggest parts of your business is that closed loop feedback from customers putting in their product insights across maybe angry NPS surveys. We've all had a couple of those. But if you put the dollars, purely the dollars in my brain just because way back in the day came from a finance background, it is a huge investment to focus on the customer side. So when you're thinking about as a product leader, who are my stakeholders that I wanna spend time with when I think about where I should invest time? Well, a whole bunch of your customers are under that CS part. And I think a lot of companies are seeing CS as way more revenue generating now than they did in the past.  We like to call them product qualified leads, PQLs. PQL, definitely. I'm a huge believer in that. I think if anything, during this complicated economic downturn, we're seeing how product-led growth is just not a tactic. It's really a company-wide strategy and leveraging existing customers can lead to more lifetime value. Because obviously you can renew those customers. Ideally, you can upsell those customers. But on the other hand, those same customers can be leveraged to bring net new customers. And I think that is now, in times where you don't really have more budget for going out there and start just pumping ads or getting net new customers, thinking inwards and realize that, oh my God, the value is in house and it's more important than ever to treat our customers well. It's becoming more of a profit center. And I think it's elevating the conversation around why this team of customer success or even product deserves more investment than thinking of it as a just cost center because of course the sales team closed the deal and now we need to deliver something. Yeah, for sure. I definitely have to ask you about AI because it's one of the hot topics that is touching every single function of the organization. So for your product, what, how are you thinking about potentially integrating or leveraging AI? Yeah, so I think in general, like I'll be the first to admit that I certainly don't know everything about AI but it's something that's so rapid growth. Like it's on my mind. I'm trying to learn from smart people like, you know, X Amazon product leader, Polly Allen and Conviction founder, Vici Seriglo. So I'm like researching with, you know, content they're posting. But in general, I think, you know, when you think about the simplest answer is probably the best. Where generative AI makes a lot of sense is displacing low value human activities. So I think right now where, you know, we're focused on is how can you take things that already exist in the product and then for ways that you would configure in the product do have a generative AI start you on this. So if you, for example, engage site across our platform, you can have a journey where you take the user through and you have engagement that you make with them based on the steps and actions that they're taking in your product. So one thing that we just announced at our conference two weeks ago is investing in how we can generate what I say I wanna create, for example, because I already used it, a digital led onboarding journey with engagements. So having it be the starting base and then tweaking it based on the things you know about your business. So it's really about like, how can I use it to like make all of my PMs and my admins of the software? How can I make their time more powerful focused on the right activities versus the low value activities with starting something they've already done over and over again? One use case I saw as someone from Gainsaw actually gave me a demo, but it's really the difference between going from data into insights. So me, as I get access to Gainsaw, I have a lot of data points and I can access the data, but then it takes effort for me to make sense of that data. So leveraging that AI to automatically give me those insights in a unified view, it's really, really powerful. And I like that these new LLMs that are technically public and accessible for everybody can help, but I think there's a lot of value in connecting that with the in-house data. Yeah, so another thing we're investing in is taking product usage data or anything that you pulled out of your product data-wise and then using AI to pull out the insights. And especially somebody who spent a lot of time in business operations, like you don't wanna spend any of the time crunching the data and just getting to the answer.  Now, how am I gonna apply that forward to either inform my strategy, change it, compare it to other parts or double click and ask deeper questions. So it's definitely like, you could be quite a bit more efficient, but also it can take us into a much more rich conversation by taking some of those steps away. So super excited for that as well. And I think you're spot on. Well, now I have kind of a challenging question for you because I heard from multiple people at Gainset that takes writing being a human-first company and taking a type of approach. So how are you thinking about combining that human touch with superpowers that someone can acquire by leveraging AI? Yeah, so for those of you that don't know Gainset focuses a lot on its purpose of winning in business while being human-first. So really putting people at the center and understanding that first and foremost we're humans before we're even employees or customers or end users. And so, one, I think just keeping that focus in mind, it allows us to have much more empathy for what the user experience is going to be and keeping that in mind. But two, I think a lot of it is about, again, displacing low value activities to really take advantage of the human connection where it matters the most. So you don't want the people in your organization to do, for instance, like really menial tasks, like summarizing notes from a meeting they've had during they're doing a user survey or something that's gonna inform the product roadmap. You'd rather have something take all of the data that was recorded on a call and summarize it for you so that you can then make the smart decisions beyond that. And I think that is both the efficiency and scale part of it, but I also think it's sort of where people end up focusing as far as personalization goes, because, again, I think our standards of what we expect when we log into a software now are because of all the B2C consumer apps, they know me, they know so much about me. They're targeting me based on exactly whatever I've done and all of the history that they've aggregated. So when you log into a B2B software, you better have at least a similar experience of please know what role I am, why I'm logging in and also know what I've done in the product. And so if I've completed some steps, don't send me a pop-up that relates to something that I've already completed or already know or one example we like to use is if I'm logging in for the first time, don't tell me about a new release because I clearly doesn't matter to me because this is just the new product, this is not a new release, this is just your product to me. And so I think keeping personalization at the center of that is gonna be really important. I agree, that's what I see a lot of opportunities for product managers to add value through new technologies, such as AI, not as a way to replace the PM, but definitely like if someone can accelerate the creation of a PRD or simplified in a way that can be easier to digest if someone can help identify some of the different paths that can be created to provide a better experience for a new user versus an existing user, I take it. I think that's giving us superpowers and of course it still takes a human layer to edit and tweak the technology to make it more personal. Yeah, I went to a product conference recently and somebody used the analogy which I'm sure I won't get exactly right but if you think about humans and speed relative to all animals in the animal kingdom we're like at best in the middle of the pack but you put a human in a car or plane and now we're the fastest. So it's really about like that enablement of AI to just propel us forward in a way that we otherwise naturally couldn't but still with that human layer. I think PRDs is a great example who wants to do the first draft but of course it takes someone to go through it and be like, okay, the AI generated one doesn't quite get us there but now I have something to work with. Well, Caitlin, I loved learning from you. It's been a great time. Thank you so much. Yeah, thanks so much for having me, Carlos. Bye.


In this video we are going to discuss about occluding junctions and communicating junctions. Now communicating junctions are cap junctions. Cap junctions are cell junctions that enable transport of substances between the cells. So here there are two cells and via cap junctions substances can move from one cell to another. At cap junctions cells are situated very close to each other. In between the cells there is a space of approximately 3 nanometers but the membranes do not make physical contact directly. Now if we magnify the area we find that there are certain channels present on the membrane of the cells and channels on the membrane of both cells combined together. So these two cells are there. So channels of one cell are opposed with the channels of other cell forming kind of a pipeline through which substances can pass. So it forms a pipeline through which substances can pass between the cells. Now if we see a single channel it is made up of transmembrane proteins known as conixins. This is the membrane of the cell and this entire unit is forming a channel. Each unit is made up of subunit known as conixins and six conixins, six of them combine together to form the entire unit known as conixon. So the conixins are arranged such that they form a central pore. So conixon of one cell plasma membrane is opposed with the conixon of other cell plasma membrane through which molecules less than 1000 Dalton's can pass. Now gap junctions are present in cardiac muscle cells. So these are the cardiac muscle cells okay and they communicate with each other by gap junctions so that ions can pass from one cell to another via gap junctions. Because of this if single muscle fiber is stimulated the action potential can spread to other muscle fibers also that is why cardiac muscle cell behaves as a functional syncytium. That means if one cell is stimulated the entire cardiac muscle will be stimulated because of ions passing through the gap junction to other muscle fibers. Now some metabolites like ATP can also pass through gap junctions. This is important in the nutrition of cells of lens syncytium. So one example we saw cardiac muscle cell another example we are telling a vascular lens where metabolites are not provided by blood supply but they can pass through gap junction to other cells of the lens. Another junctions which we are going to talk about are tight junctions. We know that specialized transport processes exist for transport of substances across the membrane of the cells but wire substances not able to cross in between the cells from here in the space between the cells that is by paracelular path. This happens due to presence of tight junctions at the apical membrane in between two cells. This prevents paracelular transport of substances and allows only transcellular root. These tight junctions are present in epithelial cells of intestine, nepron where absorption as required occurs through transcellular path. Otherwise anything and everything can enter via paracelular path. Second example is urinary bladder where contents of the bladder that is urine should not leak out of it. Third they are present between endothelial cells of capillary in brain forming blood-brain barrier Pb. So this is CSF and endothelial cells of membrane there are tight junctions. So it's kind of a security system for neurons where the contents of the blood cannot leak out into CSF but because of this it also blocks access of drugs to CNS. So it becomes a problem when we want to treat any diseases of nervous system plus also in case of infection in brain these tight junctions are destroyed and substances can leak into CSF. That is why by studying CSF composition we can get information about certain diseases. So gap junctions allow transport of substances in between cells and tight junctions prevent transport of substances via paracelular root from the space in between the cells.


All right, everybody. Welcome. So my name is Tapio Tangren. I work at Nokia. And the topic of my presentation is performance testing opens stack networking with DPDK and RT. I'm going to talk about DPDK. So a little bit about my background. So I'm now working in most of my time in the OP NFE project, where I'm in the technical steering committee. So I'm going to use examples from the OP NFE because it's open source. You can go there and sort of take a look how people are doing network testing, performance testing in OP NFE. But before I started doing the OP NFE full time, I was working in Nokia sort of doing the performance testing with our internal real time open stack platform for the radio applications. So very strict performance requirements in that side.  We were doing sort of testing, benchmarking the platform. And we wanted to sort of make sure that there's no surprises that the performance is good. And we were getting sort of new releases every now and then of the underlying platform. And we were sort of supposed to make sure that things don't break, that the performance stays at the same level. So I was working with the people doing a lot of things by myself. And so what I'm going to talk about is like experiences about how we did things in testing that platform. And also a little bit like how things are done in the OP NFE project. So just the obvious thing is that the performance testing is a functional testing, sorry, non-functional testing. And the purpose is to test how well a system performance. And in our case the main thing that we are interested in is the sort of latencies. We are mainly interested in sort of a packet performance. So how fast packets move from place to place. We're interested in throughput. But mainly the sort of the really interesting question is always like the latencies. And the packet processing latencies from place A to place B. Because that can really mess up the performance of a BNF.  And they have, there's a very lively testing community in OP NFE. And the couple of projects that I've been, I should mention here is the yardstick. That I have been actually tried myself and installed it and run it and tested it a little bit. And there's also the Q-tip project in this area. And there's many others, just take a look. So before performance testing can start it's good to know what the hardware is and there's some limitations in sort of what capabilities the hardware has before you can even consider certain kinds of testing. So I mean there's a few things like the BIOS setting for example. There's some options that you can set to maximum performance and turn off all the power saving and sort of the CPU frequency scaling, things like that. That can interfere with the testing results. We also recommend disabling hybrid trading. So hybrid trading is great for general purpose computing. But as you know that when you have hybrid trading you have two trades, which are sharing some of the same resources. And if you don't have perfect control about where things are running that can also impact the performance.  But it's good to sort of change those settings in the BIOS before doing any installations and things like that. And then there's these NIC capabilities that have to be considered like hopefully you can run DPDK on your NIC. So basically two setups that we have for testing is, and this is a little bit obvious, but I mean there's like the load generator, which is sort of an outside machine. We try to use a sort of computers as much as possible and not specialized hardware because it gives more flexibility. And then the basic model is that you go to the V-switch in the compute node and then it goes to the virtual machine and it comes back to the V-switch. And so the way I done this with the open stack is just I launched the virtual machine either with provider network or without. And let's say I'm using VLAN, VLAN tagging is the easiest. So just check like what is the VLAN that this virtual machine is using and then when I have the load generator I configure the same VLAN on the load generator and then I know that it's in the same L2 segment as the virtual machine and the packet will actually go through to the right destination. And another way of doing measurements, which is actually easier to do but it, okay, there's a different tradeoff. It's just that you have two compute nodes and then you have two virtual machines and just sending packets back and forth. And this is of course easy to set up because you can do everything with open stack interfaces. You can set up a heat template to do the testing. And there's no special setup or configuring VLAN tags or setting up the switch or anything like that on the hardware. But then some things again are not possible when you're doing it like this. So since I had the dpdk in the title, I just added here like where is the dpdk used. So in the load generator I prefer the ones with the dpdk. And now the question is why do I have dpdk everywhere? The reason is that in this kind of setups we have like 10 gig links, we might have 25 gig links, something like that. And in our case we are mostly interested in sort of small packets. So we want to have testing which is saturating the link with the 10 gig link with 64 byte packets for the testing purposes. And that's why dpdk is the only way of doing it. If you have big packets, then it's possible to use something which is running in using the Linux network interfaces. So basically I can have the dpdk in the load generator, I can have it in the v-switch and then I can have it in the virtual machine. And if I put this parenthesis, this v-switch dpdk here because of course that's something that I'm testing. So it's the system under test is the v-switch and then this virtual machine on the right hand side part of the testing infrastructure. And then of course the other setup is that I had the virtual machine and then I'm running dpdk on both of them. Okay, and then just a couple of considerations like how do we want to do testing, what to consider when you're doing performance testing. So like I said we are mostly interested in network throughput and network latency with different packet sizes. So if you're interested in something else then maybe your sort of set of tools is going to be different. But these were like the main metrics that we were interested in and then the tools that I'm going to show you that we chose is based on being able to measure these things. And then of course I prefer open source because it's obvious with OPNFV we try to use open source as much as possible but it's also nice to have sort of software based tools because that gives you a lot of flexibility. And the flexibility that the software gives you it's related to this automation so it's worth investing effort instead of automating the testing and sort of make it so that you can just have a single script that will run dual of the testing. Because at least my experience in performance testing is when you do it you have to do it many times and sort of repeat it and repeat it and repeat it and make little changes and then run it again. So easier you can make it the better it is. And then one consideration in sort of choosing the tools is sort of where do you put the results and how do you show the results. So it's good to have a lot of information stored somewhere but it's also even better if you are sort of able to somehow show what the results are and being able to sort of take a look at the data and sort of see if something is going wrong. Like I was saying that we were sort of doing this sort of regression testing that we had like one version of platform, next version, next version, next version. And what we wanted to do is sort of take a look at the numbers that we get from each release and then it's of course nice that you have like a curve that you can sort of if you can sort of graphically take a look at the results then you can sort of easily with your eye spot if something is going wrong. Otherwise it could be quite difficult to sort of just look at the numbers and notice something like that. So here's some of our favorite tools that we have been using in doing this testing. So starting with the DPDK, if you just download the DPDK like I usually do and compile it, then one of the applications, example application sort of is this TestBMD which is of course testing. I mean it's main purpose is to sort of test the DPDK installation and see that it works but it also has this nice feature that, or actually a couple of nice features. One is that you can configure it so that it just takes a packet in and changes the MAC addresses and sends it out. Like I said that one of the ways of doing the measurements is just sort of set up the L2 segment, set up the VLAN if I'm using VLAN overlay. Then I don't care about IP addresses or anything like that. I just put the destination MAC address and then it goes to the VM and then there's TestBMD which will take the destination MAC address and make that the source address and then configure another port as the destination address and just send it back. So this makes it the job of the TestBMD very simple and that makes it actually very, very fast. Even if you're running it in a virtual machine it requires very little resources when you're doing that because like I said the only thing it does is just switches two fields in the packet. Then TestBMD in my experience again it's nice because it does this like text interface so when you're setting up things and things are not working you can have a console access to the VM and look at the numbers. It has the counters like how many packets are going in, going out. So that I have found convenient. And of course when you're setting it up there's a few things that always with the DPDK you have to do is set up the huge pages, the two meg should be enough because we are not, I mean that should work enough. And then in my experience, I mean there's like a couple of ways you can do user space networking in Linux. One is this VFIO and the other one is UIO. And my experience again is that the UIO works better and then you have this IGP-UIO driver in DPDK. That's my recommendation is to use that. But I'm working, maybe I get VFIO working in a virtual machine one day. And of course an alternative would be to SR-OV. Okay SR-OV in my mind is not that cloud like but there's some certain use cases where it can be used because SR-OV has the benefit that it's almost like zero overhead. You get the same performance from the SR-OV interface that you get from the native NIC which is of course useful. That you sort of, you know that there's no overhead then instead of using the SR-OV interface. So that was one tool. The other tool is Moongen which is, it's a very nice idea. And it's a sort of load generator written on top of DPDK. And the trick about Moongen is that there's a feature in a lot of the NICs which is called BTP, Precision Timing Protocol. And the functionality is such that when a packet, sort of certain kind of timing packet goes out, the NIC itself put a timestamp on it when the packet goes out. And it sort of, it's then the timestamp is in the packet and then it also works the other way around that when you get a packet gets in, the NIC can sort of look at the timestamp and sort of compare the numbers and so forth. And this is really great because that's like the, when you're doing the measurements, you don't sort of, if you take the timestamp at the host when you're sending the packet, you still get some delay when the packet travels all the way to the NIC and then goes out. This Moongen is using this BTP feature and it means that the timestamp gets put to the packet exactly at the moment it goes out. So the great thing about Moongen is that it's still like the, if you want to use standard PC hardware, it really has like the best possible performance in latency measurements. If you want even better performance, you have to use some dedicated hardware devices. Of course the downside is that there's some, all kind of things can go wrong and there's all kind of things that sort of debugging may not be so easy but in our experience, I think the Moongen has been working quite well. And it has a scripting interface too. One fascinating thing about Moongen is also that it uses the Lua language, translated, sort of cheat it Lua, which is a sort of a scripted, scripting language. And you can write a script in Moongen in Lua, which sends the packets and does something, which is really amazing because you can get extremely good performance with just sort of a script. I mean you can send the packets from the script and you can saturate the Tengik link with that. Yeah, and as I said, it's open source, it's all done in software and it works best if you have like, I mean it doesn't support all nicks. It's an open source project, it's quite small open source project. So you have to check that you have the hardware which supports Moongen. So if I put these two projects together, I have this kind of picture that I have the Moongen on a physical server somewhere and then I have the virtual machine with dpdk testbmd. And like I said, I just do it so that I launch the VM. I have the dpdk testbmd pre-packaged inside of the VM, all set up so that it starts forwarding the messages. I configure the VLAN on the Moongen. You can do that also on Moongen that you can sort of define it so that you put the VLAN tag on the packet when it goes out. And then it goes to the switch. Now the funny thing is that I've tried this many times with sort of so that you're using a switch between the Moongen machine and then the system under test. And it doesn't somehow seem to work with the switch in between. So I wrote this text here to direct connection or switch between. You really have to take the internet cables and plug it directly to the machine and then it works best. I've been debugging it a long time and couldn't figure out why it doesn't work. And then I had this other setup and then there's another dpdk-based tool called dpdk-package. And it's not compiled from the same source code. But it's a separate project. But if you go to dpdk.org and look at the downloads you can also find the dpdk-package from the same location. It's also scriptable with the Lua language. I haven't used the Lua bindings, but I've used sort of the command line. So when you're launching the dpdk-package you can sort of tell it how to run. What kind of traffic to start generating and so forth. So it's scriptable in that sense. And like I said about the benefit of dpdk is that you can create, sort of generate the huge amount of packets, huge amount of traffic with small packets. Like being able to saturate at least a 10-gig link with this. And now, sorry, went the wrong way. And actually the way I would use this is that I would use this in the VM to VM setup. Of course the dpdk-package also works on the physical machine, but this is like my preferred way of doing it. There's one sort of blow generator with the dpdk-package and then there's the dpdk-test.bmd on the other end. And then you just send traffic as fast as possible and then you make the measurements with that. And now those questions, okay, I mentioned already that you can use Moongan or you can use dpdk-package also in a physical machine. So why do we have both? So a couple of years ago when I started this, things were easy. I mean I was looking at Moongan and Moongan didn't support running in a virtual machine. So in a virtual machine you have the virtual interface and the virtual interface was not supported by Moongan. However, it's been added afterwards, but I haven't tested it now. And then on the other hand, if you wanted to use the dpdk-package earlier it did not have the latency measurements. So that's why I needed to have two different tools. I needed to use the Moongan for latency measurements and then packaging for running between two VMs and sort of generating a lot of traffic. So this is now a little bit more complicated than that because Moongan, like I said, it does support the virtual these days and packaging does support the latency measurements. However, like I said, the Moongan has really this fancy feature of supporting the PTP packets and being able to do extremely precise measurements of the latency. So my recommendation again is to use the Moongan in bare metal deployments because that's like you get the best features if you have to support it hardware. And then dpdk-package seems to be very popular. I've seen it used in a lot of different projects. For example, in OPNFE projects in a virtual machine and then so I would really recommend using that in a virtual machine because things don't always work the way you expect. So you want to choose the same setup that most people are using and then you can go to the internet and ask questions and find answers. Okay, then I had this question, this point about the sort of reporting. How do you show the results? This is something I sort of like I said, I was playing with the yardstick in our lab instead of sitting in top and running some of the tests. So I took sort of this screenshot from the web browser. So these are just sort of some of the tools that can be used. So this yardstick, OPNFE yardstick project is using this influxdb, which was very easy to set up for me. There was a docker container, I just set it up and then directed the yardstick to sort of start sending the measurements results to the influxdb. Actually, I think like HTTP messages or something like that. Something very simple to set up. And then this is a screenshot of the Grafana tool. So Grafana is an open source tool, which is creating a web page to configure a sort of to display information. You can just install Grafana, and then you can sort of click, click, click, click on the web page. You can set up the data source in Grafana and sort of configure what is the password and such things. And then what it does show is that it sort of shows a timeline. So it has sort of timestamps on the x-axis and then on the y-axis, you have some kind of measurements. And I really had to say that, again, with my experience, this is the best use for Grafana, showing this kind of timeline measurements. And it's not perfect to show the results because it's showing the timestamps. And what you really would like to show is release one, release two, release three, release four, things like that. But anyway, like I said, it's very easy to set up and it seems to be working quite robustly. There's other alternatives that people have been using, so it's worth exploring a little bit about what the options are. So I mentioned this inflex DB. Some people are using the Elasticsearch, which is a nice sort of feature like the search is very easy. So if you have a big database, you want to look for some measurement results with some keywords, you can do that with Elasticsearch. And then this MongoDB is what is used in the opnfe testing community, so that they ended up using that. So that's also a very popular choice. And then about this visualization, I mentioned this Grafana tool. I already showed it already. My picture was not that beautiful, but the point is that the set of ways that Grafana can display information is a little bit limited in my mind. And there's another tool called Kibano, and that is used in the opnfe test community, for example, to show. And they have very, very beautiful sort of diagrams. All kind of made gauges and meters and colors and things like that to visualize the testing results. But the sort of the trade-off is a little bit that, okay, Grafana is easy to set up with different data sources. Like I said, just click, click, click. And Kibano has really nice graph types, so a lot more capabilities in that sense. But then there's, okay, the opposite is also true. And so, like I said, the opnfe testing community has been working on this, and you can sort of check what they have found out, and it's all available in the opnfe wikipages to read more about that. Okay, so just sort of a little bit disclaimer about, like you can do a lot of measurement tools with these kind of open source tools that I was mentioning to you. Like you can do latency measurements. You can easily to set up like a constant rate, constant bit rates with different sort of, I mean, okay, you can't completely freely choose what is the rate, but you can have a lot of freedom in sort of setting different traffic rates. You can also set test with different packet sizes quite easily. And if you really want to get fancy, you can have like a p-cap file. You can have these tools like the packet chain, for example, or moon chain, both are able to read p-cap files and just sort of, you know, send the same packets over and over again if you want to have like a mixture of different packet sizes and different traffic types and things like that. And then this, you can also find readymade scripts for doing this RFC 2544 measurements. This RFC 2544 is sort of, it's sort of way of measuring the throughput for different packet sizes. So it's sort of doing like sending packets and then if you have packet loss, it decreases the rate and if there is no packet rate, it sort of increases the rate and sort of you can converge on something where the packet rate is acceptable and then it will sort of tell you the result. It's very nice because then you get like one number. If you fix the packet loss rate that you want to have, then you can have sort of a single number, which is a throughput for its packet size. And it's sort of a little bit tedious to do it by hand. So you really want to have like a script, which is doing it. But so this can be done. What cannot be done as easily is that if you really want to sort of test sort of more exotic traffic patterns such as you want to send a burst of packets with fast rate and then have a delay and then send a packet of a burst like that, then really what you should look at this is writing your own test using DPDK. We have internal inside of the company, we have a tool like that and it's not a huge amount of code on top of DPDK. So it's doable. And also what I think cannot be done as that easily is if you want to test like with broken packages and sort of doing this kind of measurements. If you want to have like the wrong checksum or something like that, then you want to check what is the performance if this happens. Okay. And then finally sort of make it a little bit more complicated. So now it looks like you can just run a single test and you can get the number out. But then this is like all kind of things that can affect the performance besides the settings that you did in the beginning. And one is this of course is that you have to sort of think about like CPU pinning. Like where is the VM running? You know, if you don't pin the virtual machine to certain CPUs, then you can have like Linux operating system can migrate the virtual machine to different cores and then you get latencies while the migration is happening. And then there's also this pneumatopology that if you have dual socket machine, it makes a big difference if the two virtual machines are running on the same socket or in a different socket because if they are in different socket, then you have to go across the two sockets on a physical interface and that takes a lot more time. So this picture is about this pneumatopology. And then another one that makes a big difference that it's always worth checking and it's a lot of work to do sometimes. Is this a PCI topology? So if you have a dual socket machine and you have a NIC card and packet comes in to the NIC card, then usually it's so that this NIC card is only connected to one of the sockets. And now again, depending on where your virtual machine is running, then you can have different results. So in this picture you have, okay, starting from the bottom you have the switch, then there's two NIC cards on two different machines, then you have two sockets on two machines and then you have a virtual machine running on a socket. So this is like supposed to illustrate the worst case scenario where you have the virtual machine in sort of in both cases in the wrong nummonode. So the packet originates from the virtual machine, it goes to this. In the first socket it goes across to the other socket, it goes to NIC card, it goes over the switch, it goes to NIC, it goes to the other socket and then it goes to the VM. And there's actually big differences when you measure the performance sort of between the best case and the worst case. So if you don't control these kind of factors, then you don't really know what kind of numbers. Your numbers can be widely different. So again what I do is that I take a look at the PCI topology, I figure out sort of where is this NIC connected to and then I just tell OpenStack to run this virtual machine on either one of those sockets. Or there's some features in OpenStack to do this automatically so that it can optimize the performance. But again I like to do things by hand and sort of have control about where the virtual machine is running. Socket zero, socket one. And then of course one thing that always sort of really messes up things is that you can have other system load on the system. So you can have latency, very good. Like I've been doing these kind of measurements and seen it many times. You can have very good latencies without any system load. And then when you get some load, then you get much worse performance. And this is of course one thing that makes this performance measurement so difficult is that you don't actually know what kind of other processes or what kind of background processes are running in the Linux operating system at the same time so the results can vary a little bit. So what we do is that we have usually, since you remember initially the purpose was to sort of verify that we can run BNFs on top of this OpenStack platform as well as possible in all possible conditions. So what we do is that we run some stress testing tools, sorry, some stress BNs on the same system and then do the measurements and sort of that way sort of try to verify that even in the worst case you get some reasonable performance. So spent some time early in my life looking at this kind of different tools and identifying which ones are the good ones. So I just put the links here about some of the tools that can be useful. And here I have at least I've been using sort of three different kinds of loads. So storage, traffic, CPU load and network load. So there's a couple of tools for storage. I don't know which one is better. I think we have been using both. And they're pretty much the same in what they do. And basically just sort of define what kind of storage test you want to run. So both of these tools are able to sort of write random data on a file and read random data from a file and do it repeatedly, continuously, all the time. For CPU load there's a really great tool called CPU Burn and it's able to sort of really give high load to the CPU. And that's a really great one because otherwise without any tool like that it's really difficult. I can't think of any way of sort of generating the CPU load. Like 100% CPU load. And then for network load, I didn't mention IPerf or IPerf 3 earlier. So that's a very easy to set up standard Linux tool for measuring latencies and network traffic and so forth. The limitation in IPerf is that it's using the Linux standard network interfaces. Standard Linux drivers, kernel drivers. And the performance is quite limited. So you don't get anywhere near 10 gigs.  And of course it all depends on the packet size. But with small packet sizes you get like one or two gigs. So it's nowhere near where the capacity is. But if you just want to have like background noise, background load, IP based and don't want to just something that is very easy to set up, IPerf is great. So just make a little script that launches IPerf and then do it. And then another alternative is to have this VM to VM testing.  And inside of the virtual machines you have storage test, network test and CPU load running at full speed. So you have these VMs which are generating noise and generating traffic on the storage system and the network system and also loading the CPU as much as possible. And then you do the measurements. And of course almost all of these tools that I'm listing here they are meant for like measuring something. But in this case I don't care what the result is. So I can just throw it away. I just launch a VM, it generates some load and I don't care. It's just background, for background noise. Okay, yeah, okay, that was actually it. I hope that is useful. I think we have a minute or two if there are some questions. Yeah. Hi, you mentioned about Gen2's Mungent for measuring latency. And it's using PTP I believe. So do you know, what's the level of accuracy of resolution for timing measurements like for latency? Don't remember. But there's an academic paper. I mean it's a university project. They wrote the papers about that. And I don't now remember from top of my head what the measurements were. Okay, that's fine. I have another question. So do you know of any other tools, which are any other way to measure the internal VM latency or timing measurements if, let's say, two or more than two VMs are deployed on the same compute node. How do we measure the latency of packets between those two VMs? You mean besides the packet gen? Oh yeah, so if a packet gen would sit at the end, right, at the NIC cards, I mean it would be pumping in traffic or at the NIC card or it would be sitting at the ends of the service chain. But how do we find the latency between the two VMs? Let's say if there are three VMs, I want to find out the latency between VM 2 and VM 3. I don't care about latency about VM 1 and VM 2 separately individually. Oh yeah, we've also done measurements like that. I mean we do a lot. Yeah, we've done it. It makes actually sense to do it like that, to sort of set up a chain. You just chain them. You have the packet gen here. You have test-BMD. Test-BMD is just taking packets in, sending out. Then you have the next VM, takes packets in, sends them out. Next VM, you know, code like this. And then in the end, you go back to the beginning. And then the latency is with the packet gen. So you can move the packet gen or the moon gen point of connection? Yeah, yeah. Or you can use either one of those load generators and you sort of chain it like that. And that's a very nice way because then you can sort of, in a single test, you can do sort of, test all of the cases, you sort of send packet first inside of the same socket, the different socket to different machine, different socket, and then you come back, and then you sort of combine all of these latencies and get one number. Yeah, makes sense to do it like that, yes. Thank you. Good suggestion. Great talk with the tools for testing. A quick question. So what's your general opinion of the test results, the performance of DPDK versus the test results? Well, we didn't... Yeah, just the latest measurement, yeah. Sorry, I can't answer it right now. And because this was more like sort of verifying that the platform itself, we were not testing sort of comparing the DPDK implementation itself. We were just using the DPDK as a tool for doing some other measurements. I think the DPDK guys, DPDK people have been doing a lot of testing with sort of different latencies, but I think in general my sort of opinion is that this DPDK is good enough for sort of doing this kind of measurements. It's able to sort of saturate the link, get enough packets for sort of throughput testing. It's good enough, and for the latencies, I don't... So you mainly just... Yeah. ...for your other testing as opposed to... Yeah. Yeah, yeah. DPDK itself. Yeah, right. Precisely. Thanks. Did you ever come up with results for DPDK? Is Fonks same question? No, that's a topic for another talk, but no, it was... Yeah, we were just sort of... Yeah, Intel people have... There's some information, some measurement results on the DPDK.org. I think there's a couple of papers that have been published on that. I have been using myself in doing that. Oh, by the way, if you're interested in getting the slides, I'm using Google to publish these slides. I put the link to Twitter. So if you go to Twitter and you look for my name, so my Twitter handle is my first name, TAPIO underscore TA. You search for that. The last Twitter post that I put is a link to these slides. So you can download them and take a look what I have done. Okay, thank you.