All right, thanks. Hey. Right on. Thanks, everybody. Yep, that's me. I'm Brian here to talk about A-B testing. I've been doing A-B test implementation, development, and strategy and analysis for the last four or five years or so. Georgia State grad, go Panthers. What's up? Yeah, yeah, OK. This is my first time at WordCamp as a speaker or attendee. So thank you so much, everybody, for the warm welcome. All the stuff I've learned already. In the interest of honoring your time and attention, I want to lead with this magic number. And we need a new lamp for the projector, just duly noted. 500. So this kind of magic number is the number of conversions that you need per month on your website to really get value out of A-B testing. So I just thought I'd throw this up here right away early on. If anybody sees that and thinks, well, forget it, and you want to sneak out, I'll be heard a little bit. But I want you to use your time well this weekend. So no hard feelings. At the same time, if you don't have a website or a client with a website that's currently getting 500 conversions a month, this kind of stuff can happen fast. You might build out an e-com site, and they stumble around for six months trying to find their market, and then bam, something hits, and they're there. And it's time to think about optimization and experimentation on the site. You might build some sort of a lead capture site for somebody.  And if you're paying for your visitors, you definitely want to be doing some sort of optimization program to get cheaper clicks. And so this stuff becomes relevant when you hit that magic number. And just so it's clear, conversions, 500 conversions, as in a thing that makes you money on the website. So not just a click or a page view or a video play, although those are great. And we care about those. But we're talking purchases, e-commerce, or subscriptions to your mailing list, if you have monetized that, or signups and lead submissions if you have a sales-driven product. That's the 500 I'm talking about here. And we'll get into why a little bit. So this is my outline for today. Just going to explain the basics of A-B testing, because I don't really know who's in the room. Hi, hey, nice to meet you all. I don't really know your level or experience or comfort with the topic. It does get kind of complicated, but I'm going to try to give it a pretty basic pass. And then talk about why you would want to do it, the point of it, what you can expect to benefit from it. And then we'll dive into some specifics of how you can actually set up and start doing this yourself. And my intention is to make this a fairly light pass over these topics. And then we'll go deep on the questions. That's my hope. So I do hope that I can keep you all in the room and that you'll have some questions when we get to the end. All right, so what? The what of A-B testing, what is it? So I'm going to define it, and then we're going to do a quick action-packed live demo, brace yourselves. And then hopefully by the end of that, everyone will have a sense of, like, OK, yeah, get it.  OK, so I won't read it to you. You're welcome. You can all read. I'm going to leave these words on the screen for a little while and kind of step through and talk about some of the key points as we go. So you go ahead and read it to yourselves. I guess you probably already have. I'm going to talk about and call out what's important about this. So first off, website. Now, A-B testing is an experimental methodology that you can use to optimize or to measure the impact of changes across all kinds of channels. You can A-B test your ad copy. You can A-B test your outbound email headlines. You could A-B test pickup lines, if that's, like, part of your lifestyle. Today we're talking about website only, just to frame it. So next up, randomly. So with A-B testing, we're randomly choosing who sees what, and this is key. If you're using rules and data to say, OK, I want to show this guy this experience, but then I want to show her this one, also a thing. That's also a thing. And it's a lot of the same tools and process to get that done, but that is called personalization. And it's also fundamentally different in the sense that we're using statistical rigor to say, look, this really works. Look, this experience is really better. And with personalization, it's a lot more of a hunch-driven type of methodology. It's like, oh, this will probably work, right? Sure. Yeah, and then you go do it. So a little bit of judgment in there, but we can go into that in the questions, too. Talk about the differences and when you would use one or the other. All right. So this is the maybe mind-blown part. If you do not have a background in this, if you haven't done this before, if you don't have a technical background, this part is kind of strange to think about. And this is kind of key. So simultaneously, at the same moment in time, different people will be seeing different things on the same page of your site. And that's pretty much the whole point of it. So if that doesn't really make sense yet, that is OK. Just roll with it. We're about to do a demo that I hope will help a little bit in explaining or at least demonstrating how that goes. And then the last thing. The last thing I'll call out, but it's so important, is a given metric. So again, thanks everybody for coming to this talk. There's a talk in another room that's about measuring everything on your website. And I wish we could all be there, too. It sounds awesome. You can measure everything. You cannot optimize everything. You can optimize one thing.  It needs to be purchases or sign-ups, because you can only choose one thing to optimize for. So if everybody's willing, I'm going to do a quick demo. You have a phone. You have a laptop. If I can convince you, I'm sorry there's so many letters in my name. But if you would go type this into a browser on whatever device you have, give everyone a couple minutes. Thank you so much for being so very willing and cooperative. There's a test running on this page. So there is technically an ABCD test running on this page. And so when you land on the page, you will get one of those experiences, ABC or D. Is anybody there yet? Yeah? OK. If it's feasible and you won't hurt anybody or break anything, would you mind holding up your screen so that other people in the room can see what your screen looks like? Yeah? OK, so you convert it. Yeah, OK. So the white screen, she clicked. Thank you for your click. Thank you very much. Appreciate that. Everyone else, we see orange. We see red. We see purple. We see blue, right? Kind of a stupid test, but what we just did was we all just got, we say bucketed, or kind of channeled into a specific experience in a test. And so the two things I want to call out are, number one, as a visitor. Just as a single person in this experience, that's the only experience you see, right? If you navigate somewhere else and then come back to the page, you're going to see that same color that you just saw. Close the window, open it back up, come back. You're going to see the same color. That is your experience. It sticks with you. It's kind of a per visitor experience thing. But from the point of the view of someone who's running the test, so me and now y'all, because I've kind of lifted the veil, right, and talked about this test, I can measure the results from all of you in aggregate across the variations. So I can go say, OK, how many purple people clicked? All right, what about orange people? How many? And so on and so forth. So it's happening in real time. It's happening simultaneously, but it's random and it's distributed across the room. So maybe a third thing I call out is that this happens to you all the time, right? This was staged, and it was made really obvious by bright, shiny colors so you can see, like, oh, yeah, she got purple. Cool.  You don't know, because you have nothing to compare it to, because it sticks to you. It stays consistent when you come back. But they're running tons and tons of tests all the time, and they're measuring the same thing. Does she add to cart? Does she purchase? Does he stay on the site longer? Whatever they've decided makes them money. So not a novel experience, just novel in its simplicity, maybe. You're in tests everywhere you go. All right. So I just showed you a test. That was the first test I've talked about. It was a terrible test. So I kind of need to call that out. Like, let's not let that stand. Having kind of defined what a test is and what you need for it, just to recap, it needs to measure something that matters, something that makes you money or that you really care about. And it needs to have enough traffic and conversion. So right there, I've measured your clicks on that little click here link on the page. I don't make money from that. So that was kind of a bad idea, right? Traffic and conversion. So there's no possible way I'm going to get 500 clicks out of this room, even though you've all been really, really cooperative and I appreciate it. So not a great test in that sense. And then actionable results. I have to be able to take whatever results I'm collecting from this experiment and I have to be able to do something based on it. I cannot do that here, right? What could I possibly learn? Like purple is slightly better than blue. Yay. So you just saw a test, you experienced a test, you saw it from the inside and the outside. But it wasn't a great test. So we're going to move on and look at some great tests so you can see what those look like, because otherwise I'll be doing you a terrible disservice. All right. So we covered the what of AB testing. Take a quick breather. And I just want to say, if this is where you're at, like if this is your mind state right now, it is OK. Let's just write it out. Let's just roll with it. There is a lot going on. There's a lot of technical stuff to think about. There's a lot of statistical complexity that I've so far shielded you from, but it gets deep.  We'll step through a couple of good examples. I'll walk you through how I set up that test. We'll talk about the landscape of tooling and stuff like that. And then I want to have plenty of time for questions. So just stay with me. It's OK. This stuff gets hard. But now we'll talk about why. So I'll give you a few reasons why you should run A-B tests on your website. This is the first one, OK, without a doubt. And this, I mean, hopefully this is enough for some of you. So the year is 2008, a simpler time. This is Obama's landing page. I think we can assume that, I mean, not Obama personally logged into Google AdWords, but he's got help. They're buying clicks. They're buying clicks for people to land on this page, probably social too. Paid social and organic social. Any way we can. Let's send them all here, because we want there. So quick pause. Audience participation time. What is the conversion event that is valuable that's going on on this page? Yes? Sign-ups. Yeah, yeah, exactly. I don't know if anybody was in the presentation yesterday about video using video. Or if you're in the one just now too, we talk about funnels. These people are very top of funnel. They're just aware. They're interested. They're like, OK, who's this Obama guy? And so you can't just say, give me money. No one's going to give him money on this page. He wants the email. He's going to send out campaigns, and eventually they're going to donate. And so more sign-ups, more people receiving the campaigns, more donations. He might even win the 2008 election if he can get this right, y'all. Edgy your seats. Yeah, right? OK, so this is an experiment, a real experiment that really happened. And there are two different elements on the page that are under test, that are being swapped out with variations so that we can measure the effectiveness of them. So first is the media, the little hero image. Call it a hero. Some people know that word. Some don't. Don't sweat it. And then a button, specifically the button text, all right? So that's what we're here playing with, right? We're going to change those up, just like I changed up a red background to a purple background to not much effect. We're going to change these up to huge effect. So first round, here's four buttons. Four button copy variations, all right? So everybody, go ahead and pick a winner. It's cool. It's like Vegas. Think about what you think is going to work best. You don't have to tell anyone. By the way, I've done enough of these and been wrong so many times, y'all. So many times that I no longer pick winners. I no longer have an opinion on what's going to go well and what's not. I had to give up to protect my fragile ego. But it is fun to kind of play around with it. So go ahead and pick what you think, yeah? Good, all right? So next, for that creative image, right? That media up at the top. Three different static images and a video background, like a kind of no sound auto play video background thing that you see a lot of. This one didn't win. I mean, the video didn't win. And a quick tip, generally speaking, a static image does perform better than a video in that location on a page. You don't know until you test it. But when I see that, it's one of the first things I want to test, because I've seen a static image outperform lots of times. So you only have to choose from amongst three images. You've got those four buttons, so that's 12 possible combinations. So someone in this room is probably going to get it right. All right. We ready? We ready? Yeah, can you feel the suspense? It's getting warm in here. All right. Boom. So anybody get it right? Woo! Yeah, way to go. Okay, so you have a future in AB testing and optimization. Congratulations. Okay. So the family image with the largely white background, the learn more button. So out of all those combinations they tested, this one had 40.6% higher sign ups than the original. And if you kind of math it out and just have to, there's a little bit of good faith involved in this math, but if you multiply that by how many people were hitting the page, that's how many more sign ups. If you multiply that by how much average donation was per sign up, it's a ton of money, all right? So like we're in the why section right now, right? Why should you care? Why do you want to do this maybe sometime for some site? It's because you'll make $60 million. Yeah, good one, right? I know, yeah. Okay, but you know, Obama did and we know how that went. So to generalize, if you apply this methodology, if you do it over time, it won't be the first test, it won't be all in one test, but if you keep experimenting and optimizing your site in this way, you will increase conversions and that I assume is something you care about or your clients care about. Here's some more kind of glory stories about AB test successes from around the web. There's a million of them. I went and just pulled a few screenshots out. $106,000 increase in monthly revenue. I take some of these with a grain of salt, but like I wrote this one, so I guess I believe this one. Depending on the scale of the business, depending on I guess how much room for improvement there is, there's a lot of money to be made. And so you want to do this because you will make more money doing it. And then a final reason is if you're thinking about something like a big redesign, if you're thinking about an entire repositioning of your product line, something risky like that that could potentially make people angry, make people go away forever, testing it first instead of just rolling it out and hoping for the best is a way to take that risk away. So the top two images, these are kind of like, I don't know, low-key famous disastrous redesigns that happened. Marks and Spencer's British retailer lost a ton of money because they redesigned their website and they rolled it out and it sucked. Dig was a thing and no longer is cautionary tale, I guess, right? So here's a more recent one, right? Instagram, I don't know if anybody remembers for a minute Instagram had this new UI where I think you had to like swipe instead of scroll or something like that. It rolled out for maybe a few hours and the internet went crazy, like what is this, you know? And it was over as fast as it happened, right? And so like depending on whose story you believe, either this was a bug or it was a test that went out to more people than it was meant to go out to or whatever, but the point being like nice recovery, right? This I have to believe was implemented as an experiment so that when people started tweeting about how much they hated the new UI, they just press a button. They just press a button and it's over and it's like, oh, it's just kidding y'all, right? So I mean, imagine if they just rolled this out to production and had fumbled around for days or weeks being like, we're working on remote, like it would have been a disaster, right? So this is a way to roll stuff like that out. And even if it's terrible, I mean, I don't know, people still use Instagram, right? So they survived that. All right, so now we'll dig into how, how do you do this? I assume most everyone in this room has some level of control over some website somewhere and I don't know how excited you, it's like afternoon and it's warm in here. So like I can't tell if anybody's pumped up or not, but like I'm just gonna assume you might wanna do this yourself. Let's talk about what that entails. Getting started, not super complicated. There's a lot of helpful tools out there. You don't have to have a degree in stats. You don't have to be a developer either. So I'll show you how I set up the test that we did a demo of. All right, so I use Google Optimize for that test. And I will, I'm gonna go ahead and say that if you are new to this and you wanna try it out, you should just use Google Optimize. It's free, it's pretty easy. It's well documented, it's battle tested. And so why not? We'll talk about other alternatives in a bit, but I'm just gonna go ahead and put my stamp of approval on this one for now. All right, so you need to install it. So what that involves is you go to Google Optimize and you click a button and you create an account and you all know how to do that, I'm sure. And then you click another button to create a container, which is just like a project that kind of houses your site because you can have multiple containers. If you have multiple sites, you can be running 100 experiments and you're awesome.  So this tag right up here. And one nice thing about Google Optimize is it integrates with analytics, with Google Analytics. And so this snippet will actually replace your Google Analytics snippet if you've already got one. It's just both rolled into one. So you gotta put this onto your website. Technically, any page where you want to either be running an experiment or you want to be collecting data, like measuring conversions. But ideally and most simply just everywhere. Just put it into the very top of the head tag of every page on your site. Sir. If you have multiple tests, multiple statements like this, how do we combine them? Do we combine them? So the question is if we have multiple containers, multiple sites, I'm assuming, how do we bring them together? So the short answer is you can do whatever you need. If you have multiple sites and you want to run the same test across those sites, then you can just use the same tag on the multiple sites. And that's all you have to do. But if you have separate sites that you wanna keep separate tracking, separate experiments, you wanna keep them kinda separate, that's two containers, two tags. Yeah, cool, thanks, thanks for the question. Actually, so while we're on questions, I'll turn one around to y'all. I mentioned that this needs to be at the top of the head on every page on your site, at the top of the head. Anybody, any dev people out here wanna hazard a guess as to why it has to go way up in the top of the webpage? Why this snippet, this one special snippet can't go down at the bottom of the body like so many others do, sir? Yeah, yeah, why does it need, he said it needs to load before the website, which is true, and why is that? Why do we need this one to load? So yeah, so in case somebody clicks on something before it loads, you wouldn't track it. And along those same lines, we're using this to change the page, right? We're using this to like swap out text or change an image or whatever kinda stuff. And so if this comes in slow, if this comes in two seconds after page load, they're gonna see your original page and it's gonna be like, blah, just kidding, and show you the new page. And like, you think that's gonna win? Like, do you think that experience is a better experience?  And so no, so this has to go at the top of the page. Almost every other tag on your site can go wherever and you probably should load them later, but this is a special one. Yeah, because you're trying to measure every single interaction for the accuracy of your data and because you're rewriting stuff like before people can see it. So I have a theme that has a header, scripts, little customization module here.  Or there's plenty of plugins that let you inject scripts, CSS, JS, whatever into whatever part of the page you want. Yeah. Is there a theme that you like? I, hmm, a theme or a plugin, a specific theme? Didn't you found works well with that? Hmm, no, there's no particular themes. The question is there a theme you would recommend for the purpose of AB testing or? Yeah, the accessibility of the header. Yeah, I would say absolutely not no opinion. Whatever theme you're working with for whatever reason you chose it is perfectly fine. And worst case you install one plugin and you paste in one tag to inject in the header and you're good to go and you didn't have to touch anything else. I picked my theme just on aesthetics and it kind of does what I'm wanting it to do. But I've injected this into probably half a dozen different themes and even if you coded your own and it's janky as can be, you can still find a way to get it in there. So yeah, no restrictions there. It's not a plugin, no. It's just a separate, I guess standalone app just like Google Analytics. You go log in and look at stuff. It's very similar to that. But you said it takes over. So they share a snippet I guess and they integrate. That's right. If you have the Google Analytics snippet already installed on your site, which you should, good for you.  That's your new Google Analytics and optimized snippet. I'll be right with you. Yes, man? Okay, good question. No, no, no. I got you. I think so. Yeah, the question is so you say you wanna have two different home pages, right? We looked at an example where it changed the background of a single page. And then we looked at an example where Obama was changing out an image and a button on a single page. You can also totally test entire pages against entire pages. So if you wanna do that, the first answer about the snippet, she's asking, do you need like the snippet on both pages? Absolutely, yeah. You should shoot to have the snippet on every page. And that's why using a plugin or a theme kinda box like I showed, that's the way to get that done. And then it'll just be on every page. And then as far as how do you do it? Like page against page, what's that look like? Yeah, you would set up the new page and then your experiment, instead of changing elements on the page, it would just redirect to the new page. 50% of people or whatever fraction would get redirected to your new page. And then you're just measuring everything from that point forward. Yeah, yeah, cause like the snippet, we should get it back up here because there's so much going on in this little line of code. The snippet, like what happens, this is good, I don't know if there would be an interest in this level of detail, but like somebody goes in their browser, they type in your website address, they hit enter, right? And then it pulls in that HTML. And then way up at the top of that HTML is the script. And so the script is pulling down the Google optimized script. And that script's got all the logic for all of your tests and all of your tracking and all the stuff that you've set up inside the Google optimized UI when you were logged in and creating things, which we will see in a minute. But it's pulling in all that logic and like running it on your page. And so that's how that goes down. So whether it's changed my button, whether it's redirect, whether it's listened for this click, like all of that stuff is happening on the page in their browser. Do that answer your question? It's doing a lot of stuff. Yeah, yeah, it's doing a lot of stuff. That's free. Is there a question over here that I skipped over? Yes. That made me a bit of a... You do, I'm sorry. Yeah, yeah, I mean, well what it means if you don't is you'll have analytics installed twice on your site. And I wanna say that could lead us to the point where we don't trust our data and now we're just wasting our time because we're looking at data that we can't act on. And that's really common, but I don't want that to happen to anyone in this room ever. So yeah, you do have to take it up. Sir. So when it comes to the randomization, how does that work? Is it just automatically built into Google Automation? Yeah, yeah, good question. It's asking about the randomization, right? Like the first time somebody hits the page where an experiment's running, they get randomly assigned to one of the experiences. So how does that work, Lay? How does that get handled? Yes, exactly. That's handled by Google Optimize. It's just a quick, think of it like a coin toss, right? It's like you visit the page and Google Optimize runs up at the very top before you even see anything and it's like, have I seen her before? Don't recognize her. All right, let me see. Eenie meenie minie moe. I'm gonna show you this one.  Here's moe, right? Just like you saw last time. So that's how that works and it's handled for you. Sir. I love this question so much. Okay, so the question is, is it better to test one thing at a time so you know how to attribute, I guess, the any sort of improvement that you saw, right? What was responsible? Was it the image? Was it the button? What was it? I'm gonna try to give a quick answer now and then we can go deeper on this, probably in just a bit, I think. I've got some slides at the end that we might reference because I think a more general question there is like, what should I test? Which is a question people usually have, right? What should I test? How big a thing? How many things? So I do wanna approach that but the short answer is my stance is, thanks, 30 minutes y'all, is you'll never really know that much about why one thing is better than another and so I prefer to focus on what we do know which is, hey, this works better and now where can we go from here? And so I don't think too much about necessarily isolating the source of an improvement. Early on, early tests, I think it's good to test wildly, crazily different changes against each other to cover as much ground as you can. Like imagine every possible incarnation of your site that's feasible that wouldn't be insane and throw as many variations out there as you can and so when you get a winner, like you won't know, okay, well this has a different design, it's a longer form, the copy's a little different, I've got these images, like you will have no idea. So I guess this was supposed to be a short answer, it's not going that way. Number one, get used to not knowing and get comfortable with not really knowing and just kind of focused on what you're learning and what you can do next. Later in the optimization program, when you figure out which parts of the site really have high impact, then you can do more minute changes and then you'll have a stronger sense of okay, here's kind of what's behind that change. Early on, big swings and don't worry about it is my short answer. Well, we will come back to this. Cool, okay. And just to check in, we're about 30 minutes in. I'm hoping for lots of questions, I appreciate the questions so far. I'm just gonna step through what it looked like to set up that little orange, red, blue, purple test and then we'll be on to questions. So that's what's ahead everybody, so okay. So having installed that snippet on my site and I love the questions about how the snippet works, like you should be asking those questions, so thank you. Click a button to create an experience. I think they call it experience, not experiment because you can also do personalization stuff with Google Optimize, it's a lot of the same, like under the hood, the technology's the same. And then you click a button to add a variant and so once you got a variant, you can click a button to edit the variant and so this part's kind of fun. How many people in the room are not developers? Not developers, many people, okay great, are not developers, so good news for y'all. The way it looks to build out a variation, there's only so much you can do but there's a good bit that you can do without writing code.  And then it's got a little widget that lets you modify things on the page, so you can change the styles, you can rewrite the HTML or the copy or whatever, so any kinds of change like that, like the Obama changes, you could build in Google Optimize, no developer needed, for example. The way it works is you kind of hover over elements on the page and you click and then at the top, see what it says, body, it's got a red box around it, I've selected the body, just the whole body of the page and then in the widget, I've selected background and I just type orange and hit enter and that's it, that's my test, there we go, y'all. Everyone who saw orange, this is the magic behind it, amazing, right? For those of you who are developers or work with developers, you can also inject custom CSS and custom JavaScript into any variation, so there's literally nothing you can't do, sky's the limit, you can rebuild the entire page, you can call APIs, you can build out new functionality, if you wanna go down that route, it supports that. So, we created a variant, we're gonna add some more, we're gonna make them different colors and then we have to figure out what we're measuring, always important, right? So, you can do this too within Google Optimize, all this is like I'm logged in to Google Optimize, right? And this is what I'm seeing. So, create a custom objective, you can also use goals from Google Analytics, if you've already set up goals in Google Analytics, they'll just kind of magically appear. And so I set this one up to measure that you just converted page that some of you saw, those of you who could not resist the draw of that click here, right? You just had to, yeah, y'all saw this page and it measured that page view and it attributed it back to whichever experience you were in, whichever variation you saw. And it's like, bam, purple got a winner and it's counting them up in the background and then it's, oh, I'm a little bit ahead of myself there.  Go hit your site several times in incognito and close it out and hit it in incognito and close it out, you'll see the variations, you can make sure nothing's broken. And then you're just tracking. So, kind of a mess, yeah? Everybody okay still looking at this? Anybody sweating it? No, okay. Yes, ma'am. Yeah. Yeah. Could you, when you're choosing the variants, could you choose an entire page, home page one, home page two instead of having to then create different color variants? Yeah, yeah, absolutely. So the question is, yeah, can you do, you can test a page against a page, can you test multiple pages against multiple pages? The answer is yes, yeah, for sure. And you should, if you can, like if you have the numbers, if you have the traffic, you should test more without a doubt. And so, yeah, absolutely. And you just have variant one is my home page, variant one, home page, variant two, all the way down to five or six is a good number if you can support it. Yeah. And if it's something that could you each serve over here, like create a test site and then do it that way so you won't necessarily have the big impact to users but they're already moving. Sure, sure, yeah, yeah, good question.  If we mess this up, we get in a whole lot of trouble, lots of eyes on it, like how do we contain that? How do we QA that? How do we make sure that people only see it when we want them to? So absolutely, I kind of skipped over it, but when you're setting up your experiment, you have really specific choices you can make about the targeting, like where does this run? And so I just chose one page, because it's just running on that one page that I sent everybody to, but you can give it like a URL pattern. So you could say only run on staging.mysite.com and it would never reach the production site. And it gets even more nuanced than that too. So you have a lot of choices with like where the test runs. You can also target only for people who have like a query parameter. So like question mark QA equals true.  So like there's all kinds of ways to kind of scope it down and control who sees it. Yeah, and yeah, yeah, that's really important too. Yeah, you don't just press go, even though I just said press go. All right, so what we have here is this is a different result set. This is from the Google optimized docs. I don't have all these results for what y'all just saw. So, sorry. For each experience, there's a lot of stuff here. I'm just gonna kind of hone on to this graph, which is a good thing to look at. It's going to give you your metric on the y-axis and then time on the x-axis. And what we're looking for generally is for these lines to kind of just settle down and steady out. A thing to pay attention to is like, look at this hot mess early on where everything's jumping around like that. If you had just called the test right here, then you would have called this blue line that was at the top, but we gave it time and we saw that actually another variation was outperforming that. And so this is kind of where we come back to that 500. Like you need those 500 conversions for this noise to kind of settle down. If you're making a call on only 100 conversions worth of data, you might be making the call all the way over here and you're wrong, right? Or over here and you're wrong. And so you put all this effort into building this test and you Q8 it and you launched it and you watched the results and you thought you had a winner and you're wrong. So I don't want that to happen to you. And so that's why you need to have a sense of, okay, do I have enough traffic? Do I have enough data to make this call? Oh, yes. What age differentiation? Yeah, yeah, good question. So she's saying it's, is it important that your sample is representative of the total population? Absolutely. You make the, if you serve a test to just a handful of people or even worse just to a narrow segment of your audience and you get a win and you're like, yeah, let's roll it out. Yeah, that's potentially really disastrous for sure. So if you don't want to serve it to all visitors at once because it's something sensitive you could serve it to a fraction but that'll be a fraction of the whole population, right? Like a random fraction. So 10% of all visitors, right? You don't just serve it to people who come in on a Friday or just people on mobile, necessarily, if it's relevant on desktop. Yeah, you want a representative sample. Does that implies that you do that? Yeah, yeah, you can control. How much of the total population is in the test? Is a, that's a little toggle you've got, sir. But you could do it at just a, Yeah, absolutely. So I just said, okay, careful about only testing on mobile, but he has a good point. Can you just test on mobile if it's only a test for mobile, if it's only about the mobile experience? For sure, you can target by device and the point there is if you get a winner you're only going to roll that winner out on mobile and so you're good, right? Because you have tested a representative sample of that audience. So yeah, and you do have that capability. Good questions. Not a lot of time. I would like to kind of go through this quickly. I just want to call out that if anybody read the blurb for this talk that you've so graciously attended, I said I was going to talk about a solution that's fully integrated into WordPress. So like real quick story there. Months ago when I signed up for this I just had like a prospect that I was chasing. Tell me like, we don't need your help. We're using Elementor. Anybody, Elementor? Yeah, and there's a plugin called Split Test for Elementor and it's super easy and like we're good, but thanks for asking. And I was like, okay, cool. So for this talk, let me go learn that plugin and I'll do a little demo and that'll be cool for WordCamp, right? So this was the plugin and this is me trying to set up y'all's demo with that plugin and so like, whatever, like not throwing shade, right? It could be user error, but I could not get it to work. This is my support ticket. So we're not going to talk about that plugin today. If anybody wants to know how it plays out, like get in touch, I'll be happy to fill you in. Haven't heard back yet. I looked at a few other plugins and I didn't really see anything that I thought was a better idea than Google Optimize. Like these ones, they look okay, but either not update it for four years or like a hundred plus installations. So like, I mean, it looks really good, but not clearly not like road tested. There's a couple others you have to pay for and for something that you're uncertain about that you have to pay for, I would just go with a free thing that has a lot of support. That's just me. So I didn't bring that to you.  Feel free to explore, but I'm telling you that the free one that's supported by a giant company that's being used on websites all over the place is good enough to get started and you can, you know, make a choice as you go along. So got some resources here that you can refer to if you're into this stuff. And then I've also got the next slide, a page, wait for photos, but then if you want the actual like clickable links, I'll make sure those get to you too. Good, good. Click, click, snap, snap. Yeah, okay. Actually, I should have probably just done this and made y'all take this picture instead. This is a link. This page has a link to the slides so you can get all those links that you just saw. I'm sorry, there's a lot of pressure up here, right? It's hot in here, right? Anyway, so this is resources. It'll link you out to the things you just saw and to the slides themselves. Thank you for coming, sir. Yeah, you said the 500 number a couple of times. Yeah. I've always understood that you needed a set number of traffic and a number of conversion. Yeah. But like, we do a lot of websites with, you know, click to call, like, on our site. Mm-hmm. You know, HVAC, so in fact, you go about recording ads on versions with Google a lot more. Yeah, okay, so good question. With two questions, kind of, like, one was about the 500 conversions number, right? I didn't say visitors, I said conversions, so like, what's up with that? And then also, sites where one of the main conversions is to make a phone call, like, how do you handle that? So, second question first, if a phone call is your conversion act, you can always measure that click, right? That's fairly straightforward to measure a click on the phone number on the site. But call rail, they ever use call rail or heard of call rail? They're right down the street, local company, and they do some pretty fancy, magical stuff where when the page loads, they swap out the phone number with, like, a redirect phone number, and then they can track it back to that visitor, even back to that ad campaign. And so when the call comes in, they send an event through to Google Analytics, and I'm assuming optimize. And so you can actually track, like, down to the, not sure how much it costs. Good question. I've only used it where other people were paying for it, but it's something worth looking into. And I'm not sure of an intermediate solution there, though. Otherwise, you're kind of just stuck measuring the clicks, as far as I know. And if you don't get the 500 conversion. Right, you can carry it out, but if you carry it out more than a couple months, it just gets to the point where do we trust this? This has been running so long, has seasonality and whatnot affected what's going, the goal is to have data that you can act on. And so if it runs too long and you stop trusting the data, you wasted your time. So yeah, that's why it's a rule of thumb. Are we at time? Yeah, good question. Do you need Google Analytics before you do Google Optimize? I will say yeah. Technically you can do it without Google Analytics. You can use it as a standalone and just measure goals within Google Optimize. I would say just go do it if you haven't. Yeah, you can set up a goal as part of your experiment. You can do little one-off goals for that matter, but if you're gonna start investing time and energy into running these experiments, it's worth the setup to have that analytics installed too. So not a requirement, strong recommendation. Good question. Sir, if you're with me. Yeah. Can you speak a little bit about how to incorporate this in terms of a business model? How much do you charge for this average? Oh yeah, sure.  Yeah, sure, absolutely. Number of options, so I mean, you could just charge hourly. This is a premium service, right? Like we're talking there's like web development, there's UX strategy, there's design, there's data analysis, so like hike your prices for that. A lot of agencies charge on a tests per month basis, so we will build and run two to three tests a month for $5,000 to $10,000 a month. And depending on your client base, that number could go up. I wouldn't take it down much. It's gonna be hard to deliver at a profit for much less than that. But does that help as a range? Oh, how much do you get up front? Yeah, I mean, I don't know like all of it. Ideally, but yeah, that's a whole nother talk, right? Yeah, yeah, absolutely. Yeah, so you had a question, I'll be right back to you. So it sounds like you might change your numbers. Yeah. Is that what I'm gathering, so you can write it too long but you don't wanna cut it too short. Yeah, and let me. Exactly, I actually literally have that into like appendix slides, okay.  Cause if you call it after two days, you're probably just jumping after shadows and making a decision based on noise. But if you run it for like six months, I mean, do you even believe that this test means anything anymore after six months? That's too long. Six months is too long, I'll say it.  If you go to the link to the slides, there's a link to this calculator. This is the VWO, AB test calculator and there's a link to it in the slides. And so what's your conversion rate? How many daily visitors, which you know that's just a function of your monthly, whatever, and then it'll tell you okay with two variations, it needs to run for 46 days. And that's acceptable for like a low traffic site like that, okay, month and a half, that's not too bad. Here's some different numbers. Much higher conversion rate, much lower visitor count, 39 days, so similar duration, right? Back to that question about is it about traffic or is it about conversions? It mostly comes down to the number of conversions that'll determine your test duration. That and how many variations? So this site cannot run five variations, it'll go too long. Yeah, good question. Okay everybody, we need to take questions either in the happiness, where do you want to go? Happiness bar, yeah, let's do it, sure. Thanks everybody.