Hello, and welcome back to the Sports Biomechanics lecture series, as always supported by the International Society of Biomechanics in Sports and sponsored by Vicon. I'm Stuart McCurley Naylor from the University of Suffolk, and today I'm joined by Tony Myers, who is a professor of quantitative methods at Newman University. And he's also chair of the Sport Exercise and Health Analytics Special Interest Group for the British Association of Sports and Exercise Sciences. And Tony is kindly offered to record a lecture on Bayesian statistics for sports sciences. This is a topic that a few people requested after Kristen Sonani's excellent lecture on statistics a few weeks back now. So I was really happy that Tony agreed to deliver that. The lecture is pre-recorded. So if anybody has any questions as we're going along, either drop them in the comments on YouTube or get in touch with either myself or Tony via Twitter, for example. And one of those ways we'll try and make sure we get an answer to you. Thank you. Thank you, Stuart. My name's Tony Myers, and this presentation is on Bayesian data analysis. The aims of the presentation are to very briefly highlight some of the issues identified in traditional approaches that we've tended to use in sports science, and particularly pointing to sources where you might explore these issues and solutions further. Look at one of the solutions to the issues highlighted and look at the benefits of Bayesian data analysis. Provide an introduction to some of the basic concepts of Bayesian data analysis. Look at perform and interpret some very basic analysis using some very user friendly software, JASP, that's free to download and constantly being updated. And then show an example of Bayesian analysis workflow that involves a little bit more complex analysis and slightly different software, just to highlight some of the things that you might want to explore further. There's been a number of issues identified with traditional statistical approaches. The particular importance was a follow-up to the American Statistical Association statement in 2016 about caution with the use of p-values. And this particular special edition of the American statistician in 2019 went beyond their previous claims, and this was an editorial in that special edition called Moving to World Beyond P is less than 0.05. What did they say? What did they add to their particular in their previous 2016 statement? Well basically what they said was it's time to stop using the term statistically significant entirely, along with variants such as significantly different P is less than 0.05, non-significant or other forms where you might highlight these things by asterisks in a table or in some other way. So what are the alternatives if we take their suggestion seriously? What are the alternatives? Well there are a number highlighted actually in that particular special issue and it's worth looking at those, including the proper use of traditional p-values. So there are a number of credible alternatives and one of them is the focus of this lecture and it's Bayesian data analysis. What are the benefits of Bayesian data analysis? Well one is the use of prior scientific knowledge. This is potentially also a double-edged sword because how do we agree what prior scientific knowledge is, but certainly it's way of incorporating what we already know about something, even if it's, and we do often know about something, even if it's only about the measurement itself, about what the measurement scale is in the problem we're addressing. We produce the Bayesian analysis distributions and uncertainty estimations in a distribution rather than in an interval and this has benefits for its communication, it has benefits for particular calculations of probability, which are very intuitively interpreted. So probability in the intervals generated a sort of like common-sense interpretations of probability. When we talk about a 50-50 chance of reining, we give a chance of reining 50% and that's how we generally talk about in everyday language probability and Bayesian probability allows us to use those type of probability statements. It doesn't rely on hypothetical data for inference, so where we, in a traditional analysis, match our data to look at the compatibility of our data with a hypothetical null distribution, it doesn't do that. Bayesian analysis allows for model comparison, I don't know tradition analysis does, but we've got a number of tools to compare models, to both criticize and explore those models in a very straightforward way. And complex models that are problematic for non-Bayesian analysis methods can be constructed. So these are some of the benefits. Like any other type of analysis or any other type of endeavor, garbage in, garbage out. If your methods poor and your research is ill-conceived, the best statistical analysis will not rescue the study and make your results useful. We use a sample to draw conclusions about a population and we do that either by estimation or hypothesis testing. We then estimate differences, relationships, variation or test a hypothesis comparing either null to a research hypothesis or compare hypotheses. We tend to make inferences about groups or actually individuals potentially. So we sample from a population of either measurements, so it doesn't actually have to be just people. It can be measurements over time of an individual. We're making inferences about that population from the sample and it's important to be clear what population we're attempting to generalize to. Subgroup, geographic representations, etc. Well particularly in Bayesian analysis where we're incorporating current knowledge, using prior knowledge before we collect the data. And that prior knowledge will differ depending on the population we're trying to describe. So basic principle behind Bayesian inferences is to start with prior knowledge and combine that with data in the form of a likelihood and that produces our posterior distribution. So here's Bayesian hypothesis testing different. Well first of all it uses a Bayes factor rather than a p-value. We've got Bayes factor 1-0 which tells us how likely the observed data are to occur under h1 compared to h0. So in favor of the alternative hypothesis or our research hypothesis compared to the null hypothesis. For example how likely there is a difference compared to how likely there is no difference following our intervention. Bayes factor 0-1 is a reverse. It's how likely the observed data are to occur under h0 compared to h1. So how likely is the no effect compared to an effect for example. So Bayes factor 1-0 we've got a scale. One means equally likely either way. Evidence doesn't favor either the null or the alternative hypothesis. As we go from 1 to 3 evidence starts to get stronger but really between 1 and 3 it's still anecdotal. It's really we don't want to make claims for an effect. When we go above 3 we start getting what might call moderate evidence above 10 strong evidence above 30 very strong evidence and above a hundred extreme evidence favoring our research hypothesis or the alternative hypothesis. And Bayes factor 1-0 is the same but in reverse. So where where we've got 1 it's the same thing. There's no difference between evidence favoring the null or the research hypothesis. But as we go to 3 it starts favoring the null a little bit but we can't make any any claims. Then we've got 3 to 10 moderate evidence above 10 strong evidence above 30 very strong and then above 100 extreme. So it's just the reverse of Bayes factor 1-0. Which you choose depends on what research question you're interested in. So let's take a look at a silly example just to look at how we interpret Bayes factor 1-0 in relation to a hypothesis. We're looking at whether the person's pregnant or not and the BF 1-0 suggests that we have extreme evidence for the research hypothesis of pregnancy. You can see the little maroon graphic the pie chart which is something produced by the JAS software and killer illustrate that so all maroon means most complete support for the research hypothesis. Take our next example. Presuming we have no prior information about sex differences in pregnancy in this case we're not sure. We have one which is in between it's a 50-50 chance in the pie chart maroon and whiter evenly distributed half and half so we're really unsure about whether this person's pregnant or not. Final case is the evidence suggests that the person in the picture is not pregnant and we have extreme evidence for that with the Bayes factor 1-0 of 0.003 almost zero. So let's say it works so we evaluate strength of evidence given Bayes factor so it gives us perhaps a little bit more than a P value. So does Bayesian parameter estimation differ from traditional parameter estimation? Well firstly rather than just point estimates we get a distribution a probability distribution of any parameter that we're interested in so mean differences between treatment and control mean difference in pre and post relationship values whatever we interested in whatever parameter we're interested in we get a distribution on that parameter that represents our degree of knowledge and the uncertainty around it. So this is what we get when we've conducted a Bayesian data analysis this is a posterior distribution so it's a Bayesian posterior probability distribution and this particular example is differences in sprint times between a control and treatment group so we've got a range of difference potential differences here from minus 2.5 to probably almost six seconds difference between those two groups and we've got density on the left. So what does this mean? Well the higher the density of y the more probable that x value is so in this case something like 2.45 seconds is likely to be the most probable difference conditional on the data that we had so the most likely probable population estimate we have is likely to be 2.45 seconds in terms of the differences between the control and treatment group. The lower the density of y the less probable that x value is so this is much lower so zero is much less likely because the density of y is much lower than the midpoint of that distribution in which you should look at. With the Bayesian credible interval with this posterior distribution we can say there's a 99% chance the true population difference is contained in this interval this particular case minus 0.09 and 4.56 seconds. Now of course all of these percentages are completely arbitrary values we've got traditional sort of value of 99 and 95 and we use for traditional statistics but that they are purely arbitrary values. It could calculate 95 percent a 95 percent chance the true difference lies in this interval between 0.27 and 4.19 seconds calculated 90% chance and 80% chance and these arbitrary values we depending on what we want to calculate is very straightforward and easy to do. Another property is that we that's very useful is we can get the probability of a difference above any particular value we want so if we think if we've got an idea of measurement error or of a particular above a particular time difference would be really important in the context we're looking at we can very straightforwardly calculate a probability of a difference above a particular value. What this allows us to say which we can't say with traditional statistics if we have an effect for example we could say there's a 90% chance this intervention will be effective increasing in strength or there's a 98% chance this intervention will be effective increasing speed depending what we're looking at and obviously depending on the size of the effect but we've got direct probability statements about the effect and that's different we're not comparing it to the null we obviously can balance the null and the research hypothesis evidence for both with base factors but when we're trying to make parameter estimations we generally can get direct probability statements either way of course it could be 50 50 either way it could be any percentage we find but we are we can say that and communicate that with in the language that people tend to understand probabilities that make sense in a practical way to practitioners coaches and even researchers. Now for the symmetrical posterior distribution we just looked at the two types of interval the two types of credible interval I'm going to talk about now would be exactly the same so exactly the same values calculated in any of the intervals that we credible intervals that we calculated however that's not the case for skewed distributions as we'll have a look at in a moment so the two types that a credible interval are generally calculated and reported are the highest density interval or HDI sometimes called highest density posterior interval or highest posterior interval and this captures in the HDI all points within the interval have a higher probability density than points outside of the interval and that can be important particularly when we're looking at skewed distributions because we will get differences the other type of interval is probably familiar you're perhaps more familiar with really in terms of a traditional confidence interval the way it's described at least so this is an equal tiled interval so depending on what sort of percentage you talk about so if we talk about a 95% equal tiled interval or ETI there's 2.5% of the distribution on either side of its limits let's take a look at what that looks like and how they differ when we've got a skewed distribution so this is the 95% higher density interval of the difference in sprint times if it was a skewed distribution if we compare that with an equal tiled interval we see that the values in the interval differ and in this case because it's in the values that were measured so the non-transformed values the HDI the 95% HDI is the best units to use in transformed variables ETI will be the best bet but generally higher density interval is the best bet when we're using the units that we've measured so Bayesian analysis allows us to determine probabilities associated with both the null and the research hypothesis and that's something we can't do with traditional hypothesis testing because we assume the null's true so all the probabilities are based on assuming the null's true in the first place importantly when we're doing parameter estimation we get intuitively interpreted credible intervals let's look at some basic data analysis using JASP software so really do want to big up this software it's not only free to download and use but it's constantly being updated okay let's have a look at the data first now this is simulated data it is based loosely on a study that was conducted but is the data that was just loosely based on that in terms of me simulated some values so the actual study involved participants being blinded from each condition and the amount of capsules caffeine capsules received with the same irrespective of the dose so they got either placebo or three milligrams per kilo body weight of caffeine and the dependent variable in this particular instance is three counter movement jumps this is the data this was the data looks like spreadsheet it's the JASP software so we're going to look at JASP certainly look at i'm going to look at this software as I say there's a the range of traditional and Bayesian tests and you've got equivalence testing you've got mixed effects models there's a range of things you can do with it now and it's constantly been updated a great supportive network for the software so I can't recommend it highly enough really it's really excellent so we enter the data into the JASP package using a csv file so save the excel sheet as a csv file one of the options in excel and it looks very similar to the spreadsheet now we'll go on to the menu t-test menu and select Bayesian paired samples t-test i'm very similar to you might do in sbss or some other similar type of software i'll click on the variables and transfer them over to the analysis window and i get some output in the right hand pane if you like i'm going to add prior and posterior information visually because i think that's useful and i can also click some additional descriptives to have a look at what's happening so it looks like in the descriptives certainly the caffeine had an effect a bit more variation in it and higher standard error what's that come out as when we look at the the graph so we've got the the dotted line gives us the prior in this case it's a Cauchy prior centered on zero and a 0.7 approximately scale so we've got a fairly broad prior but it is centered on zero and given that prior we get this particular posterior there so that's a posterior distribution with a continuous line rather than the broken line and the base factor tells us that the data rate times more likely into the research hypothesis than under the norm if we wish to summarize about the differences and this is in terms of a standardized effect size so the effect size is similar to accounts d effect rather than rule values so we can summarize and say that the differences in conditions there's a 0.6 of a standard deviation difference so if you do like i'm not fun of it but if you do like the sort of standardized the sort of categories for effect size that would be a medium effect we can also get that that reflect that's reflected in the middle of that posterior distribution the median value of the posterior distribution that's why it's called median and then we get the the 95 percent credible interval which tells us there's a 95 percent chance that the true difference lies between 0.164 and 1.120 look at what that looks like visually i really would suggest that websites really excellent it also provides a range of other simulations but you can actually move the scale on this level look at what different currencies look like what different standardized differences look like and this particular point i've just moved the scale to 0.63 and what this suggests is i think also gives some other interpretations one useful one is the current u3 which tells us that actually this size of difference in terms of standard deviations means that 74 percent of the caffeine condition are above the meal mean of the control condition so it gives us a sort of common sense interpretation of that particular standardized difference if we were to run a standard pair samples t test on this we'd see that we've got a p value that would suggest that the data isn't really that compatible with the null so we'd say the data would be surprising i guess if if there really was no difference between these groups so in that way a similar conclusion in essence and we've got values of our confidence interval those are different because they're not standardized differences those are in more differences so we've got a moderate effect in terms of our base factor now we can do some additional stuff which i think is really useful to have a look at the effect of different widths of priors so if we have a look at the base factor robustness check all the priors even even a very wide priors comes up still in this moderate categories another interesting analysis we can do or check we can do is the sequential analysis so this tells us how evidence shifts with each data point so we can see initially there's some evidence for the null hypothesis but then as more data was collected evidence became stronger for the alternative hypothesis for the research hypothesis so let's take things further and conduct some additional Bayesian data analysis with some different software we'll mention well i'm here all these different softwares that i use and i think are really excellent so just we've mentioned but also stan i use an interface with r but you can use it with python and inla integrated nested Laplace approximation which is really quick i tend to use but not always but but often use the brms package in conjunction with stan because it's really flexible and helps you produce models quite quickly so let's have a look at the efficacy of a post game recovery protocol now this is simulated data but it's not completely made up i've tried to use some values that i've previously got from studies i've done in the past so groups involved treatment and control group randomized that's not actually always possible in applied context but for the simulation it's no problem measurement baseline two hours post game and then 24 hours post game dependent variable 30 meter sprint speed so this is data that's been simulated but is not you know is similar to some data that's actually been actually gathered some actual empirical data so let's have a look at the efficacy of the post game recovery protocol in this case for basketball players workflow in this sense i'm going to explore the data visually fit different plausible models based on my visual exploration of the data do some diagnostic checks compare models using two different systems one base factor the other one leave one out cross validation then i'm going to look at a you might extract some key information to address the research question explore the data visually so simply make some box plots of the three measurement points in the two groups so we can see here the baseline the pre-measurement the baseline measure treatment groups slightly faster slightly higher sprint speed meters per second both groups drop off in the post measurement and surprisingly post game and then both recover a little bit it looks like visually that it may be the case that the treatment group seems to be doing better but we actually have to look at what the outcome is of the the analysis also want to plot just to have a look at the density of the sprint speed the y variable the dependent variable in building the model i'm going to use a response distribution now we'll use different response distributions like a Gaussian distribution maybe a t distribution but i'm going to use a skewed normal distribution given this particular information and see whether that actually proves to model the data better than the other distributions one of the advantages of Bayesian modeling is that I can do this very flexibly so like in all data analysis there are a number of subjective decisions need to be made what i'm going to do in this as an overall model even though i'm going to change things like the response distribution and different prior information what i'm going to use is given i've got baseline measures for both groups i'm going to compare post measures at the two hours and 24 hour measurement points using baseline as a covariate so it's post measures conditional on baseline condition on baseline so i'm adjusted for baseline so this is similar to an uncover model and helps potentially avoid regression to the mean particularly given all that kind of affected all the treatment can affected is the post measures it can't have affected the baseline measures so the uncover model seems to me to be an appropriate model i'm going to use different priors so i'm going to use a weakly informative prior and this is a mean of zero and a standard deviation of three so it gives me a broad range of differences they're actually very unlikely so it's unlikely i'm going to get minus 10 meters per second difference in a 30 meter sprint between the groups or plus 10 meters per second so this is really weakly informed it's not a uniform prior so it's not putting probability across all values but it is but it's still very broad as far as the prior for the intercepts and the standard deviation go i'm going to use the BRMS prior so i'm going to use the BRMS package to fit this model and i'm using these as prior so it's a t distribution on the intercept so it's allowing with the long tails allowing values that are quite broad it's a very i found it useful in a number of models so i'm going to stick to the default model very similarly to the standard deviation which is a half student t doesn't allow me to have minus standard deviation so it seems it's a reasonable choice and works well as a default measurement constrained prior what this is what what i'm trying to get out here is i've tried to think about what really given this is a group that's been randomized to control and treatment what what likely differences are they're going to be in sprint times and i don't think they're going to be that much more than say 2.5 meters per second either way so i've centered this as a mean of zero and standard deviation of one to try and capture that again i'm going with the defaults on the intercept and the standard deviation my informative prior really based on some previous data so there's some previous studies that looked at similar thing and known which way around my groups are in the regression model then i'm going to put a mean of minus zero point five and a standard deviation of one to reflect the sort of general differences that have been found on average in some of the previous studies again i'm going to use the default priors on the intercept and standard deviation so i've got these weekly informative prior differences modeled by a normal distribution, the intercept standard deviation by the defaults in the package, measurement constrain model and the informative prior based on previous studies so these i'm going to include in all the models so i'm going to check someone do lots of different models in essence i'm also going to model this on a normal response distribution so as a normal response distribution a skew normal given that the shape of the y variable when i'm looked at the density of it i'm going to allow individual intercepts to vary because it looked like there's actually quite different sprint speeds across both groups and i've got several data points for each person and i'm given again it looked like there were differences in variants across the groups i'm going to allow for heteroscedasticity in the group so again i can build that into a model quite straightforwardly in the brms model or in stan run the model so this is some of the plots for the best best model in the end and so i've got some trace plots see whether they visually there's any problems even though just a visual inspection won't necessarily identify problems i can see if there is a problem i won't be able to say just because it doesn't look like there is that there isn't so i also do want to do some other checks and the r hat which comes out of the as an output in in stan and the the brms package the and some other packages as well allows me to say well if i'm looking for an r of around 1.01 but anyway it's certainly less than 1.1 and it appears that they have all converged as much as we can tell at least so it's likely that that that that's past that diagnostic check i'll point out the leave one out cross validation package but this allows me to produce this plot and have a look at whether any influential values then potentially outliers but they're influential values that might cause problems there might be problematic for the for the in terms the estimates potentially 35 dates point 35 isn't possibly 3 and 11 but i'm going to leave those in but i certainly could remove them if this was a you know a real analysis then i'll look at posterior predictive checks so i'm looking at whether the simulations in the model actually match the data so look at is it a real reasonable match for the dependent variable the y variable and it sort of looks like it is so those lines those lots and lots of lines are 100 simulations from the posterior distribution and the the black line is y so y versus y rep and the empirical cumulative distribution function as well in the same way it looks like it's captured it and then i can go down to the means and standard deviations of the groups on the bottom plots and it looks like the simulations seem to have captured the descriptives in the data so i'm pretty happy that's captured the data well the data obviously i know the rest of its random variables but i do know the data the data are fixed in a Bayesian model and i know what that is so i'm going to use that as an anchor to see how well my model performs the hubby criticisms of this so it's worth if you think you know you want to explore this further certainly look at some of the debates about using posterior predictive checks now for comparing models initially i want to use a base factor so remember there's three priors now the models that were best was a skewed distribution with random intercepts on all of the all of the three priors but now i can compare which prior is better if you like which supports the data better and comparing initially the measurement constrained prior with the weakly informed prior the measure constraint the measurement constrained prior that was restricted to sort of values around not restricted but putting probability on values around minus point plus or minus 2.5 meters per second suggests that that that model is 18 times more likely under the measurement constrained prior compared to the weakly informed prior so the measurement constrained prior came out as a better fit strong evidence if you like on a base factor scale now the data for the measurement constrained prior compared to the informative prior interestingly is even stronger so there's more evidence that actually the the measurement constrained priors a better model if you like of the data than the weakly informed prior sorry than the the informed prior which is interesting itself actually suggests that the data don't seem to be reflect previous differences in other studies now comparing models using leave one out cross-validation so it's a different system of comparison so if you choose base factor but now this is a different system and there's if you again if you want to read the literature on this you'll read literature on what we call open and close models so have a look at that if you want to look at what which one you might want to choose so loo I see the lowest value is the best value and it's a minus value so obviously the case in this case minus 19.4 is lower than minus 11 or minus 616.1 so loo I see sort of in that way similar to AIC or Akiiki information criteria and if you are used to comparing traditional models so that seems the weakly informative price so it's a bit different than what we came out with the base factor seems to be the best model now comparing the the expected log predictive densities what I'm interested in here is not just the difference but also how this difference compares to the standard error of the difference look it's looking like the measurement constraint model is not quite as good on on the the loo leave one out cross-validation whereas it was with the base factor so there's some differences in terms of that so these look like anyway the best models clearly the informative prior didn't come out very well in either model comparison method this first plot is a standard plot from the brms package a conditional effects plot on the y-axis we've got sprint speeding meters per second and on the x-axis we've got measurement times we've got two hours post and 24 hour post game and we've got the groups so we can clearly see um treatment and control group now what do those error bars represent well the error bars represent not standard deviations but they represent the credible interval so they represent the 95% credible interval point is the the most probable estimate for those particular groups so we can see quite clearly the treatment group are on average faster than the control group in the two hours post game measurement and also in the 24 hours post game but the uncertainty you can see the larger credible intervals there's much more uncertainty associated with that and they're clearly strongly overlapped we can plot distributions so have a look at that and i'll plot those next to pair wise comparisons so here we've got a posterior distribution plot of the differences the pair wise differences between each of the conditions and time points the dotted line represents zero this yellow distribution is our prior distribution and these blue distributions are the posterior distribution of differences so the peak is the most likely difference unfortunately if we're doing this really we might want to question some of these not particularly it's not so bad on the the questions we're asking but some of these are not they're multimodal points so we might have some issues with that but i'm not going to rerun the model this is just for illustrative purposes so what we've got here is one of the points of difference we want to know whether at the post measurement do the control and treatment differ there's a hundred percent chance the difference is less than zero if we look at the the second one where we've got 24 hour treatment and control comparison well there's more of an overlap and so it looked like there was on the conditional effects plot and this plot suggests that as well we calculate there's an 88.4 percent chance the difference is less than zero which means that there's a positive effect because we're taking treatment away from control and so whereas it's clearly been effective above zero two hours after the game it's more uncertain as we saw on the initial plot 24 hours after the game so as i mentioned previously it's not just above zero that we can calculate the probability of we can calculate the probability above any particular value and we certainly we can know there'll be some measurement error we might have some a better idea in fact of what an important difference is and to illustrate that i've just gone with a 0.5 difference so in this case that 0.5 is standard deviation difference between the groups i've set if you like as a region of practical equivalence we might say that this is a region that's equivalent to zero because above this is the values that i consider to be important so i'm saying that this is equivalent to zero because of the measurement error because of some value that i realize is actually important i've i could choose any value i've just chosen the standardised difference to 0.5 but i could go with a raw difference can calculate whatever you want to so it's difference is the probability above a medium effect and so this is suggesting there's a 99% chance or 99.6% chance of a difference so that the treatment group's been better when we go down the 24 hours what we've got here is a 68% so almost 70% chance so there is some evidence of a difference but we can't really claim to be sure we'd have to say that we really are unsure about how effective it is 24 hours after but we're pretty sure that it's effective two hours after so what can we suggest to practitioners well quite simply we can just use the actual direct probabilities because they're actually quite reasonable to understand so there's a 99.6% chance the intervention is successful in improving spin speed after two hours then if you didn't do it so the intervention is worth investing in if that's an important thing so where you're in a tournament and two hours between games this intervention is well worth using and investing time and resources into a quote from John Tukey I think is really pertinent to Bayesian analysis he said it's far better an approximate answer to the right question which is often vague than an exact answer to the wrong question which is always which can always be made precise so i think with Bayesian analysis you can focus on the things that you want to usually ask that is how effective is my intervention being what's the probability that that i've got an effect what's the probability an effect of a particular value i think those questions can be really usefully addressed with Bayesian data analysis so to review prior scientific knowledge can be incorporated into Bayesian models very easily that's possible with classical models but much more difficult in fact this is the basis of Bayesian models I hope you've seen that the results of Bayesian analysis are very intuitively interpreted we can talk to practitioners and coaches and other researchers and readers of our research in ways that they usually intuitively understand in terms of probabilities that they use in the everyday language there is a 90 chance that this intervention works for example and complex models that are problematic for non-Bayesian analysis can be constructed and used the last example was a series of different response distributions allowing for heteroscedasticity so we modeled sigma across the the different groups it allowed random intercepts and in some cases random slopes and we incorporated different degrees of prior knowledge so a certain level of complexity I hope this has encouraged you to have a look more into the possibility of Bayesian modeling your own analysis and that at least you enjoyed looking at a different type of statistical analysis method thank you ever so much again Tony for that and thank you for jumping on a call again now and for some follow-up Q&A after the pre-recorded lecture as well and yes I think the first thing I wanted to discuss really was what advice you've got for anybody wanting to start doing some of the things you talked about in your lecture yeah thanks to um but one of the things I'd say is the JASP package is really user friendly I mean I know SPSS offered Bayesian sort of common tests now but I think JASP is really sort of initially focused on that and I think it's useful even where I think one of the useful things you could do is even where you've conducted research before or using literature there's a there's a there's actually one of the I'll just find the name of it actually it's one of the where you can actually look at taking previous results and reanalyzing it it's called the summary statistics model in JASP and I think that's that's a useful one for simple analysis like correlations and I just to see you can play with the prior so you can you can look at the the plots to see how that shifts with different prior assumptions I think that's one of the perhaps a very easy way to get into it if you've conducted research or a very familiar research that's done fairly straightforward difference tests or relationship tests you can use those direct statistics in that summary statistics module and it will give you an idea of how different prior assumptions would shift so you know it's essentially you don't have to do any analysis that's all you put it in I think that's a useful gentle introduction and then following that is actually just data you've collected just to have a look at some of the basic tests because they're sort of very similar that it's it's user friendly in the sense that they've got t-tests paired t-tests and over and covert regression now they've added mixed generalized linear models mixed models and and mixed effects model generally so you've got you know it's it's growing and it's University of Amsterdam and they do constantly update things when a new a new program comes out you know a new update comes out there's several different modules you can add to that and so I think that's a useful easy way to get into it because it's quite familiar for those who've used traditional statistics they're also traditional statistics on there so it's very easy to compare results you know with doing a you know if you do an independent t-test traditionally then you can look at how different that might come out with different priors on a Bayesian t-test so that's probably an easy way in really okay yeah I think I'd second that as well having had a quick play around myself I've not used the summary statistics module but just I'm also a big fan of just like I think it stands for Jeffrey's amazing statistics package something like that um yeah it is as you say even if it's for frequentist or traditional statistics it's very very easy and user-friendly and like you say if you use the default settings it is as easy as instead of clicking independent t-test you click Bayesian independent t-test or instead of clicking regression you click Bayesian regression um but I guess I was going to ask this question later on but that's kind of led into it maybe something I've been thinking in my own research bit is just too easy do you think so with all the default settings for Bayesian analysis is it too easy so just a bit of a background there my own journey really with this I've initially started using some Bayesian research but then I'm almost at the point now where I've realised I don't understand this well enough and next time I use it I want to do some more of what you did in the lecture of actually playing around with different priors and I think potentially yeah it's very very easy is that necessarily a good thing or are there some negatives of that it depends what you want I mean it's the model types are limited so you've got a Gaussian distribution essentially a normal distribution for those models it's limited as well to so in a difference test to a standardized difference so the priors are standardized differences so when you you can use informative priors that's probably something that extends to other models so the moment you can use informative priors on t-tests for example but it is done that standardized difference so where you know you want to do something a bit different where the distribution and they're all symmetrical distributions where you want to do something a little bit more challenging in terms of modeling you might want to look at other programs I think the benefit pedagogically of using something like stan even though it's a steep learning curve is you really have to specify the model so you specify priors on everything I think a gentle introduction to that is the package I mentioned in the presentation Paul Bertler's BRMS package or our stan arm which is the other stan linked package which uses our code so of course you've got to make this this is only easy if you use r but the code's very similar to to using other packages with frequency statistics in r so it translates very easily you can also generate code from BRMS packages to have a look what stan code might look like it's not exactly like you write it in raw code but you can see what priors are on what elements because whilst you might in in just choose just a prior for the standard standardized difference really there are also priors on if you're building a multi-level model on the intercept all right you know so what's the beta value what's the standard deviation you're going to put a prior on now the the defaults produce you know a useful place to start because the defaults produce very similar on similar in models to frequency statistics that's what they're designed to do really so you know you can just interpret those as a basing in a basing way but in essence it gives you that once you've got familiar with it gives that chance to be able to play with different priors and it's very useful in terms of priors really I mean it was a crude example that I gave in in the in the presentation just a limited example of saying well this is the difference so I didn't look at the intercepted in you know a standardized difference across all things choosing just sort of a confidence interval from frequency statistics and a measure of central tendency and mean difference but you can do things much better in the way you can do where I showed an example of posterior predictive checks you can do prior predictive checks so in the code you just put sample prior only you know you can you can it'll allow you to play about with how do priors influence results sometimes in fact the example I gave it actually made them more conservative because it's you know it's fixing it on the difference so it made the differences a little bit less they could say well even though that didn't come out as the best model that may be the best choice because maybe my data's you know that data's more substantial it was taken from a metro analysis or it was taken from a series of studies rather than my study so it depends where you start um but that's sort of I think that flexibility and understanding comes from you know having to once some model a little bit more complicated probably gives you that idea of you know how what are these things actually and do you know what the what's the prize doing and how do they influence results I think that can be useful so I think just is a really excellent start it's also gaining in terms of complexity what you can do and that who knows where that will end up but to do things in a little bit more detail and perhaps pedagogically to understand things even if it's a bit of a steeper learning curve something like using stan or python or some of the other where you you know integrate me stan are or python integrate me stan whichever you're comfortable with and depending what background you've got I guess um let's see you know those packages like brms or arstanama useful introductions to that I think but and in terms of text on that I think a very good pedagogical text in terms of stan even though I do like john kruski's book on doing Bayesian natural analysis that's probably I've done a lecture with him in in Oslo and he was excellent he's really good um Richard McAlrith's book called rethinking statistics they're both on these are on second editions now really start very basic building up model so it's really interesting pedagogically to build up models to start with he uses his package but you know it's his stan is extended code for a range of other sort of like python etc so I think you know that's a really useful introduction is it quick not necessarily you have to work through it work with it but I think in terms of understanding that it is really good brilliant thanks yeah I think very good point you made early on in that answer was around even if you use all of the default settings you're essentially doing something very similar to what you would in a frequentist analysis anyway and I think yeah from when I've played around with it generally it's slightly or a slightly higher level of evidence if you use something like a base factor of three as you're cut off generally you get very similar results but just a slightly higher standard of evidence so I guess yeah it is quite comparable and yeah another worry that maybe you can help put to bed is when you start I think when people see you playing around with priors they maybe start to think is this getting a bit subjective so rather than just saying oh I've ran a t-test you're now saying it depends on the decisions I make around these priors can you maybe speak to that a little bit yeah I think I mean it's I mean one of the things I think it's useful is actually running different priors and seeing what effects it does have on your data and he said mostly there's certainly some of the analysis I've actually really done which tend to make the differences relations more conservative than if I use the wall value so it hasn't inflated things it's on the opposite it's sort of and I've generally used priors really that have and what am I can this is not a technical term need something I've used in the presentation called some measurement constrained priors that's not a technical term really what it is is to say that I've looked at what the possibilities are in measurement so I don't know in in the example I gave I'm not an expert in sprint speed times but but I'd say that really I think maybe the highest speech humans can get you know I'm if I if I say that the biggest difference could be the highest sprint speed somebody could get and say somebody fell over that's the maximum they could be so if I put probability over that then I'm very safe in in sort of not gonna you know not skewing anything nobody's going to disagree that people could run you know 20 20 meters per second you know so you know I'm going to sort of constrain it within the measurement framework and that tends to actually make things more conservative an extreme value is not going to necessarily pull estimates so I think um I'll to answer your question really I think I'll try out different priors you have to obviously justify it's one of the things you need to do say you know if you don't know much about anything a weekly informative prime what that means exactly useful um but try those things out you know in one paper I definitely produced estimates of different priors because it shows in fact that actually the prior just made it more conservative and and a frequentist analysis would have been like a uniform prior and it would have been a more extreme result than the actual Bayesian analysis with a with an informative or you know measurement constrained prior so I think don't be frightened of priors you know it's you're not trying to compete I mean I'm not saying that you can't with Bayesian analysis do like you can with frequentist analysis choose the date you want to do you can still manipulate things if you're that you know of that mind but but of course if most people are not most people are honest and they want to present things and generally present evidence so I think I'm going to look at the effect of different priors is the best way of doing that because sometimes it you know might be that you know that you've got a lot lots of daters priors get washed out but where we in small size use quite small sample sizes given the sort of way you test things lots of you know lab tests and etc can be quite intense can be difficult to get really big data sets on that so using prior information can be quite useful in that but seeing how it affects your results I think is one way not to be frightened of that this is a subjective decision that it's actually going to make inflate things I think often in practice it does the opposite yeah I think again the excellent point you made there was about just defying the priors and it's the same as any decision you make in your statistical analysis so I guess any of your methods but if you're justifying why you've made each decision hey people can follow that but also it's there for reviewers or readers to evaluate and I think yeah that's fine then yeah and the other point you mentioned was about some of your previous research is for the benefit of the people kind of watching this some examples of papers where Tony's been involved in used basing analysis the links to those papers are below the video so if you want to go and have a look at some examples then they're down there as well and yeah I think probably the last slightly technical question I had was around again priors but getting information for your priors from the literature so can it be as simple as another study in a related area found this parameter estimate and confidence interval or this mean and standard deviation so I'm going to use those values as my prior or kind of is there anything more complicated than that that needs to be done it could be as simple as that students I mean it can be depends really on what model you're producing so it can be as straightforward as that but what I'd say is it's really worth doing those prior predictive checks to see where that priors reasonable because in things we might not think of an impact can so I think it's it could be just those you know just doing a straightforward and then the example I gave was that is a very basic sort of difference between groups you know controlling treatment group previously so it could be as simple as that but I think that realistically you'd want to check that you also want to check that against other priors you know so what what you know a default prior what what would be changing that prior slide you remember that you know we're trying to get that that has still be essentially often descriptive you know if it's a frequentist analysis it'd be really their mean difference for example and with a confidence interval on it and and we're trying to estimate often trying to estimate if you like a population estimate the data generating processes or whatever you want to choose to phrase that but it but so that may not be a good reflection of that one single study so it's you know it's still you want to edge your bets a little bit not make it too constrained make it a little bit looser but have a look I think the thing is simulating stuff where you can have a look at what the influence of particular things has on the analysis and the outcomes as you see if you have a look at just you see that the example I gave was all of the all of the priors made no difference in terms of the conclusions so I think you know looking at that does where it does change your conclusion that's where really that needs a bit more scrutiny if it doesn't well you know there's not really an issue particularly but I think where it does change it you know it's often that's where the problems may be if you're making a different claim than the rest of the priors so what you know that's where you'd have to clearly justify the prior clearly justify what you're doing so it could be as simple as that but I think you need to check and see what against other things and check you know in terms of prior predictive checks etc okay so yeah you just then you mentioned about the possibility of it changing your conclusion which is just another thing I was going to mention just to be annoying really um but you said at one point I think in terms of frequent statistics you're not a fan of categories of effect size and where so when we then in terms of base factors when we then say kind of below three is anecdotal three to ten is moderate above ten is kind of strong very strong extreme etc like is that the same thing and are there any limitations of saying 2.9 isn't an effect but 3.1 is etc absolutely absolutely yeah of course anytime you put a category on a continuous variable you get those things like 2.9 and 3 what's the difference very little really I mean I'm in terms of standardized differences I'd rather interpret in some you know some way like commons u3 where you can actually be a percentage or if people don't understand standard deviations difference for example I think base factor the same it's how many times more likely so I think in a way you know you can say you make a judgment and claim based on that so if something is a hundred times more you know the data a hundred times more likely under the research I pass in the north I don't think anybody would argue that your data that's that's the case with your data and the reasonable conclusion where it's 2.9 or where it's 3.1 there edges to a category in there that's certain of course you know so yes the hard bound categories are problematic I think always on continuous variables and our caution against those particularly I just think in a way it's like I agree you know people want to know what what does this effect size mean and you know it's okay it's okay initially I think to have a look at those in sort of categories but there is issues with them and I think the same with base factor I think to introduce the idea of it it's useful to use as an absolute any mechanistic decision making tool and really much against generally okay um yes so I've got one more question before well I think um is this might be a yes or no but with Bayesian analysis do you need to correct for multiple comparisons like you would with frequent tests? You do if you're doing hypothesis tests so I think um in terms of I think JASP offers some comparison I'm not sure it does with base factors actually so yeah you in a way you can do that I think if so if you're doing base factory it's a hypothesis test you won't need to if you're doing parameter estimates necessarily because you're not doing the same thing you're trying to get the best estimate of a difference so where you're trying to make a decision and for against the null hypothesis then in a similar way you can I think JASP I don't really do that much hypothesis testing to be honest but I think if you look on JASP there is some controlling for multiple comparisons but you'd have to check where you're okay and then yeah I think the last thing I was going to mention you've probably covered most of it because I know when we spoke before I said about giving more information to readers or reviewers so that they can confidently evaluate a paper that's used Bayesian analysis even if they're not used to doing it themselves I think you've probably discussed a lot of things that would be useful there and especially things like justifying all of the decisions and the priors but is there anything else briefly kind of that you can think of to add that would help people to decide whether they're reading something that's done Bayesian analysis the right way or the wrong way I think the assumptions generally so if you use a linear model and you'd use a sort of likelihood the assumptions don't change they just replace basically you know the random variables but essentially if you're making a claim of the so there's similar assumptions the same frequentist assumptions you know you might get what error distribution you have around you know so you know if you're doing a regression for example you might say your errors on the regression to get the confidence intervals are Gaussian you know Gaussian error distributed the errors are distributing in the Gaussian manner you could you can do that the same on a on a Bayesian analysis it's you know you use a distribution but it's a Gaussian distribution I think it's easier to put different distributions in and check so you could do skew normal you can also look what a t distribution response distribution looks like how do those but but essentially model checking in a way is the same so if you're making claims of this is a linear model then still those assumptions hold so you don't know but you know you don't suddenly magic away some of those things those claims so you still have to justify those claims you know of of independence or whatever else you're doing so multi-level models or you know whatever type of model they say similar assumptions in terms of the model hold still you might still do some additional things about as I said posterior predictive checks etc prior predictive checks obviously you're in addition approach is different but in terms of the likelihood that's the same as it is for a frequentist analysis so don't you know so if they're trying to pull the wool over reviewers eyes in a way and saying I don't need this the other thing is I guess where they say it's okay for small samples it is but it's not if you use a very very weak prior it doesn't really you know it's still not that much better than a frequentist analysis so on the whole can it be useful for small samples it can if you're using some reasonable prior information because what you're doing is using that and then adding your data to it whereas if you're you've got a very few subjects and you're not using that using a very weak prior uniform prior it's really not that different than than doing a similar thing with a frequentist analysis so it's not magic if you've done bad data collection then you're going to get bad results excellent yeah completely agree and I think yeah that last point's really useful about because you do hear within sports science especially a lot of people with issues around magnitude based inference and things but saying oh okay now I'm going to use Bayesian analysis for small samples and that's a very good point that the benefits are around the prior whereas if you don't really have any prior information then you've still just got kind of a very small sample and you're not adding any extra information into that because there is no informative prior yes okay excellent well yeah thank you ever so much Tony it's been really useful and thanks for joining for the extra chat now as well thank you Steve thank you and just yeah for everyone else I know we've had it on the screen but just keep an eye out for the last couple of lectures over the next two weeks with and Kristen Sonani who gave an excellent lecture on statistics but she's now back by popular demand and doing a talk with some tips for scientific writing which is something else that she delivers online courses around and then I'm really excited for Walter Hertzog's lecture on muscle mechanics as well to finish it off yeah thanks Tony thank you very much thank you cheers


Okay, so this thank you page is what it's all about. What we'll do today is set up a custom conversion. And as I kind of went over a few minutes ago, it's all the same, whether it's a listing, an open house, a buyer guide, a seller guide, it comes down to these two pages. And so with Facebook, what they're really interested in knowing is out of the two, which is the most important. Like, which is the most important? You have two pages, which is the most important? What most people would think most people would think it's the first page, the first page, right? The first page, they see the ad, they click, they come to the first page. That is important. It's just not the most important. The most important is this one. And so Facebook wants to know that, they need to know that. So what we should do here is let's spend time, let's spend this time integrating the domain with Facebook. So Facebook can start reading it. Let's also install the pixel on the page. So Facebook can start reading it. And then number three, let's set up a custom conversion around this page. A custom conversion is simply letting Facebook know the most important page, right? What is the most important at these two? It's this one. So that's your custom conversion. Facebook needs to know that. And then the last thing we'll set up is our custom audience. So when people actually land on the first page, they're put into an audience, AKA a bucket. I like to refer to it as an actual bucket. Just imagine filling up your bucket, right? That's a custom audience. So we can set up a custom audience around those who land on the landing page, on the first page. They land there, but they never opt in. That's an audience. And then the second audience would be those who actually opt in and land here. That's another audience. And so being able to segment these audiences when you're running ads makes all the difference. So let's do that today. Let's do that today. So let's go to where your domain settings are at. Let's go to business settings. And what we're gonna do is we're gonna integrate the domain. We're gonna just simply get some code from Facebook and add it over to the DNS records in Wix. Okay, so to the very, very left, we're gonna go to brand safety. And let's click on domains. And this is a new feature as well. This was never really, and remove that top one. You can, yeah, go ahead and remove that top one because that top one's not gonna be a part of this business manager anyways. Just hit remove. This one? Oh, this right here? Yeah. It says, oh yeah. Yeah, I don't know. Remove that one. Okay, so what we're gonna do is we're gonna click the blue button and we're gonna add. And so yeah, this is a new setting where Facebook now wants to know the domain that your ads are gonna be coming from. So go ahead and add it there. Just the domain, no www. So just, Julie, perfect. And then we're gonna click add. And then once we click add at dot com, once we click add, we'll have a dropdown that we'll click on. So that top, add a meta tag, click on that dropdown and you're gonna go down to the DNS. So click on that. And it's that bottom one, DNS text. Okay, cool. So what we're gonna do is we're gonna follow number two, instruction number two, click copy. So we're gonna grab that link there and then we're gonna take it over to Wix. Just click it. Just click it. It's copied. Yeah, just click it. Okay, cool. So it's copied. Copy, perfect. Okay. Go to Wix and now let's go figure out these settings over here. So we got a host, we have C name, scroll down. We're looking for TXT. And if there's not one, there it is there, perfect. So add record. And then all you're gonna do is you're gonna simply, the host name is the at sign. So just shift and then two, just at sign. And then the value, go ahead and paste in that link. Perfect. And then save. Okay, cool. So normally it takes a few minutes. So what does that have to be? Click hover over that. Okay, so zero, nine. Okay, so let me find out what this setting is. So now that we got that, so leave in a blank, perfect. Now let's go back to Facebook. And now normally, sometimes this takes a few minutes, but normally it happens quick, but sometimes I've seen this take up to a few minutes. So go ahead and click on the green one, verify domain. So once you add that, once you update that, perfect. Once you update that on the Wix side, or if anybody's using GoDaddy, I use GoDaddy for all my domains. So again, in the future, no more purchasing domains here. What I would do is I'd go purchase it on a third party, which is GoDaddy. Okay. Like keep it separate. Yeah, for sure. Cause anytime, if you have to cancel Wix in the future, sometimes they're paying on releasing the ownership of that domain. Got it.  So yeah, purchase it elsewhere, but yeah, that's how you do it. So just update it on that side and then over here, click the green one, boom, verified. So now you're good. So now when it comes to the domain, when it comes to each asset within here to the left, your domains, your custom conversion, your pixel, all that is to the left, your Instagram account, your ad account, which is where your credit card is at, your pages, everything is an asset and everything is connected back to Julie cause Julie is the owner of that asset. So in this case, when we're setting up each one, everything has to be, again, connected back to you. So in this case, it was the domain. Click on add assets, see if we have to add, we either have to add your name or we have to add the ad account. Let's see what it says here. Classic pages, click on classic pages to the left and then Julie. So all we're doing is we're telling Facebook, this domain is connected to this page, to this business page, which is owned by Julie, right, technically. So hit done and that's that perfect. So we're good with that. That was integrating the domain. Now let's take the, let's go to, let's go to your pages. So you have your opt-in page and then you have the thank you page. Let's go to the thank you page real quick. So, I'll go back to Wix. Yeah. Or one of the tabs that are open, I think you have that tab open already. It's the thank you page. We need the thank you page, which was the, yeah, perfect, general dash eight. This is the custom conversion. This is the most important page. So copy that link at the very, very top, double click that link. And what you want, go to double click it. So you open the full link up, open up the whole thing, just double click it. There you go. So it has the HTTPS. So in this case, no, no, no, you don't want the whole, you don't need the whole thing. So in this case, when it comes to a custom conversion and a Facebook custom audience, all you need is the domain. So julielarwoodrealtor.com forward slash general dash eight. You just need the domain, not the WWW or the HTTPS. Is that much? Just that much, perfect. This is for a Facebook custom conversion and a Facebook custom audience. Okay. So in this case, what we're gonna do with the thank you page and it's only the thank you page that you set up the custom conversion. Again, letting Facebook know this is the most important page. So now we have that copied.  So you should have your bookmarks and you should have, or you could do it from here, that's fine. So those three lines, yeah, you would have had it bookmarked as well. At least I would think, yeah, custom conversions. Yeah, click on that one, perfect. And I think we bookmarked it before we actually set it up. Yeah, perfect. So remove the blue star at the very top right. Click on that blue star. Let's remove this bookmark. We're gonna re-bookmark it. So just hit remove, just hit remove, perfect. Click the blue button, create custom conversion. And now we're gonna set it up and actually bookmark it from over here. So the name, again, there's two things. And a lot of times people get confused. There's two things. Custom conversion, custom audience. And in both, both audiences are both, both of them are the same person, really. There is just, it's the same person who landed on the thank you page. And so what we're setting up is the custom conversion and we're gonna name it exactly what we name the custom audience. So this is just simply seller guide leads. Seller guide leads, that's the name of the custom conversion. That's the name of our thank you page. Anyone that lands there, we're telling Facebook, this is the name of really of the page, seller guide leads. Anyone who lands here is a lead. So below, now you're gonna click on in blue, select your own category. So click on that. And what we're doing is we're hitting the recommended, hit the dropdown. What we're doing now is we're looking for lead. So we're letting Facebook know anyone who lands on this page is a lead, right? So that's what we're doing here. And then in the rules, that, yeah, right there, click and paste in that link. So this is your thank you page. So again, same thing for a listing open house, seller guide, whatever. So same thing for buyer guide, everything, hit the blue one and you're done. That's a custom conversion. Again, letting Facebook know about the second page. Now this is the newest update with this whole Apple iOS mess that is kind of going on with Facebook and Apple. This is the newest setting. So now what we have to do is simply, is simply just connect a few more dots. That's all that it could connect a few more pieces to the puzzle. That's all it is. Number one, go to settings. So in business manager, go to your settings to assign the custom conversion. So right over here, we're going to simply click on add assets. So this is our custom conversion that we're setting up. So right here in the top middle, add assets, and then go ahead and click on your name. So again, everything is connected back to Julie. Done. And then now let's go back to that. Now if you refresh the page, go ahead and refresh this page. Anytime you do like any type of update, always, yeah, always, always refresh the page. Okay, cool. So we're good here. So close out this one. So we're not, we don't have too many of these tabs open. So close out this one. Now we're going to go to number two. So in the events manager, we're going to simply connect the domain with this custom conversion as well. So this is a super new update. Okay, so over here, let's give it a second. So now over here, there's the domain. So we already verified the domain. Now we're all going to just click on it right now. It's configured events, zero of eight. We're going to turn it into one of eight. So click on that and then manage events. Edit the blue one. And then over here, we're going to click on the green one, add event. And so all we're going to do now is simply tell Facebook about the custom conversion and the event name. So hit that dropdown, choose a pixel, and we're going to go down to our custom conversion. And then to the right, hit that. And there goes our custom conversion. Seller guide leads. That's what we named it with Seller guide leads. That is the link to the thank you page. So over here, click on yes, and then just apply. So now as we go and set up the ad, as we go and run ads in the future, there's two links really that Facebook is going to need to know in the setup process of the ad. The one link is when they click on the button, where are they going? That's the opt-in page. So Facebook, we add that in the ad as we're setting up the ad. And then we also add this conversion. We also let Facebook know, hey, this is also the thank you page that we're trying to get people to. And so all that is done within the setup of the ad. So that was the custom conversion. Now what we want to do, and again, I may have said this a million and one times, but here's a million and two. The custom conversion is only around the thank you page, okay, only around the thank you page. So now let's go set up the custom audience. So what we want to do now in the custom audience side of everything, click the three lines in the top left, or just go, actually no, real, real, real quick, real quick, before you leave. I'm sorry, bookmark this page. So this is custom conversions, we're rebookmarking it. So this is custom conversions. And so it'll go in that folder. Okay, cool, we're done. Okay, good. So now we're gonna set up our custom audiences, three lines, or just go to your bookmarks, custom audiences, so right here, audiences. Yeah, so bookmarks, custom audiences, perfect. So this right here, Julie, is what it's all about.  If you're putting out video, which I highly, highly, highly, highly recommend you start getting used to in practicing falling in love with, is video. Yeah, our regional manager talked us Tuesday and said, do it, just do it. I'm telling you, it's so simple. You'll break out, you'll eventually have your breakout, but it's so simple and it's so effective and it's your cell phone, it's selfie style. It's talking into the screen and giving us something, you know, a tip on what it takes to sell or a tip on what it takes to buy. So as you start to do this stuff, audiences are what it's all about, especially if it's an ad. If it's a video that you're just putting out for the heck of it and you're just posting it and it's real estate related, but you just put it on your personal page and then maybe you put it on your business page as well. If it's kind of just random, organic, you don't necessarily need a custom audience around that because chances are I'm gonna end up watching it and if you have me in an audience and I'm not even in the market, that's kind of a waste, right? So it's typically audiences around video ads or around videos that you're actually running ads on. That's really the best custom audiences when it comes to your videos or videos that you're actually running ads on because now I know that the few thousand who actually watched it are all in my backyard. Okay. So now I can put them in an audience and show them another ad, right? That's, so what I'm saying is as you put out more ads, if it's a video, it's a custom audience. If it's simply an image, then that audience that we already have right there, people who engaged with any post or ad, they'll be in that audience. And so that'll be an audience that you'll always want to retarget in the future. You'll always want to retarget. The audience that we're gonna also set up here, we're gonna set up two audiences right now. We're gonna set up one audience around those who, that's one audience right there. So what we're gonna do is we're gonna set up our second audience, which is gonna be those who land on your, they click the ad, they land on your landing page, but they never opted in. And Facebook's gonna know that because we're gonna be very specific about that and let them know how we know that. These are people who only landed on the opt-in page. And then the third audience that we'll have is those who land on the thank you page. That audience will be an audience that you'll always want to exclude. So as I teach how to run these ads, we're either including them in the targeting, people who engage with an ad, people who watched 15 seconds of my last video, people who landed on the webpage, who landed on the landing page, but didn't opt-in. Those are audiences that I'm gonna always, always put in my targeting, right? Because they've seen an ad before, they've seen a video before, keep showing your face to them, right? And they still, they haven't yet got off the pot, but they keep seeing you, right? And they live down the street. That's powerful. So let's hit the blue button. Let's set up the, let's set up two of them. Let's set up the landing page, custom audience. And so as we come over here, here are all the different ways you can create an audience, but really in your case, it's website and video. That's kind of the main ones. We did Facebook page, which is the first audience, people who engage with the post or ad, but really from this point on, that audience will never have to be created again. You have the first one there, that's good to go. What you'll always, always wanna create are video audiences cause it's always gonna be a new video, or if you create different websites, now I have an open house landing page. Now I have a listing landing page, right? You create different funnels, then you'll wanna create audiences around website for all of those as well. So this is how you segment who's seeing your stuff. It's so powerful, so cool. So this case, go with website and then next. Next, and then over here, we're gonna simply, and the red dot is there on your source, that's your Facebook pixel. It's a red dot right now because we haven't added the pixel to the page yet. So once we add the pixel to the page, that red dot will turn green. So what we're doing now is we're setting up a custom audience, we're gonna hit events, and then go down to specific. We don't want all website visitors, we want to tell Facebook, this is a specific link, this is a specific audience. People who land here are in this. So paste that link in right there below, add, or at least one of these values, yeah, right there. Click and paste it in. And then the retention, what you do. Do I have a copy already?  That one, general eight? Yeah, that's general eight. Okay. So this is the thank you page, right? Right. Yeah, okay, cool. So this is the thank you page. Okay, so that's the pixel that's gonna be running from the source. That's the event, a specific page. The retention, what I do in the retention, max it out, 180 is the max. Now, the reason why I'm going high on 180 is because this is a lead, this is a lead. So this is a custom audience that I'm gonna always exclude from future ads. I don't need to keep showing my ads to people who already opted in. All right, since I've already done it. They've already opted in. They've already downloaded your seller guide. Why do they keep senior ads on it? You know what I mean? So that's what this is, is a custom audience. Now again, we did the same thing with the link on a custom conversion, letting Facebook know the most important page. But in this case, it's the audience that we're setting up now. So that's the right one, right? General eight, that's the thank you page, right? Okay, cool. So now below, go ahead and give it a name. Name it exactly what we named the custom conversion around that link, which was what, seller guide lead? Or what was it? Seller guide leads? Or something like that? Yeah. Perfect, and then create audience. Okay, seller guide leads, create audience. Okay, and then done. Perfect, so that's an audience around leads. Again, you'll always wanna exclude that audience. Now let's do another one, create audience. And now let's do one around custom audience. Let's do website again, and let's do one around the landing page. So all website visitors, and then go down to specific on the events. So all website visitors. And on this one, 180, yeah, you're going a little too fast. On the events, hit that events at all website visitors. Go to specific. You always do specific. So 180 on this, so in this case, again, there's a different audience now. Now these are people who have clicked on an ad, got to the page, but did not opt in, right? They didn't download your PDF. So in that case, I don't go 180. I go way, way less. I go like, maybe 30 days the most. Because if you, again, and a lot of times when it comes to setting this stuff up, just put your consumer hat on. Like if you put your consumer hat on, put your consumer hat on. A website that you clicked on, let's say 47 days ago, and you only clicked on that website once. Would you remember what website that was? No. Probably not. Unless it was something crazy, freaky, you'd be like, dang, I'll never forget about that. Unless it was something crazy like that, right? Like we all come across weird stuff and it sticks with us for a long time. But typically, if you're just surfing, browsing, like clicking on different things, social media, whatever, you don't really remember. So that's this case. So in this case, these are people who've clicked on an ad, landed on a landing page. I want you to still somewhat remember that. And so 30 days is kind of the most, right? So if someone clicked on my ad maybe 20 days ago, there's a good chance that they'll remember who, the brand or maybe the name or maybe the face. So I'll keep them in this audience because this is gonna be an audience that we always retarget. And here's how we do it. So did you copy the link for the landing page? Yeah, let's get the link for the landing page. And then what we want is we want both links, but let's just do the landing page first. So this one right here, just the domain. So make sure that you're not copied the whole thing. Awesome. So there we got that. Always go with the dropdown to always hit that blue. Yeah, perfect. Okay, so look, here's what we're doing. Very, very specific. This is very specific. We're telling Facebook that these are people who landed on the landing page, but never opted in. Okay? So in that case, we're gonna go exclude people. Exclude people. And now we're gonna do the same thing. So scroll down events, go down to specific, all website visitors, specific, and then leave the 30, that's fine. And then, actually, no, this is the thank you page. So I want the 30 to be 180, max it out. Max it on the retention on this one, on the second one. Yeah, 180. And then so now grab the link for the thank you page. I mean, excuse me, yeah, actually, yeah, for the thank you page and paste it in right there. So you have the link for the opt-in page. That's what you already pasted at the top. Right now, grab the link for the thank you page and paste it in. So did we just do that a minute ago? Because we got the general eight. We did that in the first audience. Yeah, the first one. The first audience is simply leads. That's that audience. Oh, okay. This is an audience who have yet to land on this page and Facebook needs to know that. So this audience, they've landed on the first page, pasted in there at the bottom. So look, they landed, click the blue one. Okay, scroll up. So this is general dash eight, scroll at the very top. So here, this is simply Julie Larwood realtor. So they've landed on that page, but they've never landed on the second one. That's what makes this audience so specific, right? So this will be an audience that you can always retarget because they haven't downloaded yet for whatever reason. Audience name, go to name it, seller guide page, seller guide page visitors. Seller guide page visitors. Seller guide page visitors. And then maybe in parentheses, only. They've only landed on this page. They have yet to opt in. So that's a warm audience that you'll eventually start to build. Hit create audience. You'll be running ads and you just, as you run ads, and as you put out organic post as well, use this link, hit done. Use this link. You now have bait on your phishing pole. So as you start to put out organic post from this point on, what I would do also is, now that I really think about it, the link that the dot com that's on all of your platforms, Facebook, there's your dot com there. Your Facebook business page, there's your dot com. Your LinkedIn, right? I would probably put this link because this is your funnel. This is your landing page. This is your lead capture page, right? A lot of times real estate agents will just simply put their dot com and it takes them straight to their fancy Century 21 website when it's not even a lead capture page, right? So now as I put out content, now at least I have something to give, not just the content for sure. I'm giving content for sure. I'm giving good advice, content for sure, but the bait is the free download, right? So as you start to put out even organic post in the future, include your link. Hey guys, and by the way, I also put together a free seller guide that I love to give you. Click here if you care to download a free copy, right? And you can start, start just randomly capturing organic leads just by sharing this link and then keeping people segmented in these audiences. So add this to the profile where I've got all those other links. You might as well, yeah, yeah, add this dot com and even start sharing it. Why not? Now, as you start to share it, and that's what we kind of went over on our first call where it's like as you take your friend count up on any of these platforms, they're gonna be people like me across the country. So if I saw Julie posted on her personal page, whether it was her Instagram, whether it was her Twitter, her LinkedIn, whatever, Facebook, I'm connected with her. Okay, I may download it for the hell of it, but I'm not even in that market. So that's why, again, it's really straight ads because even just randomly putting it out organically, I mean, it might not, you can do it is what I'm saying, do it is what I'm saying, but that's not the only thing we need to be doing, right? That's all about targeting the local audience, but at least you have now something to give, right? A free download, I'd use it all the time for sure. Okay, so that was custom audiences. So what did we do? We integrated the domain, we set up our custom conversion, we set up our custom audiences. Now what we need to do real quick is go to Wix and now we have to install your Facebook pixel and then that'll be good for today's call. Let's go to Wix. One question I had so, so if I do the new app, do I do those three steps again or the first one's already done and I just have to do custom audiences? Very good, very, very good question. All you'd wanna do from this point on is set up either a custom conversion if it's around a different page. So right now you have your seller guide funnel, right? If you go and create a listing funnel, an open house funnel, a buyer guide funnel, all you're looking for is the thank you page and go set up your custom conversion. But if you're not doing any of those other funnels right now and you're only focused on the seller guide, which is what I highly recommend, only focus on this one for now, no buyer guide, nothing, just seller guide, focus on this one.  You don't have to do a custom audience again. The landing page right there, visitors only. The thank you page, seller guide leads, those are done. What you'll always wanna create from this point on is custom audiences around new videos, around videos that you're running ads on. So if you start putting out video ads, that's when you'll come over here and set up a custom audience, but no more custom conversions. Okay. That's done, that was only done once on the thank you page. And that's also done once here on the seller guide leads audience, one time, don't have to do that again. But what I wanna stack up here, what I'm really encouraging you to do is fill up this list and 99% of every single audience from this point on will be videos. Okay. 99% and let's say you got three audiences right now. Let's say event, let's say we speak again in seven months. In seven months, 99%, there's three audiences. There should be 27 audiences and 99% of them are videos. So to answer your question, it's really just setting up video audiences each time from this point on. Okay. And then you just simply run the ad. And we'll, again, I'm gonna always be here. So we're gonna jump on many more times to run through this. But what we did today was super, super important. Now let's add the pixel real quick. Let's go to Wix, cause I'm not too familiar with where that pixel goes, but I know it's super simple. It has to be super simple. So in here, we're looking for our pixel settings. Okay. So that was a whole different way to set up a Facebook pixel inside of Wix, but that was super easy. That was super easy. So I'm glad it was that simple. So now let's go back to the ads manager. Actually, you know what? Open up your landing page. Did I have you download the Facebook pixel helper? No. Okay. Right here in the Google search, type in Facebook pixel helper. This is how we test our pixel. Okay. This is one way to test your pixel. But let me have you just install this. So click on that top one. And this is free. It's a plugin. It just sits there in the top right-hand corner of your screen, add to Chrome. And this just simply allows you to, yeah, add extension to simply know if a website is pixeling you. So you could be on any random website and that thing will either light up or not light up. And that's how you know, there's a pixel on the page. Now I don't use it for anybody. Like I don't care who's tracking me. We're all being tracked. I don't even care about that no more. These websites, like I don't even care about that. I use it for my pages. Like just to test my pages, make sure my pixels install correctly on my pages. That's really why I have this. So if you click on that little hammer in the top right, go in and pin it. So it sits there in the top right. So in the very, very top right of your screen, you see your profile picture. Yeah, click on that little hammer-looking thing to the left of this. Yeah, click on that.  Okay, cool. So now it stays there at the top right. So here you're on this Chrome Web Store page. There's no pixel on this page. So it's great. X out of this. So close out this tab. Yeah, we don't need this. So now if you go to any other website, like all these other pages that you have open, open up any one that's not to private or whatever, refresh this page, refresh this page. So realtor.com, they have to for sure be pixeling you. And that's what that is. So now it lights up. So click on it. And it'll tell you right here, it'll tell you that yeah, there's a pixel on realtor.com and then this is the different activities that they're tracking, page view. I don't know why they have add to wish list, but page view for sure. So anyone that lands here is in their custom audience. So they could turn around and send an ad straight to Weatherford, Texas and target everyone in that area who has landed on this page. Okay. Super crazy, right? Super powerful. Super powerful. So if you can imagine farming a couple zip codes and your ad just keeps showing up in everybody's newsfeed down the street. That's how crazy this is. Super powerful. And it comes with, and it's because of this pixel, this pixel that we just installed on your pages. And it comes with integrating the domain with Facebook and really that's it. And that's what we did. So in the future, in the future, go to the ads manager real quick. You have, so the pixel will test the pixel on your pages, but let's go here real quick. So click on the three lines and go to events manager. So in the future, if you ever need, that was a super simple way to add a pixel. I've never seen it that way. Like what the way Wix has it. That was super simple. But in the future, most are not like that at all. And so you have to either get your pixel ID or you have to get a piece of code. And this is where it's at. So hit that red triangle, the bottom one to the left, Julie Lawwood, right where that red triangle is at. Click on that account. Okay, so over here, and then click on your drop down in the top right and go to, let's check something real quick. Click on your name. No, I'm sorry, your name in the top right. And then go to the other account, the top one. And then click on that bottom, the red, that second account. Yeah. Okay, cool. So now click on your three lines. Let's see if we need to add any assets real quick.  So you always wanna double check this. So the pixel, click on data sources to the left and then go down to pixel. Pixels, yeah, click on that. And then over here, so you have your name over there, click on add assets to the right. Add assets. Yeah, click on that, see if there's anything. Okay, click on that and then just click on add. And then done. Okay, so it looks like the Wix marketing system and then refresh this page. Okay, now go to your actual pages and let's refresh those pages. So go to your opt-in page, go to your thank you page, which they should be open. One of these should be those pages. Okay, so refresh this page and now there's a pixel on it. So now Facebook just read that pixel for the first time. Go ahead and click on it. Click on your pixel helper. So you always wanna click on it just to make sure, okay, there goes your pixel, right? That's what you named it. Jolie Larwood dash realtor, right? That's your pixel page view. Now everyone that's landing here is now being put in the custom audience, right? So that's good. So now go to your thank you page, open up the other tab. Look at your thank you page, refresh this page. Let Facebook get a fresh view of this page for the first time. So now there's that. So now this is a custom audience. It's also a custom conversion, but most importantly it's a custom audience and this will start to build as you start to stay consistent running ads. So now the pixel is being read. Go back to the business settings real quick. Go back to the ads manager. Let's check something real quick. So now refresh this page and this may take a few minutes, but eventually perfect. Now that red dot is green. Now that pixel is now active. So if you click on the three lines and go back the three lines at the top and go back to events manager, this is where you get your pixel in the future. So events manager, now this looks different. Now the red triangle is at the top. Now this is the top account, right? So now over here, what we wanna do is click on settings right in the top, right? And let's make sure that this pixel is tracking that link, that domain.  Click on that. We're gonna give traffic permissions and then go ahead and type in your domain right there. We should have seen it in the dropdown, but just go ahead and type it in anyways and then click on next and then confirm perfect and then close. Okay, cool. That domain is now also integrated with this pixel. So in the future, click on overview at the top, overview. In the future, if you ever need to add this pixel to any other pages, again, it's overview right at the top or overview. So at the top, the top. Oh, okay. Okay, so in the future, again, normally it's not as easy as Wix was. In the future, you'll always either need the code or you'll need the pixel ID. Some websites just require a pixel ID and that's your pixel ID to the very, very right. So it's four, two. And let's also double check that. The last four digits is 12, 18. Go to one of your pages again, your landing page or your opt-in page. So that's 12, 18. Check this pixel. 12, 18. 12, 18, perfect. That's why I have the pixel helper. Right, just my stuff. I don't care about anybody else's pixeling me. Everybody's pixeling you. There's a pixel on everybody's website nowadays. Everyone is pixeling us.  So again, what I'm trying to say is that in the future, you'll either need a pixel ID and that's it right there to the right or you'll need the code. So if you ever need the code, scroll down. All activity right here, add events. Hit that dropdown, add events. Click on that button, add events. And then from a new website, install code manually, copy code. That's it. Okay. Here's what I would do. Here's what I would do. Here's what I would do for you. Email it to yourself. You just copied it, paste it, email it to yourself and just name it Facebook pixel. Just have a copy of it. Because in the future, if you have anyone that builds you websites or anything like that, all you gotta do is just forward it to them and they'll be able to take this code and paste it on anything. So typically websites either require that code or they just require the pixel ID. Okay. And that's that, perfect. So we integrated the domain, we set up our custom conversion, we set up our custom audiences and we installed our Facebook pixel. So now if anything, just clean up those two pages, the opt-in page, take away the social media icons, add the footer, add those different links, the privacy policy. Yeah. And that's it. And we're good to go after that. We're good to go after that. So once we have all that edited, then we can go and set up our first ad and start getting that facing. Okay. Locally, locally. Right. Okay, I'll probably set up something for Monday. Sounds good, Julie. I'll be a message away from you anything. Okay, thank you. Talk to you soon. Bye-bye.