I am a behavioral scientist who studies diversity. I study diversity in many different forms, but today I'll speak mostly about racial and gender diversity with research that has been conducted almost exclusively in the United States. As a roadmap or a guide to what we'll do today, first I'll speak briefly about the workforce worldwide, the science and engineering workforce worldwide and what it looks like demographically. Then I'll speak briefly about how diversity can help drive innovation. And then I will go over three different sets of challenges regarding diversity and what the literature in my field, experimental psychology as well as organizational sociology has to teach us about diversity and inclusion. So this is the global workforce, the global PhDs. This is data collected by the National Science Foundation for 56 countries. What you'll see here is that PhDs awarded to men are in orange and to women are in green. The solid circles are PhDs in science and engineering, the ones that aren't solid are not. The y-axis here is the percentage of total PhDs granted. And along the x-axis are the countries. Now you'll see these little numbers by each of the countries. And those numbers are from the World Economic Forum's Gender Gap Index, Global Gender Gap Index. And they start at one here with Iceland and go up. They represent or measure the difference between men and women in terms of resources and opportunities in their country in health, education, economics, and politics. There are few patterns to notice here. The first one is that overwhelmingly these countries have an imbalance in terms of gender imbalance, in terms of who is getting PhDs in science and engineering. What you'll also notice here is that there are a few countries, including the United States, for which the pattern is flip-flopped, depending on whether you're looking at science and engineering PhDs or other PhDs. And that's also the case for a couple of other countries, including, I believe, Sweden and Australia. Going down the graph here, you'll also see that Ukraine, which is this dot here with the overlapping circles, actually has gender parity in PhDs in science and engineering. And if you look to the right of Ukraine, those are the very few countries which do not have the pattern where men earn more PhDs in science and engineering than women. We have very little information on race and ethnicity worldwide in terms of workforce participation in science. But we do have some data from the United States and South Africa. Looking here, what you can see at the top is that white men are overrepresented in the science and engineering workforce. White women are underrepresented in the science and engineering workforce. Other groups that are underrepresented are Hispanic males and females and black females and black males, whereas Asian females and males are overrepresented. This is particularly the case for Asian males in the U.S. You'll also find similar patterns if you look at South Africa, though, of course, the racial and ethnic categories are slightly different. Gender is, of course, the most commonly collected piece of information. So we do have data across regions in terms of who is composing the research workforce. And what you can see here is pretty dire underrepresentation of women in most of the regions. And we also have some information on socioeconomic background. These data come from the United Kingdom. And basically what you can see here is socioeconomic status as indicated by age at which parent left continuous full-time education. In other words, education of parents plotted against whether the child eventually entered the science workforce and we see a linear relationship. Finally, we have disability. Now, disability is hard to define and hard to quantify. What you can at least see from these disability numbers is that people with disabilities are underrepresented in the research, in the science workforce, at least as compared with their proportion of the population. Now, these numbers suggest some pretty drastic underutilization of certain groups. And this is perhaps a missed opportunity for many reasons, one of which is that diversity used in the right way can help power innovation. Wentz gave a talk on unconscious biases for a group of synthetic biologists. And they were very excited to tell me about a discovery that they had just made. It wasn't a synthetic biology discovery they wanted to tell me about. It was a discovery that of the prize winners in a competition, an annual global competition they had in synthetic biology, of the prize-winning teams, the ones that had women tended to win more often. And I went back and looked at the literature within experimental psychology, so literature on studies conducted more artificially in a laboratory and found that once again the studies seemed to point to groups that were composed, had some gender diversity, at least some gender diversity, tended to perform better on a variety of measures. I then looked at literature in business management. And as it turns out, one study conducted by some researchers who looked at companies in the Standard and Poor's Composite 1500 list. They looked at the representation of women in those companies in top management and found that gender diversity in top management predicted, or as they put it, resulted in an increase of $42 million in firm value. Another similar study was conducted looking at 177 U.S. national banks, looking at not gender diversity, but racial diversity, and found once again that those companies that had racial diversity, those banks that had racial diversity, tended to have enhanced firm performance. Now, for both of those studies, the companies that tended to show the biggest effects were those that were focused on innovation. As it turns out, in science, we see a hint of a similar pattern. Two economists, Freeman and Huang, recently looked at 1.5 million scientific papers that were published. They looked at the ethnic diversity of the authors and found that those papers with greater ethnic diversity were cited more and had greater impact than those with homogenous authorship, controlling for a variety of relevant factors. Of course, all these studies are basically correlational, but when we look in laboratory studies that manipulate the composition of groups, what we see is once again the benefit of diversity, such that groups that are not homogenous, groups where there are people who are different from you, cause you to actually think more and think more complexly to work harder to repair better, and those end up resulting in better team performance. It turns out that people actually also individually benefit from diversity For example, research suggests that people who live in different countries, so expats, end up actually becoming more creative according to standard measures of creativity. But diversity is difficult. Diversity is not always easy. As it turns out, what the literature also suggests is that in order to reap the benefits of diversity, you have to make sure that you provide it with a supportive context. However, three main assumptions can get in the way of that, and I'm going to go over those three main assumptions and some research that helps us to undisentangle what is going on and how we can better support diversity and inclusion. The first assumption, at least in the United States, is that in order to do diversity, you actually don't have to think very much or very hard about difference, that you actually don't need to pay too much attention to difference. And in fact, in the United States, the mantra is more that you should avoid difference or ignore difference in order to produce a fair and inclusive environment. The second assumption is that everybody experiences a class setting or a work setting in basically the same way. If you're a teacher, the students in your, I'm a professor, the students in my class, the assumption would go, are experiencing my classroom in basically the same way. So we're going to look at that assumption. The third assumption is that when there are difficulties, when there are problems related to diversity, that they are too systemic, they're too big, they're too entrenched, they're too systemic, we can't do anything about them, or rather that they're simply due to a few biased individuals. And so we actually don't need to worry too much about it. Let's tackle the first assumption, what we do about difference. A few years ago, Sodexo, the company Sodexo, was sued for racial discrimination. And in an interesting interview that was published with the CEO, Michelle Sandel, he said something that really struck me. He said that when they learned that people were unhappy and felt that they had been discriminated against, that they were surprised. So how could this have happened? How could they have been so surprised about people feeling unhappy and discriminated against such that they were taken by surprise by this lawsuit? Well, some of the research in my field can help shed some light on what possibly could be going on in such an organization. I'll give you an example of one study that I conducted. My colleagues and I went to a health care organization, a very large health care organization in the United States. And the health care organization consisted of scientists and doctors and nurses and other health professionals. We asked them about their attitudes towards diversity. In particular, we asked them about whether it was important to ignore or avoid differences. And we asked them whether instead it was important to acknowledge and embrace differences. We also asked them about their psychological engagement with their work, their standard measures for this in organizational behavior. And we did this in part because we know from other research that psychological engagement predicts productivity, turnover, and absenteeism. So those were the three measures. We had something we call in the United States color blindness or ignoring racial differences and ethnic differences. Embracing diversity or multiculturalism might be another word for that. And engagement. And we looked at 17 departments. The company was organized in 17 departments. And we correlated the majority groups' attitudes towards diversity, in this case white Americans, with the minority groups, in this case employees of color, particularly African American, Latino, and even in this case Asian employees, psychological engagement with their work and commitment to the organization. And this is what we found. In departments, if you look at the top graph, in departments where whites believed more in color blindness, minority engagement was lower than in departments where there was less color blindness. In contrast, in departments where white employees believed more in embracing and acknowledging difference, minorities were more psychologically engaged. This held controlling for a few things, including the proportion of minorities in the department. Moreover, we found that minorities in the colorblind departments tended to sense more bias. And departments in the more multicultural departments tended to sense less bias. And statistically this helped to account for this relationship. So studies have also examined what might be going on in people's minds. And as it turns out, unconscious biases and conscious biases may very well be playing a role in the results that I just showed you from the healthcare organization. In one study conducted at Dartmouth University, white college students were brought into the laboratory and not to a field with grass. And they were asked to read an article. And the article was either one that promoted ignoring differences in order to achieve racial harmony, or one that promoted embracing differences in order to achieve racial harmony. Both of them were positive. Both of them were about achieving racial harmony. Then they were asked to take what's called an implicit association test or an IAT. For those of you who aren't familiar with this test, have never seen it before, I'm going to show you an example. The job of the participant in this test is to categorize whatever shows up in the middle of the screen. In this case it would be ethnically suggestive names. So Jamal would be a name in the United States that would be a black name. Or Josh would be a white name. And they also saw positively or negatively valenced words, which I'll show you in a minute, like paradise or poison. And as quickly as they can, they have to push buttons on a computer keyboard and classify these words into one quarter or another. In those corners are different pairs of words and they switch the pairs. Here the pairs are black and good and white and bad. But they also switch them such that you have black with bad and good with white. And the computer measures how quickly you go at categorizing the words such that being faster at pairing black with bad and good with white indicates a pro-white bias or an anti-black bias. And so this is probably the most commonly used measure of implicit bias. The participants did this. They also completed a paper and pencil. Here's an example actually of how they would go about. So each time they see a word they classify it. They also completed a standard paper and pencil self-report explicit attitude task where they basically indicated how warmly they felt towards other groups. And what the researchers found was that people who read the more colorblind, the ignore differences article actually showed more implicit bias on that computer task and more explicit bias than those who read the article about acknowledging differences. So this is all one individual sitting at a computer. What about when you put individuals in interaction with one another? Studies have also looked at this. And they have found that the prescription to ignore difference and not just in race but they've also done these studies with facial disfiguration and sexual orientation and many other types of difference that the prescription to ignore difference can actually make interactions go awry. It can lead to bias in both nonverbal behavior. So in the United States that might be how much you're blinking, how far away you're sitting, how open versus closed your posture is. So bias in nonverbal behavior and bias in verbal behavior. And what some researchers did at Princeton University is that they examined the effects of priming or exposing white students to a colorblind versus a multicultural message on their ethnic minority conversation partners. And in this case the ethnic minority conversation partners were Asian and black students. So Asian and black students at Princeton University conversing with white students at Princeton University. And what they found was that the prescription to avoid race when whites were exposed to the prescription to avoid race, the minority conversation partners were more cognitively depleted. They were more cognitively exhausted by the conversation which has clear implications for their performance on other tasks. So in summary the prescription to avoid difference can actually lead to unintended negative consequences. This is consistent with research scores of studies that show that perceptions of diversity climate, students' perceptions of diversity climate on a college campus and students' perceptions of whether they have been discriminated against or experienced prejudice actually predict whether underrepresented students avoid and stay in STEM fields. You see the same thing when you look at the management literature. Perceptions of diversity climate predicts turnover intentions and actually it predicts turnover intentions among racial and ethnic minority and majority respondents. So one other study that can shed a little bit of light on how one might use difference to make a difference. This is a study conducted at Northwestern University. It's not on race, it's on socioeconomic status and the participants were either what we call first generation college students. In other words, they're the first in their families to go to college or they're continuing generation students. They're not the first in their families to go to college. These students attended panels during the first semester. In fact, I think it was in the first weeks of the semester at Northwestern and the panels were trained. In the experimental condition, the panelists were older students or more senior students, more experienced students who had been there for a while who were trained to give advice to the students but do so in a way that made socioeconomic difference explicit, that explicitly referred to class background and to first generation status. The control group were exposed to a panel that gave basically the same advice but did not expose difference, did not talk about difference, did not make difference, blatant. And then the researchers examined the grades of the students at the end of the semester. In fact, they had planned to let the study go on and examine their grades later on, but they didn't. They ended up publishing their findings right after their first wave of data collection because this is what they found. For the treatment group, this one hour intervention of going to this panel cut the gap between first generation and continuing generation students by 63% within just a couple of months. That one hour intervention, that one panel. There was no other difference between the students. They were randomly assigned to these groups. So those are some hints at how ignoring difference versus acknowledging difference can make a difference in work and educational settings. Now let's look at the second assumption. The idea that people experience a setting in the same way. As it turns out, we like to think that science is science and that if only people have the motivation and the preparation that they will join the club. This is a nice thought, but in reality when you look at the research, it's a bit more complicated than that. It turns out that a sense of belonging may be a key driver for both participation and performance. I'll give you an example from a study that was conducted by Stanford University researchers. They examined students that were incoming students. They were first year, first semester students. White students and African American students. Nothing else distinguished them. They were all students at the same university. And they gave them a one hour intervention or they were randomly assigned to the control group instead. In the one hour intervention, the treatment group, what they did was they read testimonials from older students about how when they had started school, they also felt like they had social difficulties. They felt concerns about not belonging, but that over time their confidence in their belonging grew and that things were going well now. And these were older students. So they read and interacted with these testimonials. In the control group, they did something completely unrelated. The researchers then tracked the grades and the self-reported health and well-being of these students over the next three years. And this is what they found. For white students, so those are the ones in blue, it didn't really matter, actually didn't matter at all whether they had been in the treatment group or the control group. For African American students, it mattered a lot. In fact, the treatment was so powerful that it cut the original racial gap between the students at the beginning of the study by 52%. But the effects were not just seen in GPA. They were also seen in self-reported health and well-being. So as you can see here, looking at the first graph and the third graph, which are self-assessed general health and subjective happiness, if you look at the dark orange bars, what you can see is that these African American students who were in the treatment condition reached parity with the white students in terms of their self-reported health and happiness. If you look at the middle graph, what you'll see for the African American students who were in the treatment condition, they actually reported fewer doctor's visits than everybody else in the study. So that's one study examining social belonging. And social belonging may also help to explain, not explain completely, but help to explain why historically black colleges and universities in the United States are actually stronger, in fact, the strongest producers of black STEM graduates. And it points to the importance of predominantly majority environments or predominantly white environments or predominantly non-black environments to work harder in terms of creating a supportive context. Another issue that Warren's mentioned here is the stereotype of science and how it can send signals that repels certain groups of people. So this here is an image of a white male scientist. It's actually not terribly different from what children draw when they're asked to imagine what a scientist looks like, at least in the United States. And so in the literature, we see a few different stereotypes of scientists. Number one is male. Usually white, though that is changing. We also see a stereotype depending on the field of geeky. Geeky means socially awkward, singularly obsessed with whatever their science is or whatever they're doing. Doesn't do much else with their life, which leads to another stereotype, which is that science is not collaborative. That science is something that people only do alone, which, as we know, is actually not quite accurate. Though, of course, scientists can differ a lot in terms of how much they collaborate. And finally, that scientists don't always do something that's relevant to the world, that they don't always do something that is important to solve social problems, which we also know is untrue. So my colleagues and I decided to tackle one of these stereotypes. We wanted to know what the effect was of the stereotype of the geeky computer scientist on women's participation in computer science, their interest in pursuing computer science. So we decorated a room in the Computer Science Building, the Bill Gates Computer Science Building at Stanford University, to be either conforming with the stereotype, the geeky stereotype of computer science, or to conform with a more neutral stereotype. And so here you can see when the room was decorated with the geeky stereotype in mind, there's a Star Trek poster, sci-fi books, and a pyramid of Coke cans. In the more neutral room, the non-stereotype room, we had a nature poster, neutral books, and water bottles. And we simply asked people about their interest, women and men, about their interest in pursuing computer science. And what we found was that women were significantly affected by the manipulation, such that those who answered the question in the neutral room were much more interested in pursuing computer science. We've replicated this effect not just with pursuing science in college, but also with pursuing it in jobs, jobs, teams, and companies. And we've run at least 16 studies looking at this. It's a very robust effect, and social belonging seems to be the key to it. Basically, women who are in the stereotype condition are sensing signals from the environment, even absent people. There are no people in the environment. They're sensing signals from the environment that are creating a decreased lack of interest in participating in computer science. We call it ambient belonging. Sometimes they also refer to it as talking walls. We shouldn't forget that the walls and the structure of the environment can help to create behavioral change and behavioral decisions. Another study that was done, I'll go over this very briefly, looked at the stereotype of science as non-collaborative and its effects on women. And what they found was just the simple act of having women read about a day in the life of a scientist that was full of collaboration and interactions with other people, as opposed to one that wasn't, was enough to increase women's desire to participate in science. One more note about belonging. A number of groups have sprung up. A number of nonprofits have sprung up in the United States to teach coding to underrepresented youth and girls. And many of these are actually supported by industry, including Google, has taken an interest in these. And the importance of these lies in a few things. It's not just that they're providing valuable educational and career opportunities and development to these youth, but also that while they're doing that, they're bolstering belonging, they're fostering collaboration, and perhaps most importantly, they're having students do things, develop applications that are relevant to their own lives and to their communities, which could in turn create, foster, drive innovation. Okay, so now we come to our third assumption. Is our problems of diversity so systemic that we can't do anything about them? Well, if you just listened to the first two, maybe you would come away with the idea that we should just acknowledge difference and include people, and then we have the problem solved. Researchers in organizational sociology say not so fast. There's actually a third vital component. And that third vital component is what diversity structures you put in place in your organization. So here I'm going to borrow from that literature. Researchers at Harvard studied over 800 companies. They got data from over 800 companies over 30 years. So over 30 plus years of data from these companies. And what they were able to do was see the increase in proportion of employees or managers of color. So they looked at black, Latino, and Asian employees of color, managers of color, and women of all ethnic backgrounds in these companies after an initiative had been implemented. So after a company implemented a diversity initiative, do you see an increase in the numbers? That was the goal of their study. So what did they find? First one that they talk about is having someone in charge of diversity in the company, it could be a chief diversity officer, it could be a diversity staffer. Having someone in charge who's accountable for diversity tends to increase the numbers of many of these underrepresented groups. The second one that they suggest has an important influence is having a diversity task force. Having a diversity task force is actually even a little bit more effective than just having a diversity manager. And in fact, they find that both of these contribute to the success of other diversity initiatives in the company. For example, having both of these in place can help the effectiveness of structures like employee resource groups and diversity councils that are tasked with development of employees and retention of employees. The third one, and perhaps the most important for our purposes in terms of science, is having a mentor program. And this ended up having an increase of for some of these groups almost 40% within a certain number of years. The importance of mentoring in science I think cannot be understated. If you look, for example, at science education, having a mentor often unlocks so many opportunities in science for a student. It can teach a student about what opportunities are out there. It can get them letters of recommendation. It can teach them important research skills by working in that mentor's laboratory, for example. There's the provost of Berkeley is named Claude Steele. He is a renowned African American psychologist, experimental psychologist. He recently published a book called Whistling Vivaldi. It's about stereotypes, and I highly recommend it. He wrote in his book about the importance of his mentor in terms of his own career path. And what he talked about was how his mentor appreciated him as a capable colleague, as a worthy partner, and that for his mentor, his race and class identities didn't get in the way. So that's mentoring. Interestingly, whoops, wrong button, there we go. Interestingly, these initiatives, diversity training and diversity evaluations tend not to have a big effect in terms of increasing numbers of employees of color and women in management in these organizations. Part of the reason this is interesting is because of what organizations are actually doing, which is shown here. In terms of the popularity of programs, percent of firms, which is the most popular, the one with the least effectiveness according to the research, the diversity training. That doesn't mean that diversity training is wholly ineffective, but that we need to think very carefully about the structures that we put in place. So we've talked about acknowledging difference. We've talked about inclusion. We've talked about diversity structures. There are companies and universities that have put these principles to work. So IBM, for example, is a company that noticed this, that acknowledged this over a decade ago, and they put in place a chief diversity officer and a diversity task force. And that led to the creation of diversity councils and mentoring and employee network groups. And what was important about the way that they did it was that it was fully integrated into the business. In other words, they had the support and the ear of upper management, a direct line to the CEO. But they also, through the task force and the councils, had input from local constituents at every level and across the entire organization, which led, in part, to its great success. Another company that has done similar, has implemented similar initiatives is PepsiCo. And PepsiCo has actually publicly credited these diversity efforts for its success in its billions of dollars of revenue growth, in particular with the development of new products that resulted from increased diversity. In terms of universities, I don't want to come across or leave you with the impression that the principles that I've outlined today are the only things that need to be done. This is a much more complex problem. Some universities have acknowledged this and have put together programs that take into account the principles that we've discussed today, but that also take into account much, much more, including financial aid and advising and coursework and pre-college preparation. One such example is the Meyerhoff program at the University of Maryland. And these are some examples of the 14 components that they have in place to increase the number of underrepresented minorities in science. And they've been quite successful in science, this one, yep. So, in conclusion, my aim has been to provide some principles that we can think about with research, from the literature, that we can consider in helping to increase participation that can help increase diversity and inclusion. However, it is complex. It is challenging. But what this literature suggests, all three of them, the acknowledging difference and inclusion and diversity structures, is that we have tools to do better and we can do better. And I'm happy to take your questions. Thank you.


Hi and welcome back to ESMAConf 2023. This is the fourth presentation session on quantitative synthesis part one. As always, you can ask questions via Twitter by following at ES Hackathon or using the Slack channel if you've registered for the conference. Presenters will be answering those questions after the session as well. Up first is Antonina Dilgarakova, who's going to introduce a meta-analysis of pre-clinical studies with complex data structure, a practical example of using a multi-level model to account for dependencies. Hello everyone, my name is Antonina and I'm happy to present our study as a practical example of the use of multi-level model to account for dependencies in meta-analysis. You have probably heard about the problem of low reliability and reproducibility of preclinical findings. Much progress has been made in addressing translational failures, but reports of irreproducibility in preclinical research continue to be published. Meta-analytical research is therefore very important for improving the quality of preclinical studies and needs to be done correctly. Although there are many useful tools to help you at every stage of the process, there are challenges that not everyone knows how to overcome. One of these is the complex data structure that implies dependent effect sizes. This violates the main assumption of a meta-analysis. Non-independent effect sizes can be broadly categorized as multivariate and nested. When the same subjects are used to calculate multiple effect sizes, the data structure is multivariate. This is the case for example when there are multiple treatment groups and a common control group or when multiple outcomes are measured in the same subjects or when an outcome is measured at different times. If the data are nested, you will usually get several effect sizes within a cluster such as a study, laboratory or a geographical region. Both cases are quite common in preclinical studies and in both cases the effect size estimates are correlated. If ignored, this can result in misleading statistical conclusions. Multilevel modeling and robust variance estimation are the most reliable approaches for dealing with dependence. Importantly, they make no assumptions about the specific shape of the sampling distribution of effect sizes and you do not need to know the dependent structure of the effect size estimates. Variance covariance metrics. I now present a practical example of the application of these methods in preclinical meta-analysis. I'll report on our recent work which included certain controlled studies with 21 experiments. We were interested in the experiments that tested the effects of anti-migraine drugs on two outcomes, ongoing and evoked neuronal activity in the animal migraine model that we use in our laboratory. Our main aim was to estimate the overall effect size to get sample size calculations in future studies. To refine the model, we also examined how methodological features affected the results and it identified potential sources of bias. The data extracted for the meta-analysis had a multi-level structure, as five studies reported up to four experiments, while several experiments were conducted with the same control group. To account for the likely correlation of effect sizes, we built a three level model with robust variance estimation and went through the standard meta-analysis steps such as examination of hydrogenity, subgroup and sensitivity analysis, estimation of potential publication bias and risk bias assessment. I will then explain in detail the methods used to assess all of this except the last, the risk bias assessment which was performed at the study level and is therefore similar to that in univariate meta-analysis. We calculated mean differences as an effect size measure and to fit the three level model, we used the RMA-MV function from the metaphor package. The pre-calculated effect sizes are specified by the EE argument, the corresponding sampling variances via the V argument and the desired random effect structure via the random argument. Here we have a mixed effects model. One random effect corresponds to a study and one corresponds to an experiment within a study. The resulting RMA object was then wrapped in the robust function which constructs a cluster robust estimate of the variance-covariance matrix of the model coefficients based on a sandwich type estimator and then computes tests and confidence intervals of the model coefficients. This includes by default a small sample adjustment, the CR1 estimator implemented to improve the performance of the method when the number of clusters is small. Note that the model includes two variance components for between and within study heterogeneity. This makes it possible to examine the distribution of the variance at the level of an experiment and a study. Guidance on calculating heterogeneity for multilevel and multivariate models can be found on the very useful metaphor package website using the link below. In our study, the overall I-squared which is a sum of between study and within study heterogeneity was 90%. About 59% of the total variance was attributed to between study heterogeneity and 31% to within study heterogeneity. The remaining 10% was sampling variance. The extent to which methodological features explained the observed heterogeneity was assessed using method regression. To do this, we simply run the full loop with the same model but now using the modes argument to specify the values of each moderator. In the study protocol, we specified seven factors, one of which the gas used for ventilation accounted for significant variability in the results. The high heterogeneity in the dataset may also be caused by extreme effect sizes, while the pulled effect estimate may be highly dependent on a single experiment. For the models fitted with the RMA family of functions, outliers can be detected using different types of residuals and influential cases based on Cook's distances. For multilevel models, there is an option to specify a clustering variable. Here we assumed that standardized residuals greater or lesser than 1.96 indicate experiments that do not fit the assumed model that is represent outliers and Cook's distances greater than 4 divided by the total number of data points, the common cutoff indicates influential cases. Lastly, the publication bias. To the best of our knowledge of the available tests for publication bias, only Egger's regression can readily be combined with multivariate modeling methods. We therefore assessed the potential publication bias by estimating the asymmetry of funnel plots of the experiment level data using visual inspection and Egger's regression test. To do this, we fitted again a multilevel meta regression model with a measure of effect size precision standard error as a predictor and used robust variance estimation to handle dependence. These approaches showed a clear trend of increasing effect size with decreasing precision. Overall, we hope that this practical example of using multilevel model has demonstrated the simplicity and feasibility of performance standard meta-analysis steps using the metaphor package. The data and our mcdown scripts are available on the OSR website. Please let me know if you'd like to have a look. Thanks for your attention. Thanks very much, Antonino. Up next we have Rebecca Harris with preeclampsia and pregnancy and offspring blood pressure, a multilevel multivariate meta-analysis of observational studies. Over to you, Rebecca. Hi, everyone. My name is Rebecca Harris and I'm a PhD candidate at the University of Wollongong in Australia. Today I'm going to be talking about a multilevel multivariate meta-analysis that we conducted with the metaphor package in R to evaluate the impact of preeclampsia on offspring blood pressure. In studies pertaining to cardiovascular health, systolic and diastolic blood pressure are key outcomes of interest and are both usually reported in primary studies such as randomized trials and cohort studies. And these outcomes are characterized as dependent. And what we mean when we talk about dependence is that the outcomes are correlated, in this case highly correlated. And this means that knowing information about one outcome reveals some information about the other. When conducting a naive pair-wise meta-analysis, dependent outcomes cannot be pulled together because doing so would violate the assumption that effect sizes are independent and from a different sample. And this is because the model doesn't distinguish between the different data structures. So the actual structure of the data is as shown here, where systolic and diastolic blood pressure are measured in the same participants. But in a naive pair-wise meta-analysis, the statistical model assumes that the effect sizes are from a different unrelated sample. And this is true even if the outcomes are different. And this is problematic mainly due to the artificial increase in the sample size. And this leads to confidence intervals that are much too narrow. An appropriate approach to combine multiple dependent outcomes in a meta-analysis model is through multilevel modelling, which accounts for the dependence between the outcomes by specifying in the statistical model how each effect size is nested in the included study. So as shown in the illustration here, if a study reports both systolic and diastolic blood pressure, we can add another level of variance at level two. So this is basically an extension of a random effect two-level model with an additional tau squared variance component for the multiple outcomes. So now we have three levels of variance. The correlation coefficient between systolic and diastolic blood pressure is not required. And this is because this is advantageous because correlations are often not reported in the original study. So we used a case study of preclinemia and of spring blood pressure to illustrate an example of systolic and diastolic blood pressure in a multilevel meta-analysis. So a bit of a background about preclinemia, it's a health condition during pregnancy characterized by new onset hypertension. So hypertension that occurs after 20 weeks of gestation and occurs alongside maternal organ dysfunction or fetal growth restriction. Although the cause is not well understood, observational cohort studies have provided some evidence that preclinemia is associated with an increase in blood pressure through childhood and adolescence. There have been some previous meta-analyses on this topic which use standard pairwise univariate meta-analysis. And this means that they dealt with the dependence between systolic and diastolic blood pressure by conducting separate analyses for each outcome. Another important component of the structure of the data in this particular case study is the longitudinal nature of the effect measures. But because participants in the studies were followed for a long period of time, there were often multiple measures of both systolic and diastolic blood pressure in each sample. Most of the previous meta-analyses selected one time point to extract data from, but there was one previous meta-analysis which included multiple follow-ups that they were independent and therefore violating the independence assumption. Longitudinal data can be included in a multilevel meta-analysis by just considering that the different time points as different outcomes also nested within each sample. So this brings us to our current project and our aim was to conduct a systematic review and meta-analysis to compare blood pressure of offspring brought to pre-climatic and normative pregnancies. We registered our protocol in Prosperum and to search for articles we searched in PubMed, Sinal and Mbase databases from their inception to January 31st 2022. And we searched the citations of included cohort studies and previous reviews as well as conducted forward citation searching using Google Scholar. For our selection criteria participants could be any age including from infancy to older age and they were included if they reported systolic or diastolic blood pressure on a continuous scale and this could be millimeters of mercury but could also be percentile scores or standard deviation scores. We conducted tidal and abstract screening using abstractor and this was done by two review authors in duplicate independently and to assess the within study risk advice we used the Robin's e-tool for observational studies and this was again conducted by two review authors in duplicate. When looking at observational evidence it's very important to take into account that there could be confounders which affect the relationship between the exposure and the outcome. So this is the graph showing the confounders and mediators of interest and the main confounders of interest were maternal smoking and drinking during pregnancy, education or socioeconomic status, maternal parity, age, ethnicity or pre-pregnancy BMI. We also made the important distinction between confounders and mediators because mediators lie on the causal pathway so they could because they occur after development of PE it's not appropriate to adjust for these. So therefore we only pooled results from studies which had adjusted for all relevant confounding factors and did not adjust for any mediators. For our analysis we chose to pool HG standardize mean differences to enable us to pool effect sizes which were expressed as percentile scores and standard deviation scores with those from other cohorts which were expressed in millimeters of macro and because effect sizes were adjusted for confounders using multiple regression models we also included a correction factor when calculating these to account for the fact that the variability around the mean difference decreases as the number of variables in the regression model increases. So this is the code we used to conduct the multi-level meta-analysis and also on the right you can see the data set and how it was set up in long format. We first created dummy variables for systolic and isolate blood pressure to be used as moderators in the model and this enabled us to get a pooled effect for both outcomes separately. And then to run the meta-analysis we used the rma.mv command available in the metaphor package and we specified the effect size and variance shown here and we used the mods argument for the regression to obtain different effects for systolic and isolate blood pressure and importantly we used the random argument and this was to specify how each effect size was nested inside each cohort and we used restricted maximum likelihood estimator for the level two and level three variability. So these are the results from our study selection there were 2,423 unique reports identified from database searching and 12 unique reports from citation searching and after full text screening there were 55 reports of 42 42 cohorts however like I mentioned before because we only pooled data from the cohorts which were adjusted for confounders there were only seven cohorts included in data analysis. So here are the results from our multi-level meta-analysis as shown in the forest plot. So here we have the systolic blood pressure and the standardized mean difference and the diastolic blood pressures standardized mean difference pooled. So there were only small increases in both systolic and diastolic blood pressure and when we re-expressed the standardized mean differences in two millimeters of mercury we had a 1.69 millimeter of mercury difference in systolic blood pressure and a very small 0.65 millimeters for diastolic blood pressure. So here are the results from our risk of bias assessment overall there was a low risk of bias in most domains but there was some concerns due to missing data mostly due to the long follow-up of the cohorts and this led to all studies being rated as some concerns overall. The main interpretation of our findings is that offspring born to pre-eclectic pregnancies do have highest systolic and diastolic blood pressure when compared to norm-attensive pregnancies but this difference was much smaller than previously reported due to the fact that we correctly specified how dependent outcomes were nested in the cohorts and it's not clear if these small differences in blood pressure are significant enough to infer a clinically meaningful difference in adverse cardiovascular outcomes in later lives such as heart attack and stroke. Here are our references and thank you so much for watching. Thanks very much Rebekah. Up next Wolfgang Wichtbauer will be talking about location scale models for metronalysis using the metaphor package. Hi there my name is Wolfgang Wichtbauer from Maastricht University and I will be talking to you today about location scale models for metronalysis and how you can fit these types of models with the metaphor package. So to get started let's first talk about a standard meta analysis and one of its main goals namely to estimate the size of the average effect and for this we typically use a random effects model which says that for every study we have an observed effect for each of our case studies and what we want to estimate here is mu the average effect but there are two sources of variance first of all there's epsilon sub i this represents sampling variance and these variances these v sub i's these are known this represents variance in the estimates around their true effects while an observed effect is not going to be equal to its true effect because of the sampling variance and these are heteroscedastic by construction that is we have bigger studies which tend to have smaller sampling variances and then we have smaller studies with bigger sampling variances and then we have another variance component here tau squared this denotes heterogeneity this represents the variance in the underlying true effects and this is assumed to be homoscedastic that is it's a constant it doesn't vary you can easily fit such a model with a metaphor package which of course you have to first install if you don't already have it and let's look at an example and this is one of my favorite examples the meta analysis on the effectiveness of the bcg vaccine and for each of the studies included in this meta analysis we have the number of tb positive and negative cases for the vaccinated and the non-vaccinated groups and we will use this information to compute log transformed risk ratios and the corresponding sampling variances so let's first load the metaphor package and then we can take a look at the bcg dataset which includes 13 trials and here we have the number of tb positive and negative cases in the treated or vaccinated group and the number of positive and negative cases in the control group and there are a couple other variables included in this dataset one of which the alloc variable we will take a look at this in a little bit now we can compute the log risk ratios with the es calc function we have to specify the type of measure rr stands for log risk ratios then we have to give the function the information so it can compute these log risk ratios and once you run this the dataset includes two additional variables yi those are the log risk ratios and the negative value here represents a lower risk of infection in the vaccinated group and vi those are the sampling variances and which we can see a sometimes large or sometimes smaller depending on the size of the studies and the amount of information that is provided by them finally we can fit the random effects model with the rma function we just give it the effect sizes and the corresponding variances and here are the results so we get our estimate of tau squared we get our estimate of mu the average effect and the corresponding test whether this is significantly different from zero and a corresponding confidence interval on this slide you can see a fairly typical force plot showing the results of the individual studies and at the bottom the results from the random effects model namely the estimated average effect and the confidence interval and since this excludes zero we know that this is a statistically significant effect and showing that the vaccine is indeed effective on average for reducing the risk against tuberculosis but there is a fair amount of heterogeneity in these results we can see this here up top this is the estimated distribution of the true effects based on mu hat the average effect and tau square representing heterogeneity so there might be circumstances where the vaccine is even more effective or where it is possibly not even effective at all to explore such heterogeneity we can use meta regression where we use one or multiple moderators or study characteristics to examine if they're related to the size of the effects to give a simple example say the studies fall into two groups the randomized versus non-randomized studies and we can represent this as a dummy variable which we code zero for the first group the non-randomized studies and one for the second group the randomized studies the analog variable in the data set actually provides information about the method that was used for assigning participants to either the treatment or the vaccine or the control condition although it makes a distinction between three different forms of allocation we will simplify a bit and create this dummy variable by coding this variable as one if the method of allocation was random and zero otherwise now we can include this dummy variable in our meta regression model and we are especially interested in the results down here the intercept represents the estimated average effect for studies not using random allocation and the coefficient for random represents the difference in the effect for studies that do use random allocation compared to those who do not and we can also test if that difference is significant statistically significant which is not the case here let's ignore this though for the moment and look at this forest plot which shows the studies ordered by whether they did not or whether they use random allocation and up top here we now see two distributions the estimated distribution of true effects for studies that do use random allocation versus those that do not what we see here is that these two distributions have different means but they have the same tau square the amount of heterogeneity within the two subgroups is assumed to be the same so again tau square is assumed to be homoscedastic but this assumption may simply not be true which brings us to the location scale model the first part of the model looks pretty much like a standard meta regression model but the difference is now that tau square has a subscript i so tau square is allowed to differ across studies and in fact what we have now is a model for how tau square or log transform tau square differs across studies namely as a function of one or multiple study characteristics so here now we can make a distinction between predictors for the size of the effect these are so-called location variables and predictors for the amount of heterogeneity these are so-called scale variables and they may or may not be the same and of course again you can have multiple such location and or scale variables and I recently extended the metaphor package to also fit those types of models to illustrate this let's extend the previous meta regression model by including random also as a scale variable so like before we get information about the estimated average effect for the non-randomized studies and how different the effect is for the randomized studies but in addition to this we now get information about the amount of heterogeneity for the non-randomized studies and how different the amount of heterogeneity is for the randomized studies as noted earlier heterogeneity is expressed on the log scale for this model so to get the estimated tau square for the non-randomized studies we just take the intercept and exponentiate it so we get around point two and to get the estimated tau square for the randomized studies we take the intercept plus the coefficient that represents the difference between the randomized and the non-randomized studies and we exponentiate this and we get about point four so we estimate that the amount of heterogeneity is about twice as large for the randomized studies compared to the non-randomized studies you can also see this now in the forest plot where the distribution of true effects for the randomized studies has a different mean compared to the non-randomized studies but also has a different amount of variance so this model allows the amount of heterogeneity to depend on one or multiple predictors or scale variables and so tau square now is allowed to be heteroscedastic now it turns out that the model above yields identical results to fitting separate random effect models within the two subgroups which is what i'm doing down here i fit a standard random effects model to the subset of studies where random is zero and where random is one then i collect some of the information from these two models in a table and so we get the estimated mu for the two subgroups and we get the two tau squares and these are identical to the two tau squares that we got from our location scale model but the location scale model is much more flexible for example you can include none one or multiple location and scale variables in the same model and these variables can also be different the variables can be categorical so representing subgroups as we have seen before or there can be also quantitative variables and you can also now test if the amount of heterogeneity is related to a scale variable or differs across subgroups and for this we have several different types of tests so-called wall type tests likelihood ratio tests and permutation tests for example in the model above alpha one represented the difference in heterogeneity between the randomized and the non-randomized studies so testing whether this is equal to zero is the same as testing whether tau square for the non-randomized studies is equal to tau square for the randomized studies on this slide you can see how these different types of tests can be conducted if you fit a location scale model and you look at the scale part of the output here's our estimate of alpha one then you immediately get the p value from the wall type test testing whether this is significantly different from zero if you want to conduct the likelihood ratio test you fit another model where you drop the scale variable so here I just have an intercept so this is a reduced model and then I compare these two different models I get the likelihood ratio test and the corresponding p value and finally if you want to do a permutation test now that reshuffles the data repeatedly in a certain way so there's a certain element of randomness which I can make reproducible by setting the seed of the random number generator here then I get the p value here in the output from the permutation test now in all three cases here all these tests suggest that there is no significant difference or sufficient evidence for a significant difference in the amount of heterogeneity between the randomized and the non-randomized studies as a more elaborate example let's consider the data from the meta analysis by banger drowns and colleagues who examined the effectiveness of a particular intervention for improving educational achievement in this meta analysis the effect sizes are standardized mean differences and the studies differed in their sizes so we have small studies and quite large studies and also in the subject matter that was examined so 28 studies looked at mathematics performance nine studies looked at a science subject and 11 studies at a social science subject we will now use both of these variables as location and scale variables in our model and so once we fit this model we get these results and so here the intercept corresponds to mathematics for the average effect size and amount of heterogeneity the coefficients here represent the difference in effect size for science and social science compared to mathematics or science and social science compared to mathematics with respect to the amount of heterogeneity and the coefficient for sample size represents the slope of the relationship in terms of effect size and amount of heterogeneity and here we do find some significant relationships in particular this one and this one the results therefore suggest that larger studies tended to yield smaller effects and that studies that examined the effectiveness of the intervention in science subjects tended to yield more heterogeneous effects but there was no evidence to suggest that the average effect itself differed for science subjects so this example shows that there can be different types of relationships in terms of the location or the size of the effects and the amount of heterogeneity these types of models therefore open up the possibility to examine entirely new research questions for example comparing different types of interventions not only in terms of the average effect size that they yield but also in terms of the consistency of the effects but it should be pointed out that due to the increased complexity of these models you tend to require a larger k or number of studies to obtain meaningful answers here and given that there are different types of tests for testing scale coefficients these wall type tests like the ratio test and permutation tests the question arises which type of test should you pay most attention to this is something that we are currently examining in a simulation study and finally I am looking at the possibility to fit such types of models in the context of the rma.mv function from the metaphor package which is for multi-level and multi-variate models although that type of extension is not entirely trivial this brings me to the end of this talk on this slide you have the references and if you have any questions comments or suggestions on this slide you can see how to get in touch with me thank you for your attention. Thanks so much Wolfgang Shinichi Nakagawa is next with meta-analyses with missing standard deviations with log response ratios. Hello my name is Shinichi Nakagawa today I'd like to tell you about our new method on log response ratio and some standard deviations are missing. First I'd like to acknowledge my love members at UNSW Sydney Australia and my co-authors Vex, Losia, Dan, Alistar and Wolfgang the cat he sometimes takes human form it looks like this okay I think some of you may be familiar with log response ratio but everybody probably knows coins D, edges G, this is just a small sample size collection the version of the coins D more generally they are known as the standardized mean difference and for the reason I'm gonna show so D equal me minus another mean divided by cool standard deviation log response ratio is a ratio of two means and it's log and it turns out to be the most used standardized or unique less defect size in ecological evolution so it's important defect size for evolutionary biologists like myself okay so you know the missing data everywhere and the meta like the data are no exception and generally looks like this then we collect trying to collect descriptive statistics from papers so you get mean standard deviation sample size for the control and the same thing for the treatment group or experimental group and quite often standard deviation missing sometimes occasionally and I'm missing but that's very rare so when that happens what we usually do is just delete those roles or cases because you can't calculate log response ratio sampling variance which we will see the reason why bit later on so this when we delete these cases it's called complete cases approach because we use just to use the cases where SD are not missing but this will lead to some biases overall mean estimate or other meta like parameters and this paper points out this and this paper says we should use multiple invitations otherwise or the meta analytic results are biased often yeah I guess and another interesting thing is a survey included in this study showed 75 percent of meta like data in ecological evolution have missing SD so it's a very common problem and multiple invitations certainly a solution but problem is when we run the complex meta analysis multiple invitations very difficult to implement for example when you have effect size multiple effect size from the same studies or ecological evolution we often have multiple species on the top of multiple effects per study then we have to account for what we call phylogenetic relatedness that gets very complex model in this case multiple invitation techniques very hard to implement so we I was kind of looking for the easiest solution compared to multiple invitation and I couldn't come up with tell I came across this paper by Don Castan Spake and Rebecca Spake the Bex you saw in the second acknowledgement slide uh in this study they did they conducted simulation study and they proposed this formula there with CV and I'd like to explain this so this is log response ratio you've seen this before important thing is the meta analysis you need the sampling variance inverse of this is going to be wait for each case is for each study and this has a SD mean sample size and you can see the SD divided mean can be written as a CV coefficient of variation and they propose rather than using the CV specific to study we can calculate overall CV average CV across all available study and you can use that and that would lead to less biased or more accurate overall meta mean and you are wondering why is that and the reason being they show this in the simulation is when n's are small and quite often many studies in the ecology evolution lots lots of study has a small n's or sample sizes SD are estimates so inaccurately yes those are estimated not true SD this will result some bias or an inaccuracy over all means so that's quite cool but a finale reading this paper toward the end of this paper they said actually um this CV average CV can be calculated there's missing data because average CV can be calculated without or you know excluding those missing cases and you can use average CV for those missing cases and I thought oh this was sort of everything this is why we conducted this new study and the proposed four different uh new method which I'm going to tell you about so as you have already seen this is a donkastas spec um estimator and they use for sampling variance overall response ratio they use average CV we made two improvements for this is we actually used it looks scary but it's just a weighted average CV giving more weights on the study with higher sample size another one is that those two latter terms it's a small sample size correction it's been like a hg to the coins d so what's the study so we had a four method but I want to just uh share with you the two methods of missing cases all cases so you've seen this before grades are missing data and we use this uh sampling variance let's call the v tilde that's our estimator from those missing a standard deviation we replace with our v tilde or all cases method we use all for all regardless it's missing sb or not use this v tilde for all the studies or all cases I guess so we conducted simulation and this our simulation actually we buried a degree of uh missingness from five percent to up to 30 percent it's surprising thing is bias log response ratio here is a bias in an overall mean but overall meta-analytic mean and all cases performs really well close to zero zero bias next to missing cases what really surprising so full data one is the family simulated that data set didn't have any missing data that's really counterintuitive because that's should this should perform least biased yeah with least bias but you may remember this don't don't cast on spec paper what they showed is sometimes each study has a two few sample size and that affects the accuracy of the meta-analytic mean estimate so that's what's happening so actually turns out to be using the new estimator our estimators for the sampling variance would perform the best and we don't show this here it performs in like estimating heterogeneity coverage everything is better for all cases and the missing cases compared to full data which is surprising so conclusion all cases method the best one but we should also probably run the missing cases as a sort of sensitivity analysis see you know you have a you get consistent result and what we can say the biggest conclusion is all future meta-analysis can use our method because regardless of how complex or how complex your meta-analysis model models are you can use this method easily because only you need to get you need to get a weighted average of cb then you can plug in one thing i didn't tell you about is assumptions normality uh log response ratio is passed proposed by hedges that's the same hedges hedges for the hedges d and this formulas seem to fail and there's a lot of counter data or non-normal yes so lots of counter data in ecology and evolution so we need to be careful in such case we propose several different um couple of different solutions for this all in this webpage it's connected to paper you can find this in the webpage for the paper or it's here and thank you thanks very much shinichi up next sabina patsul will be talking to us about using multiverse and specification curve analyses as an assessment of generality of effects for marzems over to you sabina hi my name is sabina patsul and i'm a phd student at the technical university of munich and today i would like to talk to you about how we use multiverse and specification curve analysis as an assessment of generality of effects for our meta-analytic structural equation model where we're looking at the relationship between creative potential and creative self-assessment measures i would like to start off with a quick theoretical background so basically it's assumed that everyone has a certain amount of specific of creative potential within them um two indicators of creative potential for example um divergent thinking and intelligence both of which correlate with creative achievement however um creative potential does not necessarily lead to creative achievements um as the relationship might be partially influenced by factors relating to the creative self such as for example creative self-beliefs this is why we chose to look at the relationship between creative self-assessment measures and two indicators of creative potential being intelligence and diversion thinking we use structural equation modeling specifically um t s s e m with an extension for random effects models and the vpl approach has multiple studies reported more than one effect size and we assume that to be dependences among effect sizes to test our robust or structural equation model is we use subgroup analysis to test um whether the parameters uh estimates are actually um equal across um csa type so creative self-assessment types and age groups um just a quick heads up these are all preliminary data so please don't interpret them and we just wanted to give you an idea how our models look like so for example we are looking at the relationship between divergent thinking and creative self-assessments and testing whether they are mediated for example by intelligence we're also um as I already told you doing subgroup analysis so we look at the different models depending on whether children's were assessed or for example adults were assessed um on downside of meta analytical structural equation modeling is actually um that you're it's kind of limited when it comes to how many uh different moderators you can include at the same time which is why we chose to uh also apply or use another approach to test the robustness of our model being models and specification curve analysis um as I already told you we're looking at different age groups um and different creative self-assessment types but there are various other mediator variables for example um what kind of divergent thinking outcome was used um the test method modality of divergent thinking whether there was some kind of time condition in the tests and so on and all these different moderators you could say could potentially influence our primary studies and in that they could also influence our moderator or our or metadata to structural equation model um when doing a multiple analysis these um different factors which I told or called moderators before I actually called which factors as you're thinking about which uh data could possibly been have been analyzed in the primary studies and you're also looking at how factors um which basically um tell you how the data is meta meta analyzed and what you're doing is you're combining all these different kind of which factors and the subgroups so for example um whether female sample was used children were investigated fluency was reported as an outcome and so on and you're combining these with the different how factors and these lead to a lot of different specifications and for all of which we are computing meta analytical summary effects and at the end we're basically comparing all these possible meta analytical summary effects and we get a mean over these summary effects and it tells us how robust um a correlation is depending on which data was analyzed and how the data was analyzed and yeah we're doing the meta of these multi version specification curve analysis for all our bivariate relationships so for csa and divergent thinking divergent thinking intelligence as well as intelligence and csa um just as a way to get an idea of how robust our meta analytical structural equation model actually is um I would like to show you our specification curve um from the relationship between the version of thinking and creative self-assessment measures um to give you an idea how it could look like or it looks like at the moment and yeah so at the top part of the model um you can see and the dark line which are the summary effects from all our different specifications and these colored areas are the respective confidence intervals so for example um this summary effect over here with this quite large um confidence interval then we're going down to the bottom part of the plot um was uh computed using a total sex sample um with adults verbal modality and down here you'll have the the how factors so this gives us um the information on which kind of which factors and how factors were used how do we are combined and what kind of summary effect they yielded we also get information by the colors um so for example or uh warmer or hot colors represent combinations of which and how factors that only included a few effect sizes and cooler colors such as blue or green represent a combination with more effect sizes and when looking at our graph you can see that um there are a lot of effect sizes in the middle of the plot and that actually the magnitude doesn't increase um a lot so most of our effect sizes are actually in the area of point one to point two and which basically speaks for the robustness of our effect of especially when you're considering all these different kinds of possible specifications next off um we looked at or we look at the look at the um total summary effect from all these different kind of specifications um which is a correlation of point 17 and as you can see 50 percent of all our different combinations of um which and how factors yielded a summary effect between point 13 and 21 so our um correlation seems to be quite robust we can also use a parametric bootstraping approach to do inferential testing the red curve represents our specification curve and the gray area is basically the curve under the null scenario of the possible zero effect which clearly deviates from our specification curve so at least for the relationship between direction thinking and creative self-assessment measures are the included correlations seem to be quite robust um so um you could say it doesn't matter that we can't include um so many different moderated variables and um as the correlations seem to be quite robust um that's it for my talk I hope it gave you an idea how you can multi-verse use multi-version specification curve analysis to test the robustness of your meta-analytic structure equation model as I already told you we would like to do this the same analysis for the other two Bavarian relationships as well to test how um robust our model is um yeah if you're generally or if you're interested in our project feel free to write me an email we also did a preregistration of this study and that's it for today thank you thanks so much Sabina that's it for this session I hope that you've enjoyed it it was a really fascinating set of talks lots to get your teeth into we'll be coming back again shortly but for now that's it please keep your questions coming in via twitter the presenters will be around for the next few hours or days so if you tag them in a question they'll be able to get back to you as soon as they can thanks very much


Hey folks, Jesse here at Mountain Vidalge. Beside me today I have a 2019 Ram 3500 heavy-duty granite black with the sports package monotone paint all the way through. We have our flip-off mirrors, heated mirrors. We have proximity. You don't have to hit the button to get in anymore. You have your box camera. We have our spray-in bed liner. My flaps go with it. The monotone paint carries all the way around the black bumper, finishes the trim. Our gold badging back-up camera with our class 4 hitch. We have our wiring inside with our LED headlights. We have our fifth-wheel tow package, prep package inside with our gooseneck on this truck. And coming up, we're going to have the video on the inside. OK, folks, on the inside of this truck, we have our new 12-inch screen state-of-the-art technology. This truck also comes with our integrated auxiliary switches. We have our air height for the trailers when you're loading. We have a sunroof that goes with it, power slider in the back with defroster. You can see on the dash, great setup, friendly to the eyes. This truck here, folks, is great. We'll haul anything. We'll look great in your yard. Come on down to Mount Vidoge and have a look. Come and see me. My name's Jesse.


Hey guys, John Hammond here and we're looking at Pico CTF 2017. We just got into level 2, so now we're moving on. We got the Michelinius category opened up again. All the other categories are opened up again. So let's start with our typical procedure. Go for the lowest point challenge that we can. Hit that low level fruit first. So 55 points yarn, the first challenge in the Michelinius category. So I was told to use this Linux strings command on yarn, but it doesn't work. Can you help? I lost the fog in a binder somewhere and I would like it back. All right, so let's copy this link. Let's head over to our shell. We've actually got all this stuff in its own folder, but let's make a directory for level one. And let's move everything into level one. Okay, cool. So now in level one, let's get the shell out of there. Good. And now let's just make directory level two, cd into level two. And let's make this challenge yarn, cd yarn, wget to pull the file here. And we've got it. Okay, what is this thing for one thing? Let's run file on this to do some basic reconnaissance. Looks like a binary. We could just run it if we wanted to. Ain't here. All right, cool. So we've used the strings command before in previous levels or previous challenges. So let's try one more time. Let's run strings on this file. And there's a lot to look through or not. There's not a whole lot to look through. What the heck? Hmm, you can pipe this into less or more if you wanted to like look through it to scroll around and actually have it be paginated or like buffered paginated is that a word? Separating the pages that you can look through and scroll through in your shell. So nothing really here. But what are the hints say? What does the strings command use to determine if something is a string? Is there an option to change the length of what strings considers are valid? Oh, huh. Let's check out the man page for strings. Strings. Print the strings of printable characters and files. Does it say okay, strings prints the printable character sequences that are at least four characters long or the number given with the options below. You can use tack a scan the whole file all knowing this default behavior. Huh, it only print from data. Hmm, you can specify a minimum length, tack n, tack encoding, huh. That might be handy. So tack e for encoding, which we can change to L and B our default for ASCII single 8 bit. So let's try n minimum length of one. And let's try to change the encoding at some point. So strings yarn, tack n one, need a space between those. Okay, that's a lot of stuff. Now we just get everything. Let's pipe that into less and see if there's anything that's in there. Submit me for I am the flag. Huh, interesting. How come it, I don't know wonder why it modified all that stuff. Is there a way to change that encoding that will allow us to like see that? Yes. Huh. Let's check out what the actual hex is. We can use hex edit to view it. If you don't have that installed, so do I have to install right. Looks like it has just no characters in between it. So that 00 is what's determining. Okay, this is a separate string, and it's getting cut up. And it's obviously less than four. It does this every single time. Submit me for I am the flag, it cuts it up into characters that are only in sets of three, less than four. Interesting. So we change it down and we got to work. You can control X and no to break out a hex edit if you want. So let's steal that. I'm a nano flag dot text. Well, didn't have that in my clipboard in the way that I wanted to. Once we've got it. Now we can cat that out and you remove all new lines with transform tacky to delete new line characters submit me for on the flag. Let's put that into a new file. And let's move that just to regular flag dot text through that in my clipboard. And let's go ahead and submit it. Awesome. You're up 55 points. Cool. That was an interesting one. That's good things to take note of. I've seen that encoding take advantage of in a sans holiday hack challenge at one point. But I always forget that it determines the length of a string based off of four printable characters. So that was kind of a neat trick that they were hiding on us. Always useful to check out the man page for every single tool that you're working with. Because if you just got a gut feeling, this is a tool that I need to solve this challenge. But I'm not getting something right. Take a look at what else that tool can do. Because you may learn something new. And that's what capture flag cybersecurity computer science. That's what all the stuff is is all about. So thank you guys for watching. Hope you enjoyed this video. I need to give a special shout out to my Patreon supporters right after I successfully renamed this directory. All right. Hey, you guys are fantastic. Thank you so much. Every little bit that you provide really, really helps me and inspires me to keep making more cool content. So $1 a month on page one will give you the special shout out the end of every video $5 a month on page on or more or more. Hey, I'm not biased or more will give you early access to everything that I create on YouTube. And that's it. I don't have any other incentives just yet. But hopefully you guys can make me think of some other cool stuff. Maybe you can have a live, I don't know, call in with me or something. Hey, whatever. If you want that, I don't even know why that would be useful. Sweet. If you did like this video, guys, please do press that like button. Maybe leave me a comment if you're willing to subscribe. And if you want to support me, check me out on Patreon or visit my website www.johnhamman.org. See you soon.


Okay. Hello, everyone. This is Melissa. I'm reviewing the PayPal, PayPal, PayPal, PayPal, PayPal. Let's go over it. So Jyra did an excellent, excellent job in the room this morning. Called PayPal short, got the drop. He had the stop at the right place. He got out and started a ton of people in the room. But I was back before the open and then obviously had time to get it. And this looked good, this looked good, this looked good. And I never got out of this with profit. And looking back, it was just, I just didn't react quick enough. I ended up getting stopped glad that I did not jump right back into it. So this did warrant what is called a retake today. But I really, really didn't think it was ready until later timed with the market. It's going to go over here to the market. Once the market started to drift off, which it absolutely did. So let's just see exactly the time it was. Oh, there you have it. So market really set the high of the day by 11 o'clock, 11am. So PayPal had a beautiful sharp drop and a big move this morning. But one of the things with day trading, one of the things I love is the fast exits, but you have to be on your game. And I think it was just too many things going on with me this morning. And my anticipation was as well, 175 was the first target. But once it broke that, I felt like it could continue even further. And it very well might, but it just didn't today. It just did not today. But this was a really nice move in here. Honest to goodness, you could have gotten almost $3 out of it. And some people did get out of it, like I said, in the room. But I thought it had more leg in it today. Once it flipped though, it just then started to get away from itself with the market. And this, this was by no means a revenge trade. This wasn't a revenge trade here. This was a real sell off and a real set off that set up that I saw happening. But I'm really glad that I ended up retaking this and doing this again because quite frankly, it more than made up for the stop this morning. And I really think if I hadn't been, you know, voting this morning, then running back to getting before the open, I would have actually ended up getting out of that if I had been more focused. So you've got to be focused in the morning. Number one with what you're doing and number two, your entries and exits. So I, I liked this last night. I saw this last night and it'll be interesting to see if this is going to have some follow through. Remember, today's only Tuesday with the election, everything going on. I think people forget what day of the week it is, but it's only Tuesday. So this says Wednesday, Thursday, Friday, three more days left to go. And then remember last week, the option I called in this one, the very last day last week, the very, very, very last day. So I can't wait to see what this does. I'm also watching the market too. Again, it's election night. And that let's just look here quickly at the market because we were selling off a little bit into the, after the four o'clock broke 335, then we are now, this is pretty much just neutral. Now we're just neutral here. Anyways, nice move here in the PayPal, PayPal was a short, you could have done in the morning aggressively got out. And then you could have actually retaken it if you stopped in the morning, or if you wanted to retake it and still get out in the morning, you could have retaken it. So I think a lot of people did get out in the morning and it just goes to show you again, the focus has to be there, but I knew was a good gap. I'm glad though, that I didn't jump back into it and looking at the chart, it really didn't set up until later. And again, was timed very well with the market. This was an earnings, a gap, just so you know, it's still earning season, this entire rest of the month and into the beginning of December. So nice movement here in PayPal, big move, move in the afternoon, had to catch it up the right time, which we did. And we'll see what happens to this going into the rest of the week. Good luck everyone. The golden gap course for November is this weekend, November 7th and 8th. If you're interested, email me at melissathestopswush.com to sign up. Stay safe everyone.


I'm a singer oh I almost fell I've stopped vlogging and I've become a full-time singer. Oh, that's unfortunate for the world So what are we doing here? Doing multiple things out This just so it just feels like we're playing Minecraft at this point with better graphics You just walk right through the mud. Yep. I've just accepted it and I'm gonna be too stubborn to move out All right, let's stop ranting. So we're here to film some drone videos because we're in a really weird place But we're also here because we have nothing to do The parks aren't gonna open anytime soon Can't do anything you're stuck inside your house life was boring So we're gonna we're gonna make a positive out of this We're gonna clean up some areas around the place over the next While until this is over with Go right it's a path. It doesn't look like it right now, but it is somewhere See that the dark forest over there, that's it We're gonna pick up trash we're gonna do this a lot Because yeah There's nothing else to do and with what's going on in Italy with the clear waters flowing China seeing blue skies finally I Mean we can do Somewhat of a small part here and maybe other people who are bored at home You can go out and do something similar Even if it's just a small little area Oh Nature insiders Corona oh can't say that word right now So there's a certain virus spreading that we can't say It's the same as the beer. Can I say that no what? Oh, I don't know YouTube's a scary world right now Okay, well I'll turn the camera back on when we have something interesting to film Right now we're gonna go fly the drone that we're gonna start picking up trash I get all righty we Are gonna fill up one garbage bag a day of Trash until this quarantine is over. We are still keeping our social distance. We're not going in groups of 10 or more But we're gonna clean up areas in our neighborhoods like that. We normally fly like near Wonderland and stuff or Areas that have a lot of trash. Yeah, we're gonna clean that one next But we're gonna spend some time picking up. There's a lot of coke cans beer cans newspapers and stuff in this forest as you can see there's Trash so this is our goal our commitment to Kind of make this a little easier and it's not it's not even work because honestly, we're stuck inside We're talking in our discord server. We're not doing anything. We're bored out of our mind They just announced bars and restaurants closing down. So I just lost my form of income outside of YouTube so I mean until I can get work and until or unless the government pays us But I mean, we're still we're good like, you know, we're not in a sticky situations There's nothing to point about what better way to make this more fun and at least we're doing something Then to pick no don't give an update. No. No is that my legal?   I mean we can make a small Improvement in our neighborhoods through this. This is just something. I mean we have a YouTube channel. We have 18,000 followers So we're trying to The saying that there's a saying that I recently just adopted a spy to inspire before you expire But yeah aspire to inspire before you expire. I don't know where I saw it. I saw it on Facebook But it's stuck with me because you know, we have 18,000 followers We're in a really tough time right now. So I wouldn't call it sticky. I would call it like an unfortunate series of events and Yeah, what better way than to Get out of the house. I mean, I'm I'm a guy that hardly gets out of the house I might as well get out of the house breathe some fresh air and clean it up while I'm out here for an hour All right, I guess we'll give you an update once we fill this bag There is so much cleaner There was a lot more garbage than we even thought we were like yeah, we thought we're gonna have trouble filling a bag But that bag's heavy try lifting it just so they there's so many beer bottles and beer cans It's pretty heavy bag and Man most it's drinking. It looks like this people come here to drink Like a lot of drinking actually. The good news is that's all recyclable So we'll be able to drop that off at the recycling facility Haven't decided if I'm gonna put it in my own home's recycling bins or just go drop it off quickly I don't yeah, you can I just don't know if they're open yet or if like they're open because of stuff going on But so much cleaner One bag a day we're gonna try and commit to that goal Obviously, there's gonna be days where we can't do that. So we'll just make it like two bags the next day or something like that So yeah, we're gonna try and stick to our One bag a day and obviously this could be days where we can't do a day because there'll be like family events or so We'll do two bags the next day. Yeah. Oh my god for it. This is what again. We're just gonna do this until The quarantine's over This happens to be our neighborhood so we're just gonna Obviously That's what we were talking about because I'm I'm obviously so to shape if you sell the time lapse I obviously have to work out more. I was looking at my heart rate. It's a hundred and it was a hundred and forty Yeah, so it's a good way to work out gyms are closed Yeah, everything's pretty much closed so might as well get out do something nice in the process again comment down below If you guys are gonna try and do this or you're gonna do something similar Or what are you doing during the quarantine to keep yourself entertained or do something nice? Hopefully this inspires even one person to go out and even pick up a little bit of trash if possible I'm sure we'll get at least a person. Yeah, we live in such a beautiful country and like oh my god What was what was the common theme of garbage here alcohol alcohol like or burnt things? Yeah And clothing as weird as that sounds. Yeah Well, I'm oddly enough a lot of plastic lots of plastic like items that didn't belong here like PVC piping and I don't think that's PVC piping I think so I'll check I'll look it up on the internet before But yeah, awesome. That's it for today. I got to go shower. I fell. Oh, yeah, I fell multiple times Oh, it's not even It looks so muddy is my butt. I know you're good. What really yeah, okay? Well, my white jackets ruined but that can go in the wash. Oh, well worth it Life goes on life goes on they got hot a yeah one from being way too cold flying the drone to being Way too hot now, but yeah, this is clearly a place where teenagers come to drink or adults whoever it is If you're watching Come the next day and pick up after yourself All I know is this look like some fun house party All right, thanks for watching guys, we're gonna go clean up and drop this off at a facility or in my own personal recycling Yeah, follow us on tiktok amusement insiders plural Shameless promotion tiktok shameless shameless promotion We're now nature insiders the amusement insiders is ruined for the rest of the year, so we're nature insiders and Yeah, we should do like what one wonderlands clothes like just go in so sneak inside and climb up Oh, that would go well. Yeah, if anyone from one of us watching this right now ban him only I wasn't a part of this that wasn't my idea Although if you do see two suspicious men one, I am not joining you see two suspicious men climbing your rides It was us Great now someone actually is gonna go do that and they're gonna immediately assume.  Obviously we can do better I'm not trying to say that this is the most amazing deed ever, but we're just trying to get out of the house because we're bored and Yeah, hopefully this inspires you. I know that's repeat. Anyways Don't forget to hit that like button I think before you ended I should really make it clear that I was joking and I do not hi at all Mean go and sneak in Wonderland or any theme park. Please. It was a joke Relax, I know joking is not allowed Share this video to help others find something to do during this quarantine if you want to we're not forcing to share But yeah, sure it if you want if you share a video of you guys doing this We'll make a montage posted on YouTube.  To put your email right? Oh, they can't you can't tell me what to put their email under the comment section of a video Your email they send it to your email Oh, or you just tell them to follow us on amusement insiders.  It's crazy I have a shirt that matches this, but I'm not wearing it. I'm just swinging run shirt.