George Bruno with 21 report. We're at the 21 convention the final event of the decade And I'm having a conversation with Goldman otherwise known as Goldman Unleashed. Welcome. Thanks, George. Yeah, how'd it go? Went well. Yeah went well different venue, but same cool audience and I felt comfortable on stage Had to get the message out there. Yeah, I noticed you're sporting a new look now. Yeah with the more full beard Yeah, well, it's it's fitting with my lifestyle right now I'm kind of on the road exploring America in nature a lot and just out there as Goldman the Explorer Yeah, yeah photography is part of it. Of course, geography drone video drone photography, and I just got into underwater Photography Wow. Yeah I'm up in the sky. I'm underwater and I'm on the streets. Wow. What is the most mind-blowing? Oh Photography you've done to date. Oh, man. I I would say last year when I spent the month in the Florida Everglades I was out in Everglades for sunrise every morning for a month and I saw some Very very violent animals killing each other Yeah, alligators eating boa constrictors That was probably the most amazing experience and I got perfect light where a giant 10-foot Boa constrictor was getting eaten by a 10-foot alligator in perfect light Wow interesting. Yeah, like a nature show on TV. I was also so happy with that one. Wow, that's pretty cool Would you speak on today? I spoke about creativity So how to access that creative part of you and what the benefits of being creative are That was my topic of my speech. Yeah, does everybody have creativity in that? I believe so I mean some people of course more than others some people are just born for it They have this position some people are just very talented, but for the most I mean everybody's got some Some inkling some spark of creativity and I really wanted to encourage guys Especially to dig deep and figure out what that spark was and just turn that spark into a fire because it can change your life What about the whole I? Remember back in the 80s and 90s. There was a lot of left brain right brain kind of stuff Yeah Does that any of that? Study apply to what you do.  All right. You just you just entered just a whole new element just came in Yeah, the soul Wow, you just you really blew up blew that left right brain paradigm right out of the water the soul Yeah, yeah, the soul The soul. Yeah, and I've explored the soul through many different mediums before and Tell us about the different mediums. Well plant medicine. Yeah, that's what I'm talking about mostly, ayahuasca ayahuasca was something that Legitimately changed my life. Yeah, I wouldn't be here if it wasn't for those ayahuasca ceremonies And that navigating that soul navigating that it allowed me to do it brought me to those places where I could do it Yeah, and I'd never been before never access that part of me before. Yeah, and it just I mean Goldman unleashed there's a reason I use that word unleashed because that was unleashed that Goldman is what was really inside me Yeah, and ayahuasca Made him come out. Yeah. Yeah. Wow What are the effects the long-term effects of and I'm not talking physiological But how does it affect the life of a person when they are unleashed? I? Much more open I'm gonna speak from personal experience and I'm trying like other people I know that just the openness that you're able to Project whereas before you feel like you were kind of living life very closed or in boxed in and when that happens You just you want to express yourself more you want to understand other people more You just want to put yourself out there, and you're not afraid You're not afraid of yourself anymore, and that's what ayahuasca did. I confronted myself My very very dark parts of myself confronted them saw them for what they were Was terrified for a big portion of it was very very ill for a portion of it, and then but I know they're there And I know the strengths and weaknesses, which is that self-knowledge is just power. Mm-hmm.  That's something that everyone could benefit from right? Yeah, self-knowledge, and that's a big part of my road trip now Is I'm spending a lot of time in nature, and I think the nature is the one place where you really want to explore yourself Without you know psychedelics or you know crazy plant medicines nature is one place where you can really go and Figure out I mean set your mind free Observe nature see what she's telling you But that's the one place if you're really interested in self-knowledge Every time I go I learned something new there. Yeah, it's amazing Years ago. I heard about a lady that was counseling another lady after she lost her husband And she said One woman asked how do I deal with this grief and she says Go out and lay down in the ground and let the earth minister to you. Hmm and I was like blown away by that and She did it and she felt like the ground Actually helped her in her grief She wasn't like literally in touch with energy in the earth or something to that effect. She was grounded Grounded. Yeah. Yeah And I've heard that people who are grieving need to go and Literally lay down on the ground with naked if possible. So there's no interference I knew a woman who would go to the beach without a beach towel. Everyone has beach towels and umbrellas She would just go and lay down on the sand. That's what I do and she just yeah, and she said she felt like that That it was affecting her in a positive way. Oh, yeah, this is I mean part of my trip I'm always you know going to bathing myself in mud going that's hot springs cold springs Oh, they're burying myself in sand, but I can now that you mentioned it as part of the grieving process I can see how that really would help because I think we're all grieving in some way Yeah, like it's a spiritual grief and this is what I've noticed in America. It's this Sick spiritual war that's going on. Yeah, so we're all we've got that hurt inside and getting grounded and back to nature Not to sound too hippie-dippy-ish, but man, it makes a difference Yeah, and it just it might sound silly to people, but just God don't knock it till you try it Yeah, like it might be just what you need especially in our plugged in wired lives where we're never far away from Scream. Yeah Yeah, that's a Digital detoxing thing. I noticed it's a trend right now digital detox It's a trends if because I'm using Airbnb a lot and a lot of these places They're kind of off the map. They're close to national parks and a big buzzword is digital detox And just like being off the grid So you don't have any cell service or anything and that's how they they promote themselves And I go there and I'm meeting great people very successful people, you know Millionaires politicians celebrities are going to these places to leave the city and just cut themselves off and be out in nature And I am all for this trend because everybody there is just like Man, did I need this? I needed it big time because like I make my living online And then when I took my couple weeks off completely it was just my brain felt refreshed So I'm hoping this trend catches on and I think it will because it works. It's not just a fat Well, it used to be an amenity, you know Wi-Fi internet Yes, all like the selling points now a selling point is no Wi-Fi no internet. Exactly. Yeah That's amazing. What are the effects of I? guess maybe Digital intoxication then what do you think? That was so for me especially because I was pretty active on Twitter for a while before I got suspended from Twitter Would I actually ended up being a good thing? But Twitter especially in social media like that. I found the dopamine just it just cut down on my ability to focus And because I did pride myself on my long form content and my essays and my books and my emails Yeah And when you really because you can get good at Twitter or you know if you put your ideas out in a certain way It can be fun because you get the message out there more and more people do it It's good for your business. Yeah, it feels good getting the information out there and it is fun like making a stir Mm-hmm, and but all that it does cut down on your long-term ability to focus so taking those breaks and Getting back into the deep work is like vital if you want to stay creative I've recently discovered mushroom coffee and the new tropic effects of mushrooms It's like a deluxe. No, just Like lion's mane clarity and focus Rashi shaga and these literally they They like if you're reading a book your retention rate of what you read is Higher than I don't use these things And it gives you a clarity and a focus and energy Unlike caffeine which physically amps you up this does it to you mentally The whole mushroom family the whole fungi thing Yeah, it's it's medicine, isn't it? Oh, they are something special. Yeah, the fungi. Yeah, it's something very very special Yeah, yeah, and I've always been a fan like with psychedelic mushrooms and micro dosing I think it's very good for especially for certain personality types like mine You can get a lot of benefit from that But yeah mushrooms in general that whole family is just they're amazing if you do the research on them And you find out what they actually are are they plants are they animals closer to animals and plants what yeah?  Yes. Oh It's better than a lobster like so good. Yeah, so that mushrooms or something else. Yeah Yeah, glad we can talk about mushrooms. They are close to my heart for sure. I noticed that My friend that you met earlier today she She goes mushroom hunting for origin. It's a very European thing Loves it. Absolutely loves it. I say, how do you know what's a good one?  They they can tell they have it So it's interesting. I'm texting and I noticed Well, I didn't when I was looking at poisonous mushrooms these red ones with white dots will kill you I know that and when I look at the emoji for mushrooms like on the average cell phone with my phone The mushroom emoji is a red mushroom with white dots on it And I wonder if those creators realize that that will kill you I think they were copying the Super Mario mushroom Everybody's familiar with that's that's a that's my inkling. It's interesting. Yeah The 21 convention has evolved you've watched it evolve Comment on that Yeah, it's involved It's fun I feel like I'm evolving along with it too Which is great because it's always good to reach different levels of you know creativity maturity, but yeah I've seen it grow from just you know rag tag group of guys who are trying to pick up girls Yeah, and so like a serious life-changing event where it's just gone full spectrum. Yeah, and like just reach different levels So it's very nice and I think important Very important evolution is very very important, especially in a counterculture like this like it has to evolve, right? You can't stay I mean if you figure out there's some really evil things in the world Yeah, feminism sucks, but what are we gonna do about it guys, right?  Yeah, or like how to pick up chicks Yeah, like that's which can be done in an afternoon. Yeah, exactly. Yeah. Yeah, I mean we We're so much more than a how to get laid Workshop, yeah, there's so much more going on here and that's I think mainstream society They look at men and they love to just keep men shallow and dumb right give the guy some bacon and beer He's happy they can I get my blowjob every once in a while. He's happy. Yeah, where it's like the depth of Men the men here the speakers here just the the depth that they have the lengths They're willing to go to gain new knowledge is mind-blowing and amazing I haven't seen it anywhere else and like I was working in the educational system Which didn't hold the flame to the 21 convention and the knowledge that's here. Yeah Yeah, so it's very exciting and like I said important. That's the it is important very important. Yes important Your voice is important in the man a sphere and in the men's community How did you find your voice? Oh Man my voice It when I first started writing in the beginning I didn't I didn't have my voice So just kind of like putting the stuff out there When I started getting feedback From my information which was mostly about sex and seduction why I first started but it was like I had my angle like Yeah, it's this is how you pick up girls But the important thing is you want to form a connection like it's I don't want to say a spiritual connection But if you're picking up a girl like what is that connection? And that's what I was good at my writing I was exploring the feelings behind it and The emotions that were going on like the real emotions and when people started giving me feedback on that I'm like, oh, okay. I've got a talent here. It's time to develop that and like take it more seriously Once I did that and I was like, all right I'm gonna focus on the craft of writing now and really like trying to get to the core of the truth I think that's what writing and storytelling is you're trying to express truth In that process when I started taking it seriously like something clicked in my mind And that's when I knew I had my voice. It was like that I would I would write and I was like I know This is going to work Does everybody have a voice? No That's like asking if everybody's a born writer. That's no some people just can't write like it's just Yeah, they'll have their inner voice, but when it comes to their writing voice, I think it's it's a talent thing Yeah, yeah, maybe they should be musicians. I think during our last interview. I said I want I wish I could be a musician But I can't I was a born writer can't my brother's got the musical talent, right? Where do you see the men's movement going if you can foresee the future I Mean I do see it getting bigger Because we do have the truth behind us. So I mean self-improvement and the stuff we're doing is Making people's lives change for the better because it's showing them the truth, right men see this They literally change their lives around because they're exposed to the truth for the same time that They're exposed for the first time So I do see it getting bigger and bigger because you can't stop the truth I see more hit pieces on the manus for your coming, which I don't mind. I think they're a good thing But I also see people who are gonna wake up and see the positive side of it And once that happens once we get more and more positive press Which I think is inevitable because there's gonna be too many men's lives who has changed too many voices speaking out there that it's gonna catch on more and more and I don't think it'll ever become completely mainstream, but people are gonna know about it, yeah, and People more and more people are gonna be drawn. So right now. It's in this little movement Little counterculture movement and eventually the counterculture if they have the truth behind them They become part of the popular culture. So that's what I see happening. I have no idea what the timeline is gonna be but I do you can't stop the truth. Yeah Let me ask you this AJ said something he said if you are the best in the world at something Why aren't you getting paid for it? And that just Kind of made me stop for a second. Yeah, it seems like you have learned the art of Getting paid for something you're good at. How did you get to that point where people gave you money for your talent? I did not start doing this for any I didn't even know you could make money writing online How'd I get to that point It just it kind of happened naturally. I had other friends Tell me I should start charging for services and it was actually friends telling me you need to start doing this Like you're just putting way too much stuff out there for free Start doing this, but I wasn't competent yet my skills. They hadn't been built yet When I started doing the coaching and guys were paying a lot of money to get my nightlife coaching and just You know how to approach women and handle women in New York City Paying me a lot of money for that that did give me a big confidence boost. I was like wow I should start charging ridiculous. I wasn't doing it before. Yeah And it's it was a slow evolution a very very slow evolution for me, but I did not Think I was gonna make it done doing this. This is why it's it's still bizarre to me very bizarre to look back and see how many people have been affected. Yeah, and Yeah, willing to get me their hard-earned money. Yeah for my information my teaching my writing So it's it's strange to think about Try not to think about it too much. I think about the blow I'm thinking about the song New York New York by Frank Sinatra where he says if I can make it there I can make it anywhere. Is that true about New York? Oh, yeah different than Johnstown, Pennsylvania Man, yeah, you gotta be blood thirsty you literally like what AJ said in New York You have to be the best at what you do. Yeah, because you will get a place like that. Yeah, like there's You literally have to strive The thing I love about New York City is that people there are super ambitious I and I like that in a person when a person wants to be the best not necessarily an elitist or pretentious But they just want to be the best. I value that in the characteristic and New York City has that going for it Yeah, what do you hate about New York?   I'd lived in for 10 years Wow.  Yeah and so it's like that across the board and I really I think that's the worst thing right now because it's worse there than a lot of other places Yeah, is there an alternative to New York or what is the next? The next stop for Goldman. I'm on the road right now. I'm looking for it So we'll see what I find but right now. I'm enjoying the traveling and the exploring I can do and I'm grateful to have that opportunity Yeah, yeah, and you'll talk about that with us the next time we sit down and talk, right? We will excellent. All right Goldman unleashed. Thank you. Thanks again, George What was your experience so far with the 21 convention? Oh I was dead. I was dead professional all across the board really good energy with a lot of people and I just like this is very positive positive direction this George this is a this has been a first-class event. It's fantastic. You guys were in a really tight ship I've been in a lot of conventions over the course of my business career And I can tell when things are well run and when things aren't and this is a very well run operation I was very impressed. It's pretty incredible to see where Anthony's brought it Especially from last year which is my first year here and to see the upgrades he's made. It's been incredible I've got my notebook and with every speaker. I've written down about two or three lines Under each of the speakers of just just the key prime stuff that that's good That's great. It's very surreal man. Yeah, I'm really enjoying it I'm happy to live in such an era where it's such a thing like this is possible I have never seen a group of guys like this a group of 200 men were focused Squared away and working on their values just never met a bigger group of wonderful guys It's kind of neat because I've been to a fair amount of conventions in my day But you never see one where the guys like here You can see Ed Latimore talking to Tanner about boxing.  Yeah, it has been fantastic And it's been four days of guys all on the same page working the same direction fascinating meaning some of the people Hearing their stories. Yeah, you got people traveling from other parts of the world come here just to see yeah some of the speakers That's amazing. I think it's impressive. Everybody here is very serious. Yeah, they're taking it close to their heart What a great convention. Thanks George


Hundreds of people have been given an alternative take on politics at the Scottish Parliament's 3D Festival of Debate, Film, Music and Exhibitions, which turned the famous Garden Lobby into a summer cafe bar. The Festival of Politics featured talks by figures including Vince Cable and Chris Mullin, former party special advisers and academics and policy makers, on issues ranging from American politics and polling to the refugee crisis. Well as you can hear, it's not the normal background hub-up here in the Scottish Parliament, this is a chance for everyone to engage in politics in Scotland. This year's festival partnered with the Festival of Architecture and included events focused on built space and technology. A big Lego exhibition saw a giant map of Scotland created over the three days and a family day featured Lego and other activities for children. This building is not the property of MSPs, it belongs to the people of Scotland and this is an opportunity for people to engage with thinkers, policy makers, politicians about the big issues of the day, a chance to engage in politics, which is what the Parliament is all about.


Cool. Hey everyone, can you hear me okay if I speak at this volume? Yes, good so My name is Gabe Hollenby A little about myself before I spend the rest of the time telling you what you really want to hear I'm a principal engineer at a place called Pivotal Labs We are a company that helps all kinds of other companies build custom software of all sorts front-end stuff back-end stuff Mobile apps the whole shebang I've been programming for a long time 15 years professionally a little bit more than that in total I'm also a hobbyist game developer. I've built one game so far available for iOS and Android It's really fun, but I'm not even gonna plug it here But if you're interested in knowing about my gaming talking about it later, I love teaching my favorite part about my job Is actually not even the coding. It's all of the time I get to spend with the junior engineers helping them get better at what they do I've discovered that's what I feel most rewarded by so far in my career, and I hope to do more of that You know for the rest of my life That's also why I'm up here tonight, right? I think that like Lysha said we have a serious problem We need more women in the tech community and instead of just lamenting that fact I wanted to be part of the solution So I said what better way than to vote with my own personal time and give time back to you guys So I hope that You guys are all interested in this program and you'll stick with it and apply after doing all of the workshops I love sushi. It's my favorite food and my favorite movie is the princess bride So they know a little about me so We're gonna be working with months rescue. That's the NGO that I've been assigned with They're not here to talk to you tonight. It's actually testament to the fact that they're super busy, right? They're actually out there Rescuing dogs right now. So I'm here to talk to you for them. I spent some time I got to know the two ladies who lead it and I'll do my best to represent What they're all about right now So what Mets much rescue does is basically they are adjusting the problem of there's too many stray dogs out there And the government doesn't do a lot of help To take care of the problem and the problem is obviously not like just get rid of the stray dogs I mean if you're if you're humane and you're interested in taking care of dogs So what they want to do is educate the community to have more compassion for strays To employ ethical and humane practices and to establish healthy population control Through a few different techniques, right? One of them is to do it like trap neuter and release But besides that the other thing they can do is They do feeding and boarding, right? So this is I'm gonna show you this is like the least happy slide I have but some some dogs that they can't catch they'll actually go and put out food for them And dogs that they have caught that they haven't found homes for yet They have to board so they can find homes, right? Obviously all of this stuff requires money and it takes a lot of logistics and so They I talked with them for a while and found out that the thing they need the most help with is actually adoption a Little bit just in general message from them Here's some pictures of puppies I feel like I have an unfair advantage because I have cute animals in my slide deck and these are all animals from Mets rescue So these two cute little puppies were both rescued. These are both successful adoptions So there's the the puppy picture and they're sort of young adult pictures They both had a very infectious disease called Parva virus Which is something that most puppies get vaccinated for when they're born, but obviously strays don't get vaccinated So not only do they just like try and catch like healthy dogs But they also do everything they can to catch the strays and if they need help nurse them back to health So here's two happy success stories Here's another look at his before and after he's just the cutest little thing. I love I love this I could just look at puppy pictures all day and if you end up working with me We can actually go and like play with the puppies too, right? So anyway In general, how can you help if you're interested in this not just by Becoming part of tech ladies and helping me build something for them But in general all of you whether or not you even join Well, you should adopt right if any of you are looking if you'd like to get a puppy definitely talk to much rescue They're great super coincidence. I didn't even know this after I got hooked up with Mets rescue I noticed that a friend of mine had liked them on Facebook and I put two and two together and realized my friend had Adopted a puppy about a week before she got her puppy through Mets rescue So small world small country, but they're obviously doing good work. I thought that was a cool coincidence Okay, so if you can't adopt you should foster right so if you don't want to adopt full-time Maybe you can take care of a puppy in your house and tell somebody else can find it's forever home You can't foster you can donate right taking care of all these trays costs a lot of money And they're always accepting donations through Facebook if you can't donate you could at least share their posts on Facebook with your friends So you can find them on Facebook at Mets rescue. So just look for that So onto the interesting stuff. How are tech ladies going to help? So their biggest problem is organizing and managing the process of adoption right now They're doing everything through Facebook Facebook wasn't designed to handle like dog adoption process so the biggest problem that happens is You go on Facebook and Mets rescue and you see all these cute puppies and you're like, oh my god, that one's really cute I'm looking for a puppy. I'm falling in love immediately And maybe you don't even realize that puppies already found up been found at home because you have to like read all The comments and say oh this puppy is no longer available So we want to make it easier for the the Mets rescue Organizers to keep track of all the puppies that are available What state they're in if it's someone's with foster care if a dog's looking for a home and boarding whatever and Make it easy for the public to come in and see all that information and get hooked up Quickly and easily so I have a couple. Sorry As a phone, okay, I have a few mock-ups to share you real quickly. They're just mock-ups There's no promise that this is exactly what we're gonna build but they're based on a conversation I had with them and they said that the slides looked good. So we'll make a homepage with just some general info Those X's by the way are placeholders for images. You probably figured it out But if you're not looking used to looking at tech mock-ups, that's kind of what they represent The more interesting stuff, right? We'll have a listing with all the dogs like their nickname some vital stats about them a picture and maybe some descriptions Then you could click on them and you could see more details most importantly lots of cute puppy pictures More stats more description and then also I think importantly Information about you know, what it takes to be a responsible dog adopter, right? What are your responsibilities going to be making sure that this is really something for you? On the back end, they need a way to manage all this content So we'll have some sort of like index dashboard where they can see all the dog They're currently trying to find homes for maybe sort by different attributes Keep track of which dogs are in foster care. Like I said or which dogs are certain boarding facilities And then obviously they need to be able to create and manage the details of each dog So we have some kind of form for that, you know, where they can enter all the text. They can upload photos They can change the status of them, etc. So that's the rough outline. We'll probably build something similar to that. I Promise if you work with tech ladies, these things three things are going to be true at times. You will be tired Learning to program is hard work. Okay, but you will learn a lot And most importantly, I promise that we'll have fun. So if you're excited I know it's not fair. So If this sounds cool to you, I hope you stick around do all the workshops make it to the end Let's do this. Thank you


Welcome everyone who's here. I'm Kathy Zeiler. I'm on the faculty at Boston University School of Law, and I'm a board member, a current board member of AMOS, which is the organization that is co-sponsoring this webinar with the Center for Open Science. AMOS stands for Association of Interdisciplinary Meta Research and Open Science, and I wanted to make a quick plug before Chris gets started for the annual conference. This year it's going to be held in Brisbane, so if you're looking to have a nice vacation in Australia, it's going to be on November 21st through 23rd. Please see our website for details and the registration should be up soon. I went last year. It was absolutely fabulous. It's two days of really great talks. But we're going to attend to the Meta Science Conference now, and I'm very pleased to be moderating this panel, this webinar, as we lead up to the in-person conference in Washington, D.C. next month. And I'm very happy to introduce Chris Koch. He is going to talk to us today about a metastatic approach to examining the structure of implicit memories. And I'm going to stop showing my screen so that Chris can share his and then take it away. Chris. All right. So I should also say, sorry, Chris, before I hand it over. For those of you who have questions or you want to post comments, et cetera, please use the Q&A button, not the chat button, but the Q&A button. And raise your hand, and you can either speak your question. I can elevate you to speak your question to Chris. Or you can type it into the Q&A, and I can read it to Chris. Whatever you prefer, raise your hand if you want to jump in on the screen. That's fine. And then you can have a dialogue with Chris if that's going to be easier. Either way, you can do that throughout. So Chris invites questions and comments as he is speaking. So I'm going to monitor those and make sure they get in. Okay. Take it away, Chris. All right. Well, I'm going to talk to you about some metascience related things that we somewhat uncovered as we're working with a larger task. Looking at implicit association tests. So I just want to do a few things before we really get started. These are some of the students who have been working on different aspects of the overall project and have contributed to today's talk in a variety of ways. So metascience, I probably don't have to really go over this very much for this particular conference, right? But the goals are really to help identify basic elements of scientific inquiry, right? To figure out how these elements might interact. We're looking at how we're going to determine scientific knowledge, how it's produced. But the last one is really where I bolded a few items because that's what we're going to probably focus most on today is really improving the quality of the research that's being done and also reducing inefficiencies. And if we look at the Venn diagram over here, we're really looking at mostly at data reuse and maybe a little bit on statistical critique. But those are probably the areas that will hit most throughout this. So just a little bit of background. This is going maybe a little bit more than what we need for metascience, but it gives you a reason why we're actually looking at any of those to begin with. When we look at memory, memory is generally broken up into explicit memory, things that we're consciously aware of, and then implicit memory or the things that we are not consciously aware of. Semantic and episodic memory. So memory for facts and events and events are typically semantic and episodic. And then you have a variety of different types of implicit memory. And the one that probably is most tied to the implicit association test is that of priming. And you can see priming it. We have changes in perception and belief caused by previous experience. And so that's, that's what we'll probably focusing most on just within implicit memory. But there's also how to knowledge and perceptual learning and other types of associative learning. Now the implicit association test has five blocks of trials, and they're all structured basically the same way. You start off with some concept words. So that's in the first block and, and you have keys usually on two different sides of the keyboard. So I was just for to demonstrate for the talk today, I'll just mention a and L. And so you may have a concept word related to individuals weight. Right. So you may have, you may have like muscular or you may have heavy or something like that. So these are concept words around weight. And then if it's, if it's a positive trait, let's say we'll press a, if it's a negative trait, we'll press L. Okay, so that's your concept, those concept words and then you have a value of words and the value of words maybe like, you know, just basically just like good, bad, right, lazy, right, whatever, whatever words. So that with a particular I concept you're looking at. So we may have words that are related to positive aspects of individuals and then also maybe some negative things, and negative things probably might be related to the stereotype that potentially out there. So they're going to press a and L for those words as well. And then they have a third block in which they get both. So now you press a, if it's normal weight and good, you press L if it's heavy or bad. So you have the combination of those underlined, underlined congruent here because those are kind of congruent with the stereotype that we might have like a person who's very heavy might be lazy. So those two things will go together. And then, and then what happens is you switch the words so you switch the keys that they press. So now when it's instead of lazy and they're hitting L they press a key. So you switch the coding of the concept words, and then you have them do it again.  And so those then would be the incongruent pairs that you would have, and you're really interested in the difference between these two conditions between the congruent condition and the incongruent condition. If something matches our belief or implicit belief system or bias, we're going to respond faster than if it doesn't. So that's, that's really what you're looking at with this IT. And IT has been around for quite a while, which is part of the reason why we looked at it. It's been researched fairly well for about 25 years. So here are some typical words in IT just to give you an idea what the evaluative words might be like. So you have words like happy peace joy right on the good side and terrible war agony could be on the bad side. I bolded war just because the analysis I'm going to show you in a moment has to do with word count. And this is just matching word war or just matching the word in this case war. So it doesn't look at variance. So it doesn't include like worship or plain war lord war torn pre war post war right all those are in a in a corpus but they're not the words that were matched. So here's, here's what you find with this and this is, I took the labels off because it was a little bit harder to read but you're looking at frequency on the y and response time on the x. And so it happens, as your frequency goes up, your response time is slower responding to the words as a frequency goes down and takes you longer to respond. This is kind of typical of a frequency that we find with explicit memory. The other thing is this is barring response times from another study so the one nice thing in terms of like that. This conference is that there are some larger data sets that are out there that allow you to get some of this information without doing the study yourself in a sense so you can really get a good idea of what's going on with. In this case the words that are being used, and the good words are responding to actually faster than the bad words, just by responding to the words so this is not including them in the it yet. This is just if the words are on their own, you respond fast to the good words and the bad words. Right, and you respond fast to the words that are more frequent than less frequent. This is looking at the good versus bad words are used actually in the race 18 to see a similar pattern. So the good words are respond to faster than bad and you have a frequency of bacteria as well, where are the more frequent with the word the less the slower the response, I mean the faster the response time. So, when I just kind of like take those kind of try to account for those word frequencies from existing information. You can actually see a decrease in the t value that you get for the study for the comparing the congruent incongruent condition, and there's a decrease in effect size and that's probably not too much to triple a thing because one of the criticisms of the it is that the effect size are not always very large to start with. And so, when you account for the word frequency it seems to even decrease some more.  And so that's an important one important reason for looking at the frequency and the it. The other thing is this, and this is part of the reason why we started looking at this, and we don't have to worry too much about the rest of this diagram. This is part right here the semantic system. So when we talk about frequency effect I mentioned that it's usually associated with explicit memory. So if we go back to that slide, right explicit memory we're dealing with semantic memory and episodic memory and we're oftentimes probably even more focusing on semantic memory, which is general knowledge. So, what's different here though is we're using an implicit task, and those implicit memory tasks are, as kind of suggested right here right these are indirect measures of something which is why the it is actually a really good task, because we're just looking at a race it that you say, you know, you can ask somebody or you racist and they're probably not going to say well yeah I am thank you for asking a minute. They're going to cover that up right so we have self presentation skills that we engage in when we know that someone's asking this question that would be inappropriate for us to respond in a particular way. And we want to present ourselves in a more positive light than maybe we actually think. The implicit association test it doesn't allow you to do that because these are immediate responses that you have their actor, they're operating below your level of awareness right below consciousness. And so you don't have the ability to control in the same way as you would if you're filling out a questionnaire. So, the thing that be interesting and part of the reason why we started taking this on is that, if there's a frequency effect with these implicit tasks. It could very well mean that we're asked we're, we're taking memory, right and we're we're accessing memory the same with the same type of mechanism it's just a level of awareness. It's a process that changes between explicit and implicit, and a reason here's some different ways you can you can kind of conceptualize that right so explicit memory is going to be more of here a conscious mind, you know, interfacing here with thoughts and feelings. And then you have insights intuitions things like that these are kind of more pre conscious. And this way, it's kind of hasn't to show this because this is usually attributed to like trying to describe Freud's theory but but you know I feel like an iceberg above the water this is all conscious mind right below is this pre conscious but then you have this big part of the iceberg is really much lower and that's that's the unconscious mind. And what we're really talking about is, is that if we kind of think of this as memory. We're accessing memory, but some of some of the time we're aware of it and some of the times, we're not. If we're aware of it's explicit or not. It's implicit, but we're getting the information the same way if we show these similar effects across implicit and explicit types of memory tasks. And that's a way to kind of look at why this might be an important thing to examine, because this has those different types of memory that I've been talking about, but also associated with the different parts of the brain. And so usually when we're talking about things and we think well semantic memory is really more the medial temporal lobe, whereas priming is going to be more like neocortex. And I was thinking that okay since they're operating in different parts of the brain they're actually different things. Right. And so what kind of suggesting with this frequency effect. Within within an implicit memory task is that suggesting that we're actually have a memory store that we're possibly accessing either consciously or unconsciously. That's that's a big reason for the motive behind the study itself. Now if we it's relatively easy to do to look at the frequency effect, and that all we all we have to do is kind of recode the data so that you know when we know what word is presented or what images we can see if it's a higher low frequency, we can actually put in the frequencies from like a corpus or something like that, and coberry them or we could categorize them as high low medium frequencies and then we can analyze to see if there's a difference between those. In order to do that we actually have to have trial by trial data and so that's one of the things we'll get to but there are some other reasons for picking the it for this and that is, like I said it's been around for a while. And there's also the project implicit website through Harvard, and they have a lot of people going there and doing these tasks. And so they literally have millions of people that have done various types of it tasks, and that data is largely available. And you can look at things over time so I'm going to give you an example of looking at something over time and one of the, one of the things that you know we've kind of started exploring just by looking at these differences over time. Now, what they have found Charles was in the nausea found is that there are certain biases that seem to have decreased over time. And there's certain biases that seem to have increased so the one example that we've had on the slides as a race it. That seems to be decreasing. The other one I mentioned the body weight that one seems to be increasing along with age and disability. So there are changes that can happen with the implicit associations that we think people are making. And this is an example of one of those things when trying to figure out why. Okay, so we can look at these over time and again so this is kind of you can see visually what these tasks are like so here's your concept. And the word in this case it's an image so you have to say whether this this individuals are European American African American, then you have your evaluative words have to say whether this word is good or bad, and then you have both together. Right. And so that's your combination. This is from a different site I mentioned implicit project implicit site. This is the OPL stands for online psychology lab. And this is done through a PA or the American Psychological Association. And this is looking at data actual response time data from individuals from 2006 on. And you can see that the incongruent times are slower than the congruent times, pretty much across all that time. Right. And that's what you would expect based on what I've been saying about those associations are making this though is interesting right because you have this big bump in the response times right and, and the difference between the congruent incongruent trials, and not only is it a big bump or the times are much slower and the differences increased, but it's also opposite of all the other times. So, seem to warrant some further investigation. Right. And, and just to point out right so this is primarily when you look at OPL site you're looking primarily at college students, some community college students and sometimes are some high school students, but they're probably junior or senior so that's kind of the age range you're looking at, maybe to 22 something like that. And, and we took samples we looked at samples from primarily introductory courses and from various regions around the country so we're both the country up and took different samples from different regions and we use both public and private institutions. And when you do that you see something like this here are the times. Or here are the differences that they have and you can see this is actually starting in 2010. So in 2010 11 and 12 they're up higher like they were before, and then you can see that they drop. Okay, this is actually looking at each individual school as a sample, and you can see the difference across the different different samples or classes across time. And it really is those 1011 12 times that are giving you a slightly different level of it than more recent times. So the question then is if this really is something that's out there why. So we start looking at a few different things and this is a great way to look at some other tools are available that look at large amounts of data to try to help us figure out what's going on in our in our studies. This, this top graph is really just from, it's from psych info, actually. And so looked at the number of publications, regarding it per year. And you can see that there was a steady increase I said, it's 25 years so the original article is 1998. Looking at it, and you can see that steadily increased to about 2013 where peak is kind of dropped off. This is Google searches. So using Google trends looking at Google searches for both either it or implicit association tests. They, they, they're right on top of each other so they're pretty similar. This peaks the timeline so much the same this peaks in 2008. So it seems like they're the public interest in it kind of about five years before it did for researchers. So, that's, in a sense, interesting but part of the reason we looked at this was, well, was there an increase in interest in the it around 101112 and did that promote some type of difference in how people responded. And the answer that is probably not because the peak increase is much is earlier than the change we saw in that data. But this is looking at just people just looking at race. And you can see this is 2010 starting here. And there's no difference in people how people search for race before or after that time period where there's that change. If this bottom line is, you know, maybe they're looking at other things or besides just race. This is a critical race theory and that's where you see it's just flat. There's no differences there. Is it due to some high profile event that happened that you had a lot of media coverage. And we started looking at some of the earliest, you know, really big media events around. In this case, violence. And those all occurred after right so those didn't seem to be good. Hate crimes actually seem to decrease during the time period where this increase in the it happens so that doesn't seem to be a great explanation. So when you look through databases that had the tops news stories for every year. Nothing in particular really stood out. Looked at economic factors in 2009. The wealth gap had increased to its largest amount at that time. That directly preceded when there was this bump so is that possibility for why we saw that bump and possibly. And when you look at additional data, there was also an increase around 2021 and there wasn't an additional bump in that it curve we saw. So that doesn't seem like a great explanation. So let me start looking at other things right could it be a could it be a political figure right someone who's popular in this case. President Obama was elected and was elected in 2008 inaugurate 2009. The change came in 1011 and 12 so these are shortly after he assumed his presidency. Now the interesting thing in terms of this being an explanation is that we didn't see the bump start during the campaign, and we didn't see it continue throughout his presidency. So if this is a possibility only lasted for a relatively short amount of time. Right. So what's that cause for that spike in implicit bias. That's an interesting question. It doesn't seem to be a fluke in the data because there's the same task. And we had a consistent response from students from around the country. There was a period of time, the various reasons that I just talked about they all are possibilities but they don't seem particularly satisfying. And it may be that we're not looking for a a factor but it's, and it makes sense if it was a combination of factors actually that all kind of came together at a particular time that changed the associations that people were making. And if that's the case, though it does suggest that there's some economic political social sociocultural factors that can have some type of temporary effect, at least on the associations with implicit bias that people have. Now that if you can figure out exactly what that is. Well now you have some ability to possibly influence negative biases that people may have. Now that may have some additional ethical concerns associated with that, but that's those are those are some possible implications. So the it again is great because there's, it's been around for a while you have a site that collects a large amount of data. And then, you know, we can see there's actually multiple sites have a large amount of data on it, and we can look at trends and can, you know, download the data and we can examine it. The part is all great. But what we needed to do for ours is not just look at the data that they had, but we needed to look at the, the trial by trial data mentioned that before. The trial by trial data tells us how they respond to the positive word or the negative word, or the, the word that, you know, like if you if you're looking at depression, for instance, which is an it. Sad is used much more frequently than something like melancholy. Right. And so, if you're if you're looking at those we need to know how do they respond to each word. Every time those words are presented. And that's what you don't get when you go to these data sites, because they have the basically the clean data right the data that is in that final analysis. There are some summaries for each person, but you don't have their individual responses across all their trials. And for an it that can be, you know, we're just giving an idea, or talking about an action I'm kind of jumping ahead of slide but this part I just want to say you know there's there's different things that we decide about data and one is like what data do we include. This is just from a study looking at inequalities and education. And you can see they're starting off with, you know, just over 485,000 people that seem to fit the general requirements. But as they start looking at more and more of their requirements related to the study, they end up with just over 24,000 people that that they can look at. Right so and then, depending if you're doing other research you may say okay I'm only going to include people that respond at least 75% correct right there if they're making more errors that they're not going to be included in the data analysis. We can look at people that have an ordinarily long response times and say okay anybody who's responding more longer than this, which is really beyond the what's typical, then what they're not going to select them. There might be people who you know that might skip a question or two and there are techniques to fill in missing data but at some point there's too much data to fill in right so we have to say if they if they don't complete a certain amount, then we're not going to include them. Right, so there are ways that we have their criteria that we set as cut off values say this is data we're going to keep this is data we're not going to keep. And then we do that analysis will come up with a data file, and that data file that we do the analysis on is the one that we usually post in these data repositories. And that's the kind of data that was available for the it. There's the other thing about related to this right is the data maintenance part which is, it's not just the data that we collect right it's also the data that we make available and the data that we keep. So for instance, I can tell you that the OPL site when they did the it, they, they automatically calculate the congruent and incongruent times, and that's the only thing they say. So you can't go back and get the trial by trial data for that site because they made a choice in the beginning what would be saved and what would not with a project implicit site we can actually go back and get original data. So you're looking at a huge amount of data. So, I was looking at the race it just for 2022 there over 750,000 people who've taken that. That's a lot of participants. And then when you look at their individual files individual files are probably going to be about 120 trials. So in each trial you're going to record. You can you can vary how much you record but you're at least going to need like what condition are they in. What was what was presented what word or what picture was presented, how long it took them to respond what response they made, and whether or not the response was correct. So you're looking at probably at least five pieces of information for each trial for every participant. So when you start adding it up that's a lot of data, right, and try going through individual files for like 750,000 people I mean that's, that's just a lot. So this is actually from an article he was talking about data storage and not research itself, but is a copy because it was, it was seemed relevant here. So some things that we do we want to have it be cost efficient that's one very practical thing. But then, how well can we access the information. Right. And when you start having so much data, it's almost becomes overwhelming as to what to do. And so that's an issue and then also how much space do you need what's what's the storage requirements, and how easy is it for you to distribute these data files. So these are all all important considerations. And you notice what as he was talking about data storage, his algorithm is basically you pick two. Right. So we may get something that's really cost efficient, and it's really flexible and easy for people to access information. But it means that it might not. It might be reduced information. Right. And so we're making, if we pick to the other one might be compromised. And, and so we I think we kind of see that a little bit here, but also it also makes sense, right. These are just practical issues we have to deal with when we're dealing with especially large amounts of data. So here's here's before I go to the next slide. So what I've really focused on mostly here is is the words. Okay. So the words we can look at we can get a corpus we can look at we can actually go to Google we have an n gram viewer we can get word frequencies are pretty easy to get, and we can control for frequency. Right. But there's another aspect to some of the it is and that is it's not just the word that's presented. It's also a picture that's presented. And so that's what we can we're looking at here. Now this is great again because project implicit they also have an associated OSF site. And on that site you can download the data from 2000. I think it's 2000 might be earlier in 2010, but you can get data for every year. And you can also get these these are their picture files that they showed. Now, one of the things that is important when you're looking at faces in particular there are certain dimensions that are that help us recognize who people are. And there's certain features that are kind of less important. What I did is I took we took these these pictures right these 12 pictures and and borrowed some questions that was used in another study. And so we took these pictures individually put them up on the screen fall to the design of another study where they showed a picture, the picture went away, and then they asked a couple they asked a series of questions. So here are the questions right how blank was the previous face. And so on a scale one to seven or you have dominant threatening right trustworthy, all those things what we're really interested in this particular case.  So here's one set of questions, here's the other set of questions. You know how likely you sit next to the person on the boss or sharing or things like that. So these are kind of more face, more behaviors related to the individual, but this is this is kind of what their initial perceptions are about that particular face. Now there are other things that we could ask.  You know, we can ask questions like that those are all important for face perception as well. Symmetry is important. And that seems to be tied to attractiveness and attracting this can influence responses depending from what we're asking. So there are a lot of things that we can ask about a face to see how good a face is to use in a particular study and bring it up because we're going to come back to that in just a moment. But in this case, looked at these, all these faces based on those questions and this is looking at them specifically for how stereotypically white or black that people perceive them. And this was done using put the face and put the start the questionnaire into we posted through m Turk and had a couple hundred responses.  All of these women are more stereotypically black than white. And all these men are rated as more stereotypically white than black. I move these to the side because these faces were not rated that way. So there's no difference between how stereotypically black or white they saw these faces. And what's significant potentially significant about this is that they're all the male faces, all the black male faces. This is not something that's kind of spread out across all the conditions of the face. This is one can one set of faces that seems to be responded to differently than the other three sets of faces, which may introduce some systematic bias into the responses. It's important when we're using different different types of stimuli that we understand exactly what we're showing and what we're asking people to respond to. Now, in order to do that we kind of need like a stimulus repository where we've gotten much better at data repositories. But what about repositories for the stimuli themselves. We do have some good places to go to, but they're not really stimulus related. So there's, there's this one there's psych test so if you want to find a test, and I didn't add this bold it was right there right instantly find and download instruments for research and we're teaching right so you can go here and you can get a test. And if you're interested in some type of personality dimension, you know, other some other type of individual difference that you can look at, maybe a particular condition. There are a variety of things you can get a measure for and evaluate where does each individual that have in my sample fall. There are other sites like this, you have like the positive psychology site the University of Pennsylvania, they have a variety of measures that are used for like authentic happiness and mindfulness and things like that. And not only do you have the names of these but you have the description of them, you have links to them if they're available. You have the original sources that for the references and all the psychometric information so they're great places to go to to get a lot of information about instruments that you can use. There's an IP IP site international personality item pool, you can go there and there's a lot of information about different sets of questions that you can use for different types of things that you want to look at really to personality, and it has again all the reliability that it has links to original articles. Again, another good place to go to but they're not like faces or voice files or object files or things like that that we might use another types of research. And so that kind of gets us to this idea of, well, do we need to start developing stimulus repositories in a more strategic way. So, here are some of the desirable characteristics of data repositories this is from NIH. And so you can see a variety of those. And most of these are really applicable to a stimulus repository as well. So, I won't go through all of them right but you know, persistent identifier so being able to create a DOI for these, having them being available for a period of time, but then also this metadata. And not just if we go back to the phase example, not just the face here's a set of phases that I use in the study, but all the information that goes along with that face so, you know, before we did it we tested it and here are all those, like those mathematical measures that I said. Here are some ratings that people had about how attractive they thought the person was, or how, you know, stereotypically whatever they thought the person was. Right so you can have all that information about every face that's included in this in this stimulus set. And then as a researcher I can go there and say, when I want to do a face perception study, someone has basically done all the background work on those faces for me. And that's, it's I can reference it. Alright, so the people contributing to it have reference and people that are using it have the reference as well. But that metadata is going to be really important to know how well to know what what's appropriate use for those. And then also, like the curation and quality assurance. So those are going to be some of the big ones obviously some of these others are going to be important as well. But this is, these are some practical things to take into account when we're looking at maybe creating a stimulus set. Now, these sets can go here. So dimension in a more systematic way. And so what might be great to do is to actually have specialists in these areas so you may have specialists associated with face perception stick with that one. And they can get together and say that these are all the important dimensions that we need to evaluate for a face to know if it's good to use in a study. And then those become the dimensions that you collect data on those become the things that you fill in for your metadata with associate with each picture. Right. And so now you have a set of experts are saying this is what needs to be done to have a really good quality stimulus and then and all that information is available for all the pictures. And it could be faces. I mentioned voice files. If you're doing like, oh, you can do like emotional processing of voices or something like that. Well, what makes a good voice for a particular emotion. So those are kinds of things you can use if you're doing object perception research when you're looking at objects and you're seeing what people what people can how they identify them. Right. So, you know, just you can look at a number of different areas of psychology, which was where I'm focusing. And then and then build out these data set stimulus sets based on the recommendations of experts in the area. So you basically have like recreating a set based on best practices. This is what needs to be included or considered when you're using whatever type of stimulus. And I think this is kind of I've heard this. I've heard this people say this. And it's kind of interesting to talk about like in about this at a conference. Right.  Right. But we're all together, right, dealing with meta science. Okay. Now, so what happens like in psychology, what's happened in psychology is that these large conferences that used to everyone we get together. They've even become specialized. And so what will happen is, you know, like I'll go to a conference and it's going to be mostly like perception people. And the social psychologists who might be interested in prejudices, prejudicial and stereotypical behavior. They're not going to be at that conference anymore. Right. So we're missing the interaction between different disciplines that might have been or different subgroups within a discipline that might have been helpful in in developing, you know, really well and, you know, tightly done experiments, because we don't interact with each other anymore. And this kind of tears that down because it provides a space where all that can take place and we can benefit from the expertise of people in other areas, as it pertains to the studies that we want to do. And that's kind of what we're thinking as we're talking about. Not data repositories, but stimulus repositories to try to help improve research. And so here are some of the things just kind of getting down to a summary here is that the IT is great in that there are some things that it does really well. And I didn't mention this earlier, but you can go on a GitHub, you can download an IT example, where in theory all you have to do is change your stimulus files, and then everyone can be running the exact same study. So if we can all agree on this is the best methodology, you can download something like that and we're all doing the same thing, which is obviously going to help turn the visibility as we include all these different samples that we're working with. So the data and the stimuli are all available in OSF. Right, so that's great so we can go there we can get all that information. So some things are really good about the IT. But when we start looking at specific things like does frequency of the different stimuli matter, or does how stereotypical face use or an image used in one of the other ITs. We can't get that information because that information is not saved. So we may have to look at something like do we need to include trial by trial. And the one that I the one that I mentioned with over 750,000. So what we may need to do in that case is say, look, you just have too many. That's too much data. So if you have a situation like that, maybe take a random, a stratified random sample of all your participants and you, and we make these repositories, you know this type of data available in repositories, like up to this many participants. So we're going with maybe 300, 300 is a lot less than 750,000 but it's a lot easier to deal with something 300 or less than something that large with the IT. So, so there is some need for additional information that is collected that we don't all get. And so sometimes these things sort of make doing other follow up research a lot easier. And that kind of thing about it. If we're able to show a frequency effect with someone else's data who didn't design the study to look at that. Right then, neither of us are potentially biasing our study at all right because they're they originally didn't collect the data to look at it, and we didn't collect the original data right so the some of that experiment or bias is going to be eliminated. So we have a developing standardized stimulus sets that are a variety of sets that are available for researchers to use to really make sure that they don't have any potential confounds in their study, and that they can be assured that they're using the best possible stimuli for the studies themselves. So, those are kind of the recommendations we had going through this process. I mentioned some of these already in terms of what a stimulus data set should include, and what days data repositories to potentially include as well. And I believe, yes it is that got me that's that. That is everything I wanted to say about this based on the project we were engaged in, and it does leave us some time so if you do have questions I'd be certainly happy to try to answer those as best I can. So if you have a question if you want to jump into the queue please raise your hand and I'll see it pop up I can click on a button that will allow you to speak at your question, or you can just write it into the, the q amp a. Um, so we do have one question in the q amp a from Judy, Evara Sue, and Judy says, does it, does it mean that associations and implicit memory can change due to extrinsic factor social events political events etc apart from inherent individual differences. I think it's possible. Yeah, I mean I think it's possible especially like if you, if you think back to the one slide I had with priming and it had based on experience, all those social things that are happening are actually part of your experience as well so it's very possible that it can. Now to the extent that it changes it over a period of time. I can you can't tell from the data that I showed you how long the impact is going to be. It looks like it might be temporary. But I would suspect depending upon the type of experience that will be longer lasting. Great. I have a, I have a question. So, I completely agree with your recommendations I think they're, it sounds like they're, they're really valuable. You know the more data we have, for example the better we can, we can attempt to replicate work and minimize differences between the original study and the replication attempt and the other positive benefits that you suggested are great. One question that I've been thinking a lot about is kind of institutionally how do we create incentives for people to post these sorts of that information on, you know, OSF or their data repository to think about those pieces as data kind of how do we, how do we get this going other than coming to conferences and presenting the ideas and hoping that they'll the people listening will catch on and actually do it. As a good, it's a good question and part of it. Maybe, I mean the journals are probably going to, they can play a part right. So some journals now won't even review unless you have posted your data somewhere right. So that that's a possibility. The other thing is, you know, if you spend some of these things creating a stimulus set that's really good can actually be quite time consuming, but it hasn't really been considered scholarship. It's gets you to the ability to perform the scholarship that you want to engage in. And so maybe reevaluating or, you know, somehow giving different different level of credit to developing high quality data stimulus sets would be a way to encourage it as well. So if you can actually create something posted and that becomes a valuable reference for you for your research other people's research. You know, because these these. I don't think I mentioned it specifically but you know I mentioned about using experts as a way to evaluate like what things should be included, but you can have experts evaluate what's in what's actually uploaded. And so if you include like a peer review process to that, then it becomes something that might be deemed much more, you know, source of scholarship and would encourage people to do as well. So I think there was probably those two things would have the would kickstart at the most. Yeah, that's great that the Open Science Foundation actually had in their guidelines has one of the recommendations to formalize citation practices around data set so that people actually have an incentive to create data sets that other people are using and they get cited for them, which, you know, is our kind of currency of credit. Do you do has a follow up question. How do we select concept words for studying personality concepts when they're extrinsic influencers as well how do we partition between inherent influencers and influence due to extrinsic factors. So, so the concept words. Yeah. I think I had a part of that one slide that said it's, I think these, these words are really tied to stereotypes. And so if you're looking at, I mean the concept words, right for sure. If you're looking at weight, you're going to look at normal weight and overweight because that's that's where the bias normally is with people that are heavier. And then what are the things that people think about naturally when they think about people that are overweight. I don't know if that's a great. There's almost like a circular definition in a sense is, you know, what are people normally associated with heavy people and then then we put an implicit memory task and then we, and then we get it, we get a different way. Right. But you're looking at things that are tied to really more of a stereotype of what they think. Now it doesn't always have to be about a stereotype as well. Sometimes the stereotypes are not as clear. There are certain things that people just like more than other things. So there's one of the first it was with flowers and insects. And so you had a different set yet a set of flowers and then you said okay this is a flower this is not right. This is an insect and you just you just categorize that way. And then you have your value towards and people rate the flowers as being not, you know, better than the insects that people generally don't like insects the bad words get associated with the insects. Good words go associate with the flowers. And so in that case, you have like here.  And so you want to include those if you start using names for flowers and no one knows what they are than the name and I even know it's a flower. And then for insects so it kind of depends on the concept some of these are, like I said are kind of stereotypical types of words or words associated with a stereotype and others are kind of like I'm looking at flowers and looking at insects. I'm going to pick common flowers and common insects. So it kind of depends on the concept I hope that answers that. Great we've got another question from Emma Mills. She says, did you say at the beginning that you may touch on statistical analysis critique so could you talk a little bit more about that. Were you referring to researcher degrees of freedom and analysis choices here. Yeah, you know, I was, I was thinking if you go specifically for like the example that I use about work frequency. That is something we can statistically control for that was not done in the original. We can either go in and say, oh, they use this word. Here's the corpus I'm using here's word frequency, and then we can covariate, or we can categorize things by like low high medium frequency words, and look at it that way. But that was really kind of going back and saying okay if we if we want to take a task like the it and look at a different variable that they didn't include but we can, we can reasonably include based on the data based on what should be in the data files. Then we can add that variable in, or we can recode something and then we can redo the analysis accounting for that other factor that we think might be important. So that that's really what I was I was thinking about in terms of, you know, something was missing. And the data sets allows to go back and add that in. After the fact, then we can actually make the we can make the correction off the original data, instead of having to do the whole thing over again. Another question from degrees of freedom yeah there's a lot of choices researchers get to make and, and when you're doing something that's really large and get a tremendous sample size, you know you're getting these really nice p values but sometimes not very high effect sizes. Yeah. Good we've got another question from our boundary, a lot to be this person asks stimulus there's it's a comment actually, and we want your thoughts on this stimulus availability over time may lead to familiarity which could potentially influence the responses to them. Yeah, that's true. And so, you know, you may need to exclude people who have participated in a similar state before that cuts down on your potential to collect data so maybe that's not optimal all the time. That would probably be something to look at and, and also with the other part though is, is that, you know, these, ideally, these would be like dynamic stimulus sets that you're developing and so, you know, you may have a group of images for instance that you're looking at but then people add to them over time, and then you have a pool that you can pull from instead of just looking at a set but but yeah you probably you may want to consider asking a question upfront about whether or not they've seen any of those items before or something and then they'd be excluded. Yeah, so there's there's a good part in the bad part right influences the research in a negative way that's obviously not good but if it also increases our ability to use better stimuli and generalized findings better than that's a that's a good part so there might be a little bit of a trade off and an appointment which you know we have to control for that. So just in our last couple minutes here wrapping up and anyone else can jump in. The meta research community has been looking at it pretty intensely and I think one thing that people seem to be talking about is whether it's possible to separate implicit bias from explicit bias you know how do we know where that these implicit association tests actually measure implicit as opposed to explicit bias. Any thoughts on not more general question. Yeah, that does. That is one of the things that people talk about like what exactly is it measuring. I think I mentioned kind of in the beginning, ideally. People are making associations that they're just making these quickly near courage to go as fast as you can right but you also don't want to make. You don't also don't want to make mistakes mistakes are based on how they, how the pairings are expected to be. So if you don't respond with whatever with a good trait with with a race it mean it's kind of set up as bias so that you have a positive bias toward the white faces and so if you said the other thing then it's considered wrong. Yeah, the. It is harder. I really do think it's harder to consciously control what's going on with an it just because of the speed at which you're responding. So is it is it really tapping more implicit associations it probably is. You could, in theory, I think probably slowed, slowed down just enough that you're now consciously. You're consciously controlling your responses. That's possible and but not slowing down so much that your response times look really off. So, and then if you go back to what I was talking about with memory I think. You know it kind of goes along the same path is what I'm what I'm saying really is that you have you have your memory. And you're tapping into that memory. One you're doing it you're doing it without a lot of awareness and when you're doing it with more awareness. And so it's the same if we're dealing with the same source of memory. And if we're dealing with the same amount of awareness, we may be able to move on that's on that continuum of awareness, but we're still accessing the same thing. So, yeah, I mean what what exactly is the measuring can you consciously control it. I mean, it's possible. I want to squeeze in one more of the people that want to get in there. There is great capacity for planned missing data design here so that stimuli are not overexposed. What do you think about that missing data design. Yeah, so like exactly what what are you referring to for the missing data design part. I'm just going to click on a lot of talk so we can hear your voice if you want to jump in. You have to unmute if you want to be heard. Can you hear me now. Yes, we can yeah. Thank you, Chris for a fantastic talk. And so I'm having the same kind of thoughts with cycle linguistic research across like 29 variables and differing effects in the literature, and a standardized stimulus set, like the question would possibly become overexposed over time. So a way to mitigate that might be to have them in different kinds of bins if you like, and when people take them they take them in random selections. And over time, we piece together the data so that not everybody sees every stimulus but we're collecting enough where we can do missing data imputation. And so it also works with if you're using very expensive and stimuli, you give that to a core set of your participants, and then you give the cheaper stuff. So it's something that I'm thinking about. So that when you get your experts together and they do that huge standardized yeah what do we really need. You can use it. And because everybody I'm guessing everybody would want to use it to begin with, but you could kind of ration it. And there is a kind of theory around it or a method around it that I've heard about. What do you think. No, I think that sounds like a good idea. And, you know, like I said, if it's dynamic that the more it grows the easier it is to do I think what you're talking about, and prevent the overexposure that's kind of come up. But yeah, I think that that'd be that'd be a good way to do it.  I mean it's possible. So, instead of having, I mean, you wanted to be freely accessible, but you also may want to say well how many of these are going to use and which, you know, so we may have some level of awareness of how many times in particular. Well, you actually can do it by downloads. Right. So even if you do it randomly, you can know how many of the individual stimuli have been downloaded. And then you can make sure that those are not if they're frequently coming up that you spread them out. I mean, I don't know if I'm saying that particularly well, but, but yeah, I mean, I think I think the more you have the easier to do stuff like that and that'd be that'd be great. That'd be really good for different areas of science. So I'm glad you mentioned something totally different than what I mentioned. I think I think the curation of it becomes that scholarship idea that you were talking about as well. Yeah. Yeah, I think so. Really interesting talk. Thank you. Thanks. Alright, so thank you everyone for attending. We hope to see you at other webinars. Check the meta science 2023 website for additional webinars coming up. Thank you so much Chris. Really interesting. Take care everyone.