Hey, I'm Kevin, the CEO of Xtrades. In the last 10 years, we've seen an explosion in the popularity of digital finance. However, despite all this growth, market participants are still facing many challenges. The online world lacks an immersive experience that incentivizes financial veterans to contribute their ideas of beliefs about the markets.  These are problems I ran into myself as a professional trader who spent the last 10 years interacting with other traders online. I wanted a place I could go to to learn, share trading ideas, and receive valuable feedback from traders that I trusted. I wanted to build and monetize my own trading brand. To my surprise, a trading platform like this simply did not exist. That's why we created Xtrades, a platform made by traders dedicated to the trading experience. We're bringing retail investors together in a way that's never been done before. It's an immersive social experience that's tech driven and transparent. We're using technologies like machine learning and AI to bring crowdsourced insights to work for traders, providing analytics and services that give retail an edge. On Xtrades, winning influencers are able to monetize themselves through their statistics and their activity. Let's dive into how Xtrades solves problems that retail runs into. We use a strict price validation system that only executes trading signals within a slight threshold of market price. So when users get an alert, the alert is captured in real time and actually matches the market price. Xtrades works with Discord to gather data and collect insights so traders can efficiently gain analytics about themselves, what they trade, and the markets. We use math and technology to create custom formulas that allow traders to gain a more gamified experience during their journey. All of this is made easy to access via our web and mobile applications. Features like these are what allow users to collaborate in a stronger way and receive analytics that they're looking for. Xtrades started as a small newsletter service and has grown to become the leading platform for trader collaboration. Since 2019, we've grossed over $6.5 million and paid out $2.4 million to our traders. We grew from a user base of just one user to over 240,000 users during this time. To set ourselves apart from being just another Discord trading chat, we created our own fully Discord integrated mobile application that gathers insights from Discord to create analytics and visuals on the web and mobile realms. We're listening to every one of our 16,000 monthly active users to create products based off of their trading preferences and principles. We're using gamification and education to give new traders a go-to platform to learn from and to seek out profitable opportunities. Xtrades is committed to making sure our shareholders receive the best opportunity for growth in their investment. By using data, research, and business acumen, we're making successful decisions that are going to drive the company forward. Be a part of Xtrades. Be a part of the future of trading.


One of the big mistakes writers make is drafting a message before considering who is actually going to read it. Spending time now with getting to know your audience will increase the chances that your document gets the attention it deserves. Once you know something about your readers, figure out why you're addressing them. Most professional documents are written to motivate action. You write a resume and cover letter to encourage someone to hire you. You write a web page so that customers will flock to your products. In short, what do you want your reader to do? That desired outcome must be your main point and everything else should be organized around it.


Felly, ddod oeddwn ni'n gweithio i Professor Ann Oakley, yng Nghymru yn ei wneud yn ymddi'r gweithio. A'r fath o'r cyfnod o'r fath o'r cyfnod o'r gweithio, yn ymddangos i'r fath o'r gweithio, yn ymddangos i'r gweithio i'r gweithio i'r gweithio i'r gweithio, ac oedd y gweithio i'r gweithio i'r gweithio i'r gweithio i'r gweithio. Felly y bydd ar y fath o'r gweithio i'r gweithio mi srae. Mae gydag ei thysgliad yr egocirio eich trwy peth yn yma yn ei fath o'r cyfnod o'r gweithio i'r gweithio? Nid efallai sy'n golygu i'r gennw gweld, hwnna'n meddwl gweld i gydag gennw gweld, mae hynny'n gweithio'r welwydd, ac hiwn i'n gweld i'r gweithio i leisio diwethaf.  Yna, rydyn ni wedi gweld i'n gweld i'n gweld o'r cyfnod o'r cyfnod o'r cyfnod. I was actually doing a study of housework and I was genuinely puzzled as to why it was women worldwide who did the housework. So that sent me to the anthropological and medical and psychological literature to look for the answer to this question and what I discovered with this whole complex system of gender differentiation in which for the most part women are treated unfairly but not always and that's what I found. So that's why I wrote that book and that I know that book has saved in print and it is used by A level students. I get maybe one email a week from a student saying could you just explain to me this and often they want me to do the work for them and I tell them to go and read the book and that doesn't go down very well. So it's the gender system and then obviously my work on housework I think has helped to elevate the status of housework both as an academic subject and as an area of labour and that is, I know that is also quoted quite a lot and some of that work has been read by quite wide areas of the public. It hasn't actually been just sort of academic theory and then the third area is methodology. So I think the way in which my work has used different approaches and has been at times quite controversial for doing that. I would like to think that I have made some contribution to the evolution of social science methodology. Most recently, well I say most recently but actually since the mid 1990s, setting up a resource for doing systematic reviews of social research and that continues today and I'm very proud of my role in helping that off the ground. I'm just going to follow up a little bit. Why do you think it was so controversial that you were using systematic reviews for social research? What do you think was controversial in that for people? I didn't think it was controversial but parts of the academic community didn't respond particularly well. Particularly I have to say in education, there were professors of education who got quite cross. This work I didn't do on my own, this was very much a team effort but when we came along and said about an area of education research we read these studies, we can't tell who you interviewed, who they were, when you did it, what questions you asked them. It was more a question of taking them to task for not reporting their research in such a way that you could work out whether or not it answered the question that you were trying to answer. We were, I think, regarded as being unnecessarily panickety about that but it isn't a question of unnecessary panickety, whatever the word it is. It's a really important criterion about the doing and reporting and disseminating of research results. I was surprised, I was surprised not so much by the controversial aspect but by the hostility, hostility. I remember going to conferences and presenting these results and just being shouted down. It didn't happen, of course, well not of course, but it didn't happen when we were talking to healthcare audiences because the notion that you need to look at an area of research properly and you need to be sure that you have evaluated something in such a way that you can be confident that you're saying this works or this doesn't work. That is much more accepted in healthcare settings. It is not, was not and still is not accepted in the same way in settings outside that. Do you think that those responses are reflective of the discipline at the time, the people who are publishing in those areas and developing methods in those areas? I don't know because some of that response continues today. I know, I am not in that world in the same way that I was but I know from talking to colleagues as it does. You know, you don't have very much control over how people read your work and people read other people's work in such a way that makes sense to them at the time. I do it. We all do it. So I can see that some of what I was saying about methodology didn't fit with what people wanted to hear. And they didn't actually take the trouble to understand it. The degree of misunderstanding will come on to this later but some of my applications particularly to what was then the Social Science Research Council, the peer reviewers comments about methodology reveals a pooling level of ignorance about these different methods. I really was a pool to other people who were part of that effort to get that work funded. That's really helpful. Thank you. We'll come on to that question I think straight away which is what parts or what funding applications were unsuccessful. Were those applications where you got such ignorant feedback and comments or such lack of understanding of what it was that you were doing? Yes. Well every social researcher has a whole finding cabinet full of unfunded research. When I look back at having many years I was doing this, I am amazed by the amount of time, the amount of labour that goes into this. It is a very inefficient way of conducting a research enterprise nationally I think and it also speaks to something which has seemed throughout my career which is the lack of a proper career structure for research. Social research is still the pool and we are not even called academics. There are academics and then others and researchers are others. You can get a permanent contract to be a member of the teaching staff but not to be a member of the research staff and that means that researchers are constantly chasing the next contract. That has all sorts of implications for the way in which research is done. One of the implications which again we may come on to later is that the attention to research materials when a project is over, there is no attention. Stuff gets shoved in a cupboard and it never gets sorted, it never gets looked after. So the public money that goes into research is not reflected at the end of the day in the public resource that other researchers can access. So unfunded, I think in the end I tried to think of a project that I couldn't get funded and I actually wasn't able to think of anything that I didn't eventually manage to get funded. But that might mean several or eight different applications and I used to reckon that one application, even if it was quite a small amount of money, would take you a month. So that is a lot of time spent trying to chase funds for something. I think I would have liked to have done more comparative research looking at other countries, looking at the international context and some of the issues that I've looked at. I would have wanted to do that. What do you think that would have added to your research? Well, it would have been fascinating to have, for example, done the housework study in early 1970s in other countries. Some of the work later on, looking at, I mean we did a project on evaluating the value of childcare, formal childcare, and of course Scandinavian countries have a wealth of experience of that. So to have done the same study, this is, I hesitate to say this, but this is in the sense the medical model is when you're trying out an intervention, you look at it in lots of different places. And I think social science really hasn't got its head around that. And that would have been in terms of producing a social science which is relevant to social policy. And that's what you need to do. And you need to be an international context. That's really helpful. And just finally coming on, so at the beginning of the first question was looking at the themes that have been taken forward and have been really impactful. Are there any themes in your work that have not been taken forward that perhaps have surprised you that they've not been taken up? Well, I think some of the work has been taken up in some circles and not in others. The books I wrote about the history of methodology, experiments and knowing, I know is very widely used by healthcare researchers. I was like it to be used by people who are learning how to do sociology more than it is. And then I think there is a piece of work that I did in relation to the study of women having their first babies. I published two books about that. One was called Coming a Mother and it was re-printed under the title From Here to Maternity and reviewed by some people twice. They thought it was a different book which shows you how well they read it in the first day. But the second book was called Women Confined Towards the Sociology of Childbirth and what it put forward was a theoretical model for explaining post-natal depression. And what I argued on the basis of these interviews which I had conducted with women doing their pregnancies and afterwards, what I argued was that you didn't need to have any kind of theory about women's psychology to understand that some women having a first baby got quite miserable afterwards. They had gone through a series of traumatic human life events, occupational change, complete change in bodily shape and bodily identity. Taking on a new job with no training at all and surgery often as well, institutionalisation in hospitals. All of this was sufficient to explain why you weren't necessarily overcome by the joys of mothers. That has not been taken up and I don't know why. I think it's a loss because I continue to think that that is a valid, I think it's a useful way of looking at it. I think it would be more helpful in some cases for some of the women who still have this kind of difficulty. It links back, I think, what's really interesting about that is what's seen as controversial. I do wonder if topics like that, which are part of the lived experience of at least 50% of the population, have actually not been taken seriously by broader disciplines, by funders, by those who fund social research. That's a theme that goes all the way along. I should have mentioned earlier in relation to the first question that I think the motherhood work, the early work, drew attention to the impact of medicalisation, which is what it was then called. What that was part of was a move mostly outside academia, but some within, and there were other sociologists who, around that same time, were particularly in America, Barbara Capsarossa and Nancy Stollersaw, people like that, who were saying the same thing. These helped to change maternity care policy, not just here but everywhere. It helped to alert people to the fact that quite a lot of what was done to women who were having babies was not based on good evidence and was actually harmful. I think it empowered, it's not the word I like very much, but you sometimes need to use it, it empowered the user organisations to bring about change. I think that's really interesting in the context of research that I'm currently doing around young fatherhood. We're working with professionals and practitioners and we're talking about the benefits of father inclusive practice, not just for men but for women and for children as well. I just wondered if you could reflect a little bit about why might we need to ask questions about men and their welfare and do you think that contributes to that broader conversation and discussion around challenges that women experience as well and questions of women's welfare? Well, gender is not just about women, although it is mostly women who go on about it, if I think it's a problem. So gender is about men and if one is looking at the impact of cultures of gender, ideology and practices on human beings, you have to look at the impact on men. Fatherhood is a good example, it's an area where I think men suffer from being excluded to some degree from experiences of love and care, which are deeply meaningful human experiences, so which women get probably rather too much of. But that is an area where men would definitely benefit. But this whole thing about gender being about women is such a problem. So if you look at social policy issues today, one of the biggest problems is the culture of masculinity. I've done some work on this with a colleague called Cynthia Coban, we wrote some piece in the Guardian and in various other places, looking at the culture of masculinity and masculinity as why masculinity is not a social policy issue. So you look at crime, over 90% of violent crime is committed by men. One of my social science heroes, heroines, is a woman called Barbara Wharton, who did a lot of work on the treatment of crime, finology, criminal justice. She made this remark along the lines of, if men behave like women, the courts would be idle and the prisons empty. That remains true. So if you want to save, I think Cynthia and I did a calculation that if men, this was about 10 years ago, if men behave like women in terms of crime, 42 billion pounds a year would be saved. Right, you could spend that usefully. So why do people, this is not about, I mean our piece in the Guardian got a lot of responses along the lines of men hating, men dating and all that. It's always women who are the problem. So the Ministry of Justice publishes an annual report on women and the criminal justice system, which is to fulfil the requirements of some equality act. Don't do a report on men and the criminal justice system, because if they did it would be policy dynamite, because what you might show is, it was like I said, that a major area of social policy that needs attention is the antisocial effect of a culture of masculinity and its impact not just financial, but obviously psychological and social. I don't think that was the answer you were expecting. Well no, it raises some questions for me I think, which is great. For me I suppose I'm trying to look at fatherhood and people's everyday lives. I guess my question then is what's our role as sociologists in developing those explanations and what can we do I think to contribute to those broader discussions in a way that benefits men as well as women I think in society. I don't think I can answer that. I think that's for you, younger generation. I pass that one back to you. It's difficult, it is difficult, it's difficult to change. Same thing with me, we managed to get an interview with John Snow. He was very nice, we met for coffees on where near ITN, and he was very engaged with the whole topic, but he said probably it's not newsworthy. It's not newsworthy. 42 billion isn't newsworthy. Yeah, it's not newsworthy. Interesting. But then that comes back to why certain themes of your work have endured and others may not have around not only in terms of who's interests they may serve, which touches on your question that you know who's writing at what time and from what set sorts of purposes and for whom, but also then what is seen as provocative and why is motherhead, why is this example of women saying actually it wasn't good for me in hospital when I was having children and I was being infantilised in particular ways, which wasn't great. Why that push back against an orthodoxy, like a medical orthodoxy becomes very provocative in those sorts of situations? Well, the answer, one answer to that is obviously if you start saying that motherhood isn't necessarily a matter of joy and roses and all of that, you are criticising, you are taking apart a really important cultural ideology that keeps the whole kind of gender family system in place. So you're corroding that and that is threatening to the establishment. And in terms of medicine, in the 70s, I hesitated to compare that with the situation today, but doctors were very patronising. They really did. It wasn't just women that treated in this infantilised way. Patience didn't have rights, patients didn't have experiences that you needed to listen to and that has changed, but it was in the process of transition. One thing I'm interested in is the extent to which we individualise families still. That lets the state off the hook in lots of ways, providing access to resource and welfare. I think we expect families to carry a huge part of the load of the whole range of social problems and that gets talked about at discourse level as well I think. I certainly see my role in trying to develop those explanations really around. That's problematic. And within families it's still women who carry the main burden of sorting all of this out, not exclusively, but still a lot. We've mentioned which were the most impactful at the time. Why do you think that your work continues to have such significant impact? Because it does. It continues to have importance and resonance for sociologists, social scientists more generally. Clearly young people as well as you were saying.  And yet we're talking, for me, I think there's a couple of things here which is around this idea, or well 1975, this is the students that are writing to you. That's very much in the childhoods of their own parents, probably, likely. So for me that raises questions around enduring social trends, enduring social processes. And that also then connects to why some of your work does continue to have, and which ones do you feel are the most important, that continue to have resonance for people? I think the question about impact is better answered by other people, but I will give you two answers. One of them is that I think a great deal of my work has been about everyday life. And that is something that people find it relatively easy to identify with. And in relation to that, my orientation, I hate to say this, but I don't actually hate to say it, it's the truth, has not been predominantly towards the academic world. I have not been trying to become a sociological theorist and I wasn't trying to become a professor at an early age. Those were not my ambitions. And therefore I think that I probably have a reputation as quite a practical kind of sociologist, somebody who doesn't write in a theoretical, jargony kind of way. It's accessible and I've always tried to write. That is racist, a fact that I never wanted to be a sociologist and firstly I wanted to be a writer. So writing and the translation of research into some kind of text that makes sense to people outside that world has always been very important. But also those themes in your as well. So whilst we've seen tremendous social change since 1975, those themes still endure, they're absolutely at the heart, as you say, of human experience. Motherhood or parenthood or housework, home, family, the most profound of human connections really. And mostly immediate everyday lives, they're fundamental to what we do. The becoming a mother project was, we did do very much help, led by other people in this unit. A follow-up study contacting some of the women who were originally interviewing them. And then we did, we repeated the methodology with another sample and now there are a couple of a team of researchers who are going to try and get money, probably spending months and months writing on funded research to get money to do another such study. So that tells you that those themes and that kind of approach, it still has resonance today. So when you were talking about gender earlier and you were talking about men, you touched slightly on masculinity and I wondered if you wanted to say a little bit more about that. Yes, what I want to say about that is that the, as I said before, the gender system is about men and women and really the current culture of masculinity is, I think, very difficult for men. I think it's health damaging. It helps to explain one of the enduring mysteries in all of this, which is why on the whole men have higher mortality and more literacy than women. That may very well be something to do with the expectations of male behaviour and emotional life or rather the lack of it and all that kind of, all those related areas. So if you're looking at inequalities in health there has been so much more work done on structures, class structures, on gender and inequalities in health and like that. I've done some work with a public health expert called Alex Scott Samuel about the fact that patriarchy is damaging to men's health. This is a really important point. If one is looking as a social scientist at this whole picture, this is part of the picture. We aren't just talking about poor women and the oppression of women. We are talking about the oppression of human beings by a divisive and unhelpful way of thinking about human life. So your work has had a huge amount of social and intellectual impacts. What does an emancipatory sociology look like in your view? You have to start by looking at what sociology is at the moment. I don't only think there is good evidence that it remains a discipline which looks at the world from the position of dominant social groups. Not just men but white people and other dominant minorities. So in order for sociology to perform an emancipatory function, emancipatory is leading people out of slavery. It has to reimagine itself. It has to reimagine its aims and its methods and it has to recover its history. So a lot of the work that I've done in the last ten years I've been looking at the history of social science and I have discovered that most of what we're talking about now has been talked about before. Not just in the 70s but in the 1880s, in the 1890s and the 18th century. We've been there before but sociology has a very kind of amnesia about its own past. That is such a shame because it's a bit like unfunded research all that they have gone to waste. All these people have done all this work before. For example, if you look at the evolution of sociological research methods we shouted a lot in the 70s and 80s about the discovery of qualitative interviewing. But they were doing it. They were doing it in the 1890s and the early 1900s. There were many social researchers, mainly women but not exclusively. They were doing covert ethnography, qualitative interviewing, diaries, formal surveys, the whole lot. They didn't necessarily, they didn't call it that but that's what they were doing. They said we had the data from here and the data from here and we looked all together. So why do we constantly need to reinvent the past? That's not that's not emancipation for anyone. I think there's a lot of work still to be done and I think universities obviously got a lot to answer for. When I started out in all of this something like 80% of academic staff were men that's not the case now but do we actually have a culture which doesn't reflect dominant views about not just about masculinity but dominant views about important social issues. Housework is not an important social issue. The way men behave is not an important social issue. How babies are brought up is not an important social issue. So why not? I mean it has to I was when I was awake in the middle of the night as one is as you get older. I'm thinking what we really need is we need a degree course which actually takes all these texts from the past. So it teaches, it doesn't teach you know vapour, dirt time whatever. We look at not Max Vaber but Marianne Vaber who was a very important sociologist in her own right. We look at what she did. We look at the methods texts from the point of view of people like Margaret Harkness who was a cousin of Beatrice Webb who did this amazing work studying Sweaty Vaber and also wrote novels. This is another thing. It's one of the things that I have done that women social researchers and male researchers in the past did a lot. It wasn't because it was extraordinary to write the themes that you were interested in in fiction as well as in factual forward. And now that's not regarded as a sensible thing to do. But why not? If you want to encourage debate if you want to encourage people to think about something and to do something about it then why not? And that leads us to our final question which is one of the themes of this interview or this conversation has been legacy what do you feel how do you feel that people may take your work forward that sort of legacy into the future of the work that you've done? Absolutely no. I hope that the emphasis on the kind of sociology that I've done which is attention to the living of everyday lives and people's experiences on the one hand and as I said before the notion of social science as social science that those two things are the things that get remembered and taken forward but I would really like to know what happens and it's really frustrating because I won't know. Thank you very much indeed thank you for your generosity in your time it's been a real learning experience and for me too it's made me think about the past thank you.


Hi, welcome to the MIX, the teen center at the San Francisco Public Library. My name is Ileana and I am your humble host for our STEM Challenge Yourself series where our remarkable librarians come up with science experiments that you can try at home. In today's challenge you're going to learn a little bit about static electricity and the powers of observation with everyday items. So I think you're ready for the challenge. Take it away. Hello friends, I'm Mr. Joseph. You might remember me from previous STEM challenges or during class visits. I'm super ecstatic to guide you on how to make electricity. Alright, so what do we need for this experiment? We need our precious library card, two balloons and make sure they have strings on them and paper that's been shredded into tiny pieces, foam balls, tiny foam balls, make sure you put them in a clear plastic cup, also foam balls in a ziplock bag. And finally the most important part is a fabric that's made out of wool. If you don't have a fabric that's made out of wool, you can also use your hair. So now that we have our materials, let's get started. Let's generate electricity by rubbing this library card onto the wool fabric. And you want to make sure you rub it about 30 times, on 345679, I think that's 30. What do you think? Okay, before I place my library card on our first testing material, the paper, what do you think will happen? So let's find out, oh right, so we have a magnetic effect. The static electricity on the library card attracts the papers to cling on and stick on the card. That's so cool. Okay, let's try our next material, which are foam balls in a cup, again so we need to rub it about 30 times.  Look at all those foam balls attracted to the library card, it's a magnetic effect.  So we're going to hover and slide the library card and you can see them moving around and jumping around inside the bag. How cool is that? So what is happening and why, how does electricity form on our library card when we rub it on the wool? All around us, my hair, this paper, this foam balls, the balloons, this fabric, this library card made up tiny particles called atoms. And atom is a building block, all matters in the universe. And we have an illustration here of an atom and circling and whizzing around the center of an atom are even tiny particles called electrons. Electrons are extremely electrically supercharged and sometimes they move to another atom. And that's exactly what's happening is the electrons from the fabric, from our hair, escape and move to our library card. Thus, our library card is now electrically charged with electrons. So let's play, let's do another experiment with electrons. And this time we're going to use different materials. We're going to use a balloon and make sure your balloons have a string on them.   And I'm just going to place this balloon right here. We're going to test this balloon and where I will use this balloon instead of a library card. And also, I have an extra foam balls here that are different colors. This one is a blue color and this one is red. Okay, so we are ready to test these materials. So let's transfer some electrons onto this balloon by rubbing. Okay, did you see that? Sometimes you can hear static electricity. And let's see what happens. Whoa! Actually, we grabbed everything here. So as you can see, the papers are attracted to the balloon and the foam balls as well. So we just get that magnetic effect. Whoa! And here's the coolest thing, okay? Watch carefully, watch my finger and you can see they're trying to get away from my finger. I can move those balls with just hovering my finger. Let me get this out, this paper, okay? So again, we get this magnetic effect. The objects are attracted to the static electricity we developed onto our library card. Okay, and I'm going to move this paper, we're done with the foam balls. Now we have these different foam balls, okay?  Okay, and I'm going to hold it up so you can see. This time they're trying to get away from the balloon. They're moving away from the balloon and that's called repel, okay? And so we're going to do another experiment with another balloon. So we have, let me put this one right here. So let me generate some electrons on this balloon. I'm going to tie it onto the table here so it doesn't escape. It's trying to follow me. I think I've generated enough electricity on myself that they're trying to cling on to me. Okay, and let's see what happens. First, they're trying to attach to myself but they're also repelling from each other. How close that?  And so, so far we've witnessed two electrical force. We've witnessed the electrical force called a track where two objects come together, the papers are attracted to the balloon and repel where the blue foam balls somehow repel from the balloon. Cool is that?   You can explore all materials around your home and come up with the same electrical force. But not only it's fun, it also plays an important role in our society. Thanks to this awesome Asian American named Peter Tsai, we'll supercharge our N95 mask so that we can better protect it from tiny particles including viruses. That's awesome invention. Have a happy experiment and hope to see you again. Bye and back to Eliana. Wow, Joseph, that was electrifying. Thanks so much for sharing how you can experiment with household items to learn more about static electricity. And I appreciate you noting Asian American inventor Peter Tsai with his role in COVID-19. So keep experimenting. Be sure to check out sfpl.org slash STEM challenge for resource lists and stay STEM-tastic.


Anyone else out there a huge fan of La Cruce? It is incredible French cookware that I have personally had in my house for the last, I don't know, 20 years since I was a kid, because my mom loved it too. So they got stainless steel pans, non-stick pans. They got much more, but they are most well known for these enameled cast iron pans. And the plethora of incredible colors they come into, this is a new color called Chambre. All La Cruce owners know about the amazing colors this company provides. And usually if you find them in someone's kitchen, they're just played nicely somewhere, right? They're not tucked away in a cabinet because they are so beautiful. If you're looking for somewhere to start with La Cruce, all their cookware is obviously amazing, but their Dutch ovens are particularly desirable. And with winter approaching, trust me, you're gonna want one of these. I could not live without it. Any kind of braising I love to do in these Dutch ovens, low and slow in the oven, or on the stove top, that's up to you. So this company's been in business since 1925, almost 100 years, which is insane. These pans have been a joy in my life for many, many years, and I'm hoping the same for you. They also make amazing gifts for people like us who love cooking, so keep that in mind.


The study found that adding certain biocides to atolocollogen matrices did not significantly affect the viability and proliferation of keratinocyte cells. This suggests that these materials could be used as an effective alternative to traditional antimicrobial biomaterials. This article was authored by Jorge Lopez Garcia, Marion Lihake, Petrohumpelasek, and others.


What's up everybody? Jason here for jazzy's case breaks calm 2016 optic football hangar one box break number 13 Just sold out this unlocks panini one football random team and teams breaks hard Where we're giving away eight spots in this break. This is the last one So this is a 32 spot break. Remember no veteran common ship We're gonna do the break first randomized customer names and teams to shoot with the hits out Then we'll randomize all 32 customer names again in a separate dice for the randomizer and Customers in the top eight out of 32 we'll get spots in that break itself and that'll be coming up next Here's a dice roller It's a three and a five for eight times. Let's do the break one two three four five six seven And what was it? Mark done a Raymond eight times one two three four five six seven eight Chicago Bears out of the Carolina Panthers All right again guys this 2016 so I think Titans of course Derek Henry is a big one Of course, this is the Eagles draft with whence you have golf with the Rams Dak Zeke with the Eagle or the Cowboys are I Think Michael Thomas will be Saints Tyree kill of course as well with Cincinnati, Kansas City So I get no trace though, but just stick to what you got but quickly show you guys All right guys, let's quickly do the break Thankford we got Derek Watt Teddy Bridgewater Hollow Derek Watt purple Chris Jones the Quantred well Josh Doxon massive Jalen Ramsey is a rookie there in Devontae Booker Now let's switch scenes and give away those last spots Good luck everybody top eight get in let's roll And it is a one and a two three times. Good luck one two Three Boom one or two three times top eight get in Roy C Eric Roy Daniel Steve Dayton Joseph and Mark There you go, man Three times boom top eight get in coming up next guys is the break itself. Appreciate it


Hi, everyone. It's good to see you and thank you for attending the quantitative analysis and survey research workshop that will be conducted this afternoon by Kevin Fomalant. I'm Sue Boffman from ARL and really pleased to welcome everyone here this afternoon. It's good to see you all, especially after our summer hiatus from our workshops. Today's workshop is part of our series of training opportunities for the Research Library Impact Framework Initiative, and those of you who have attended some of our earlier sessions have heard me say this. Our overall goal for this series is to help all of us develop or improve our skills in conducting research and assessment in our libraries. We have a few remaining workshops in this series that will address reporting our research. When we share the materials from this workshop with everyone, we will include registration information for those workshops as well. We are recording this session and we will share the recording and slides and other documentation that Kevin has prepared with everyone. You're welcome to share this information and these materials with colleagues who could not attend today, and we do have a few people on our list that we knew couldn't attend but wanted the materials. So Kevin, welcome. Thank you again and let me turn the podium over to you. Okay. Well thank you, Sue, for the introduction. And today's session, today's workshop is about quantitative analysis and survey research. But before I get started, I did want to ask that if anyone has any questions to feel free to break in verbally, I will be monitoring the chat, but it's a bit easier if you just go ahead and interrupt me and I'll answer them as they come. And so just to get oriented to where we are in the process of survey development, we've had earlier workshops on designing a survey and then also doing visualization and tableau after the data collection period. So now we're into the preliminary quantitative analysis part of our survey research, which includes exploratory data analysis and hypothesis testing. And so survey research is a little bit different from some other methodologies that I've worked with in that your hypotheses actually change and probably should change after you finished your data collection and you've done your visualizations and you see some initial patterns in the data. So of course, when you're doing your survey design, you're thinking about what questions you want to answer, what themes you're interested in learning about from your respondents and their opinions about those themes. And you design the questions to get at those themes. But then you get your results and you may find out that your respondents actually think about those themes in different ways than you anticipated. And it's totally okay and encouraged to adjust your data analysis methods in response to that information and to include that in your statistical work. So in survey research, we're not only testing respondents' answers, but also testing respondents' interpretation of our survey itself and using that information to redesign and reevaluate our hypotheses after data collection has ended. And so I'll take you through the process that I go through after data collection, which I think is one of the most important parts of survey research since that's really the guide that sets up the statistical analyses that you will do at the end of your research. So it's kind of the neck that turns the head of your research project. So today I'll take you through several aspects of quantitative analysis. The first most important part is data cleaning. And data cleaning is essential because it helps preserve the reliability and the validity of the survey research project that you've just completed, the data collection that you've just completed. So you need to be eliminating responses that are extraneous and that are actually detrimental to learning about your respondents' opinions on your work. There are several methods that I'll go into with that part of quantitative analysis. And then a more flexible, so we go through pretty strict methods for data cleaning and then we move into more flexible methods in exploratory data analysis that help you revisit initial questions and assumptions by looking at response rates, actual answers to your questions and to cross tabs, which I think are actually an undervalued part of survey research. And I'll explain to you what a cross tab is and give you a visual, just to see what it looks like. Although you may have seen it if you ever looked at a polling memo around the election, they tend to publish those. I'll go into item analysis as a part of composite creation. Some people pronounce it composite, depending on which person you're listening to. And so in item analysis, what you're really doing is figuring out which items are most important and which items fit into which themes. And it's those themes that are we use to create a composite. Scoring methods, which are just quantitative methods to calculate what your respondents' scores are, how they're responding to your questions. And if you just approach it simply, it might just be a mean score, but there are actually several different methods to calculate end scores, depending on what you are emphasizing over other methods. And I'll take you and do some statistics work in hypothesis testing comparisons of means and then Pearson correlations, which I think are a key statistical test to use both to evaluate your results on your own, but also as a way to share with stakeholders who might be interested in your work. It's a particularly intuitive statistic that I find helps not only to just report results, but to drive change whatever program it is that you're asking about in your survey, whatever project that you want to evaluate to actually implement for the next year changes in whatever organizational structure or do any other different methods that you want to use to use the survey results that you get from your project and some conclusions about best practices in quantitative analysis. So data cleaning is the first stop on our tour through quantitative analysis. So data cleaning, I don't actually have this on here, but it does come up sometimes depending on what sort of web design you're using. So if you're using a web survey, maybe it's not so likely that you actually get someone from outside your population responding to the survey. It's common for people to email their surveys to specific email addresses. So you have a pretty good idea that those are the only people who are responding to your survey. At the same time, it's possible that someone else could get access to the survey and answer it even though they're outside of the population of your survey. So say you have a survey that goes out on a listserv and you really just want people who are on that listserv to answer that survey. But you find that when you get your survey results back that some of the email addresses that are listed in the respondents answers actually aren't on that listserv. So it could be your decision that because they're not a part of your population that you want to study to remove those responses. At the same time, this sort of removal does require typically for you to collect some sort of identifiable information about the respondent in order to eliminate them. So that is something that I do when I'm doing survey work for clients. I use email address to confirm that that's the respondent that we think it is or other information linking them including birth year. But for your work, you might have a more expansive definition of your population that you want to work with. So it's up to you whether you want to be strict about excluding people who weren't necessarily in the population that you want to research. But one thing I definitely recommend for any survey research project is a deduplication of responses. This can be a problem with paper surveys or with web surveys as well in that a respondent may respond twice. They may, looks like I may have a question. So let me just pause there. Got a question. In your analysis, do you include partially completed surveys? So that's a that's a good question. And part of you know part of what I'll go into is what in a few slides is what constitutes a completed survey. In some cases I have created, I have declared that a partial survey is a complete survey. But a lot of it depends on what you value. So in some survey research projects I've set a minimum number of questions that make a respondent qualify as a complete survey. In other surveys, it's the number of gate questions. And so as a reminder, a gate question is typically a yes or no question that asks if a respondent has experience to service if they've gone to the help desk, if they visited the library. And if they answer, say I've had one survey project where if they answer a majority of those questions, just the gate questions, and then qualify as a complete survey. So the point is to understand what it is that you want out of your survey. So if you need people to experience a certain number of services that you'll offer, and you're evaluating that through these gate questions, where you get a yes or no answer, and a respondent only answers, yes to one of them, you may want to eliminate that respondent. And any respondent who answers too few of those to reject that as an incomplete survey. Or you could use a mean number of questions. So if there are 40 questions, you might want 20 questions to be answered before including that as a complete survey. So yeah, just make sure that you're setting the standard, the same standard for the entire number of respondents to your survey. There are some other methods beyond just deduplication for data cleaning. There's also disqualification, which not all surveys use. Sometimes when I'm working with a client, I'm required to include all survey responses no matter the responses that are given since they're for regulatory purposes. And other ones I have more leeway, I have more leeway, excuse me, to eliminate surveys that I believe have been answered inconsistently, or just with a lack of attention. So dealing with any population, some respondents will answer questions randomly. You can tell this because they will answer. So the example I have here on the slide on the slide is that two questions are really asked about the same topic. If they had a positive experience about this library, the Petit Library at Penn State, and then sort of the repeated questions asked in a different way if they've ever had a negative experience at the Petit Library. And so if they answer in contradiction in both of those questions, you might feel that you should eliminate just those questions for that respondent, or you can also eliminate the entire survey. I recommend using those last resort. I'm more concerned about beyond just inconsistent responses is when respondents will answer the same answer choice to every question, or it's clear that they're just picking a certain letter pattern like ABCD over and over again. Those are the things you want to be looking at when you're doing your data cleaning to eliminate those responses. So the next step in data cleaning is actually skip logic enforcement. So if you remember from our earlier workshops, skip logic is the structure that's designed in your survey to take respondents from one question to another question. So this typically involves a gate question. So if they have a yes or a no to a question about a service, if they've gone to help the help desk, and they say yes, then they can go on to the next series of questions that are about help desk service or if they answer no, then the point would be to have them skip that series of questions because they wouldn't have a relevant answer to any of those questions. And you want to eliminate those extraneous remarks. Now typically in a web survey, you can enforce skip logic so that the person who answers no to that gate question never sees those questions. But things can go wrong. Sometimes even in web surveys, I've seen people somehow answering those questions. Sometimes it has to do with them leaving the web page and coming back and then the the web survey programming forgets their their response and they're somehow allowed to answer those questions. So it's good to check this at the end of your data collection, you go into data cleaning. And the easy way to do that is to actually look at frequencies. And so I have a frequency example here. Questions one through three of the survey represented. So you can see that there are 160 respondents in question one and say there's no skip logic. Everyone who answers question one goes to question two. So you would expect 160 responses for question two, which there are. But say I put a skip logic restriction that only people who answer question two with answer choice a would be allowed to answer question three. So that would mean that there could only be a maximum of 80 responses in question three. But I can see from my frequencies that there's many more than 80, 80 responses. So I know that something has gone wrong with my skip logic. So the first thing in data cleaning that you can do is to look at these frequencies and then check them against your skip logic to see if anything has gone wrong. And then once you're satisfied that the frequencies make sense that it's mathematically impossible for the skip logic to have not been enforced by your frequencies, then you can you can move on. For finalization, we're moving the extraneous responses. So those responses that the skip logic doesn't allow. We're checking for completeness of administrative data. You may, this is something I didn't answer before. So if you may require the respondent to at least provide some identifying information, whether it's their name or maybe just a pin code that you sent them, if you want, if you're anonymizing your survey. And that way, you know that this is a person you wanted to answer the survey. You have only one survey complete from each respondent. And then finally, we're using some metrics to determine what is a complete survey. So I have a few slides about that. So what is considered a complete survey? Another example is that maybe a respondent hasn't answered a lot of your questions, but if you want them to at least have seen all of your questions, you can require them to have at least progressed to the final question, even if they've skipped them. You can require that all answered questions have been answered. It's been dangerous since most people will skip some questions. Or you can set a certain threshold of questions that have been answered. Or you can even calculate a score based on weighted importance of questions. I think that's a little bit beyond what a lot of us need to do, but it can be done. Or set a few questions that you think are important as the bare minimum for survey completes. So I plot the number of questions answered by the number of respondents that have an understanding of how many people are getting to all of the questions. Of course, I want them to answer as many questions as possible. And just to point out, in this chart, a question is considered answered if they were forced by the skip logic to skip it. So I'm not penalizing them if they have been forced by the skip logic to skip it. It's only those questions they have skipped on their own. They've admitted for reasons that aren't clear. If they just were busier, didn't choose to answer that question. So in this survey, there's actually quite a high number that didn't answer that many questions. And so I don't want to probably put the barrier too strict on this because I'll lose an enormous amount of information. So for this, I would probably put, I would probably actually favor a gate question method just to make sure that they're answering yes or no to a service. Be pretty lenient about the number of responses that they're giving to the other questions. And so I do the same thing for the gate questions to map it out. Same survey. And I'm finding that a pretty big majority of respondents are answering at least two gate questions. So five, seven services. And I want to have people who are at least able to answer about some of the services in the library. Then I would probably set the limit at at least two gate questions answered to be considered a complete survey so that all of the respondents who have only answered zero to one gate questions would be eliminated from the survey and it would not be considered a complete survey. Let's answer the question about the complete surveys or any other questions that may have come up. A lot of times in survey research, the rules are flexible depending on what your intentions are. So if you really want to encourage opinions from people who haven't had a lot of interaction with your services but you are interested in their opinions on that specific service that they've encountered, then you can set pretty loose restrictions on the survey complete whereas if you really only want your responses for people who have visited every single one of your services or you know are committed to your library or spend a lot of time there, then you can set more strict regulations on what is considered a survey complete. So to continue into exploratory data analysis, we're approaching getting getting a final data set. We've established our survey completes. We know how many people have answered our survey. We've done, we've calculated you know how many questions each respondent has answered. So now we're thinking about actual themes in the questions and what trends we should investigate. So we may look at scores immediately. We're going to look at the themes of the questions and how they fit together and then some initial statistical analyses to determine what sorts of differences we have amongst the respondents that have answered our survey. So it's really at this point that we revisit our initial research questions. So if it's something that we prioritize, I would plot answers to the different questions based on the demographic of identification of each survey respondent. So you may be interested in different answer patterns amongst genders, amongst races or ethnicities. And this is really opportunity to start to do that before you go into statistical analyses. Even more just as important I think is that we're looking at the experiences that you're evaluating. When you put your survey design together, you probably had in mind the certain surveys, the certain services that you offer and what you want to learn about them. And it actually may be the case that your respondents think about those services in a different way and that you need to reformulate your hypothesis about how your respondents are interacting with your services. And I'm thinking of a time when I actually had to do this. I had to reformulate a hypothesis from a client, a health insurer who came to us with some data from a different survey and asked us to run a new survey in the following year. And they really were focused on customer service and overall satisfaction. So sure we've all had an experience of talking to a health plan over the telephone so that they were very focused on the telephone service operators, the customer service assistants on the phone who answer questions about the health plan and how that affected overall satisfaction. And so we ran our initial analysis, the initial mean scores and then we started to group the questions together into composites, which I'll describe more later. And this is when we can start inferring things about what's happening in the data. So we noticed that a lot of the customer service questions were heavily correlated with the questions about claims processing. So people were answering the same way to questions about the people who pick up the telephone answer questions about the health plan as the same as the questions that we're asking about how their claims are processed in a timely manner. And so for the health plan those are two separate services, sort of the administrative and policy of responding to those claims and then the customer service, which is really a communication about those services. And so what we were telling the health plan is that the way that the survey was designed is that the opinions about the claims processing session, the claims processing services was bleeding over into the customer service question. So people's opinions was colored much more by the claims processing than by the customer service interaction over the telephone. And so we encouraged, we worked with the health plan to actually redesign questions so that we're asking more specifically about customer service on the phone and separated them structurally from questions in the survey about claims processing so that we could actually tease out the differences in those services. And although that's, you know, it's an example from a health plan. This can happen in a lot of different venues. Sometimes one service offered is much, much more important or just more in the forefront of the minds of your respondents. And you may see at the end of your research project that a lot of those answers are correlated altogether. And that's likely because one experience is coloring everything else, not necessarily that they're all thematically and organically related. And so that could prompt you to redesign your survey to try and reword it and really get the respondent to think about the specific mechanics of the service provided, whether it's the actual interaction at the help desk or whether it's the software they use in the library. So I just, I use that as an example, because this is really the time when you're able to start to see those patterns. And you can think about the ideas will start to pop up about, you know, redesigning the survey for next year or just not overinterpreting your results. So if you get similar answers across questions, you don't want to, you want to get in the mind of the respondent and think about what is their most important, what is the most important priority for them? And I'll get into some statistics that actually help you identify that. So for, you know, for subgroup analysis, you know, I like to actually, I like to track by age group and do response rates. So it's pretty common for younger respondents to be less responsive to a survey. You'll just get fewer responses. So it's a good idea to plot that to see if you're looking at your overall results in the survey. If a large number of your respondents are older, it may not be as representative of younger opinions than you would have thought. And so that's something to keep in mind when you're reporting your results. And I will get into a discussion about waiting results as a method to, as a method to deal with that, although there are some concerns about using that method as well. So another thing that comes up during exploratory data analysis is time sensitive evaluation. A lot of survey projects are run to evaluate a program that has happened over the last year. And so when you're looking at your data, you can focus in on those time specific questions. If there's been a change, a new initiative that you're measuring, you can see if your respondents are answering different to those questions. Maybe they're more positive about the new initiative, or there's just more responses to that question. So you know that there's interest in it. So just to sum up, when we're revisiting our initial research questions, we're thinking about which components of our services are most important. What information can we gain from the survey that is actionable? And this is important for working with stakeholders. I find that when I report, you know, cross tabs by themselves, they're just mean scores to a bunch of questions. Service can tend to, can be pretty long. They're getting longer. So it's hard for people to orient to 30 questions on a survey. And so if you just, if I just report the mean scores, it's going to be hard for my stakeholder, for my client to know what to do with that information. And so the statistical analysis that we do will help you present that information. And how do we gain information from our survey to create or modify a new program? There's the time sensitive project. We can use information from those questions to touch on that project to redesign the survey or to, you know, reorganize our initiative for the next year and test the change in that initiative and how it was implemented and the respondents reaction to it, which is something that I think has come up in a lot of your surveys with respect to the pandemic and how your services have changed. So when there's a big event like that or an initiative, survey research can really illuminate your respondents reaction to that and how they perceive the implementation, particularly when there's a fast change. You may not have had a lot of interaction with your population. So particularly when people aren't seeing each other, survey research can be even more important to gain information about events like that. So I think you mentioned once or twice about revising the survey. Is there a caveat, I guess, percentage of revision that would impact your longitude no exploratory analysis? Yeah, that's a great question. So yes, it is true that when you're redesigning your survey for the next year, you're changing your wording. It always comes with a caveat that you may not be able to make a direct comparison between those individual questions. However, I'll get into this a bit later, when we talk about composites, I'm putting together questions that are related to a specific theme. So if you're reporting just composite scores, according to one theme, say customer service, and the individual questions change within that composite, you can still report the customer service metric composite score. And that can be a thematic comparison between your change survey from one year to the previous year. But it may, yes, it may have an effect on, it will have an effect on the individual questions, which is sort of why I push back against reporting individual questions, results, and relying on them since you may find out that you get rid of that question the next year. But if you create a composite, you're really killing two birds with one stone, you're making it more interpretable, and you're also making your survey more reliable and stable over time. So it's really, if you're testing the same themes year over year, I don't think there's a big question about the fairness of doing a longitudinal comparison. But whenever I'm doing a survey report, and the question wording has changed, I do make a note of it to alert stakeholders that part of the reason that the response opinion has changed may be because of the wording. So I just want to make sure I'm good on timing. But yeah, these questions are great, so feel free to break in. For severe response rates, I don't want to spend too much time on this. It does have to do with waiting, which I'll get in two minutes, but yes, yes, please. Oh, I just had a question around just sort of a follow up to that. When it's apparent in the responses that the interpretation of the question was obviously, there was a high degree of variation, right?  And you try not to provide an example, so not to be leading, but then obviously the interpretation. I'm thinking, for example, around research data management, there's many different types of ways that, and job types. And so there are obviously respondents that respond in a certain way, others respond in another way. You see that that's a potentially fruitful area and yet, you know, where would you go with that in terms of, you know, comparison, you know, adjusting for say the next survey? Yeah, that's a good question. So if you've already conducted the survey, you know, I would follow up with a few of the respondents, if you're able to, if it's not anonymous, from sort of the two different interpretations or maybe the different interpretation they have of research meant to see what they are actually thinking. If you have, if you see this problem coming, the other thing you can do is to allow them to write in a response that explains in more detail what they're thinking about that topic, so that when you redesign your survey, you sort of know the, the points of differentiation in their opinions. But I think, yeah, when there's, when there's confusion about the question itself, and the survey has already been completed, the best thing to do is to follow up and do some interviewing to understand how the respondents have responded, have interacted with your question. Is that something that's possible in your case? Our, the particular survey I'm thinking about was actually a national survey and we asked, like, so the responses were collated around by institution and, and then for anonymization at the institutional level. So, we did have a textual answer, but as you can probably imagine, even within the textual answer, I mean, people are reporting, you know, fairly broad job titles. Job titles are not necessarily descriptive, depending on what you're trying to answer. So, I mean, we definitely saw it as a place to explore, you know, for the next go round and sort of getting maybe a gate question with like logic based on that. But, yeah, as a comparison, we struggled with that as a group. And so I was just kind of interested to hear. Yeah, I mean, that can be a challenge. It is, it can be, it can be difficult to write a question that you don't want to lead them into giving you specific responses, but then you get responses that are too disparate to really work with. So you can use, you know, typically, and since our language is pretty regulated, this is something we deal with quite a lot is to have a specific definition that, that tries not to be leading. So we'll have some expository text. So we might describe what research management is without mentioning any titles, and then, and then guide them into answering questions about that field. But I think at the same time, you have to, there's the element where you have to take the responses at their word. So if, if there's sort of a different differential interpretation of what research management is, then that's actually sort of an answer that I think is actionable, that problem isn't so much, the problem may not be the responses to research management is that, that some institutions may not have a concept of what that is for them necessarily. And so you can create a follow up survey to go into more detail if you have that ability. Okay, yes. So survey response rates have a breakdown here by race, since I want to see the difference between the response rate by, by racial subgroup. So I know the, the natural makeup of the population itself by, by race. And so in this survey that I did for a, for a client, African American respondents were underrepresented relative to the population and Asian respondents were overrepresented, while white respondents were about, were about, we're responding at about the same rate as you would expect from their number in the population. And so this is useful information to know that when you're reporting your results, that there is a slight difference in the demographic makeup of your respondents compared to the population itself. Now what to, what to do about that is something that's very tricky in, in survey research. And that can involve waiting. Let's see. Yeah. So for, for waiting, and I do want to talk a little bit about waiting because it comes up as a, because I know that people get frustrated when they don't have an exact when their sample isn't the same as the population by demographic makeup, which is pretty typical since it's very difficult to get that to match perfectly. It's, it's common. And some groups just have different response rates than others. One of the challenges with waiting is that, so for, for, for with waiting, what you're doing is you're taking the responses from that, that population. So for the African American group, for example, in this, in the survey project, and then you're reweighting them as a percentage of the entire population. So you're taking the responses from that small, that group, and then expanding them as more important than the number of respondents, but as important as their demographic makeup in the population itself. So you're using that smaller than expected response group and expanding it to its proportion, the population itself. So this controversial, controversial because what you're, what you end up doing is you're sort of doing a mini survey within a survey.  And you're actually expanding that error by increasing that weight within the population itself. And so that can have real life consequences. And we've actually had a good example in the 2020 election. So in the 2016 election, a lot of pollsters made a correction when they, after finding out that their results were erroneous, that voters without a college education were less likely to respond to surveys, but nonetheless go to the polls. And so their answer was to actually weight those voters more heavily. So in the 2020 election, when they got the same response pattern, those the same voters weren't responding to their surveys. And then inflating that population to its projected proportion of the voting population itself. The problem was is that they're the people who didn't respond to surveys were characteristically different from college to voters without a college education. And that actually made the problem worse in a lot of states. So you're actually taking that error from the, the, the characteristically different people who didn't respond to the survey and expand and expanding it because you're just taking the people who did want to respond to the survey, which has ended up being the cause of the error in the 2020 election. So it's one of the challenges when dealing with weighting is that if you think that there is a characteristic difference about of the group that's not responding to the survey, then weighting it, weighting without their responses may actually make the problem worse. And it's better just to go with the demographic makeup that you've received. However, if you think that there is no characteristic difference between the people who have responded to the survey and who haven't within that group, then it is safer to do a weighting protocol. But you know, there's, there's a pretty big controversy in survey research. So there's no easy answer to this. So I encourage you, I encourage people who are running survey research projects to be open about the different demographic makeup of their survey respondents compared to the population as a part of your discussion and then to weight carefully. Now, for a lot of you, this may not actually come into play since you're just getting responses from everyone and you're doing the best you can. There's not a whole lot of things that you could do, whereas pollsters have millions of people that they can, they can reach out to in addition. So the better solution in a lot of ways is to oversample, to talk to more people so that you can actually get more respondents from that specific subgroup, whereas that may not be an option to use since you're already surveying the entire population. So answer choice counts. It's a simple way to look over your responses to the individual questions of your survey research project and get a sense of your respondent preferences as you're going through your exploratory data analysis, which I'm sure many of you have done just naturally. But then cross tabs, which I mentioned before, I think are an underutilized tactic in looking at survey results and exploratory data analysis, which I'll describe here. And so what a cross tab is, is you're looking at how one group was responded to a question in one way, how they've responded also to a different question. So for example, you can look at the people who responded yes to question one and see how they answered question eight. So if they answered positively to a question about customer service, how are they also thinking about software? For example, software in your library. So that can reveal hidden subgroups and preferences within your data. So you might find that across tab with younger respondents, they're really interested. They have positive opinions about the software or maybe your older respondents don't. We can also find that people who respond positively to customer service are just responding positively to everything, they're just a positive group. So you basically just put the, you pit the two questions against each other and look for trends for when you're doing your exploratory analysis before you go into a statistical analysis. So one of the things that I am looking for and doing cross tabs analysis, you know, I'm looking for errors and sorry, design because it can help you look at the counts and you get one more chance to fix skip logic. If the frequencies are not matching up with what they should be, should be based on the skip logic. And then something that I like to look at is sort of confounds and demographic variables. So maybe you're interested in ethnic subgroups. So I work with a lot with survey with clients who are interested in opinions from Latino respondents. And so one of the issues that I always have to deal with is that Latino respondents tend to be younger than other demographics. And so it's not always clear. And I can see this in the, in the data, whether I think whether the statistical analysis will determine that the effect is due to the, the youth of the respondent or because of the ethnicity. And so it's good to know those confounds in advance before you go into a regression analysis, which will actually be in the next workshop that we have, or before we even do some correlation work. Let me show you what, just a visually, visually what a cross tab is really just a matrix. And so for here, you can, you might want to zero in on the answers to question one. So the most frequent users of your library and to see how satisfied they are with your library services. So from these cross tabs, I can already tell that those who have visited the least, who have visited the library the least, are the least satisfied, whereas those who have visited a lot are most satisfied. So you might actually think that that is a good thing that the more that they come to your library, the more satisfied they are, because you, those are your, you know, that's your key audience, the people you want to to, whose attention is most important to you. And then also affect, affective age can be important too, whereas in this question, I'm seeing that there's an effect of age, whereas older respondents are less satisfied with library services. And from this, I might, you know, I can re-evaluate my hypothesis of the hypotheses. When I do my correlation work, I can try to see what it is that older respondents think is important compared to younger respondents. Maybe older respondents are more interested in opening hours, or younger respondents are more interested in software. So you can do, so it's really when you're doing your cross tabs work that you're, you're thinking of questions that you can test with your correlation work. And item analysis. So item, an item is just a survey question. We're going in and identifying survey questions and putting them together into composites that are, that have a specific theme. So this was a little bit covered in earlier workshops. I just want to go over it again, survey validity and reliability. The two keys to, to a successful research project, validity being a survey that intends, that measures what we intend for it to measure. So we're looking at wording questions in the way that we'll get at the concepts that we're interested in, and not something unrelated to what the concept we're interested in is. Reliability, which is the ability to get the same result from the same survey if it's given a second time. And then the right reliability in the sense will each respond and understand the question in the same way. And so when reliability statistic that's become very common in survey research is, is known as some crown box alpha. It's not typically used for survey overall, survey reliability, but it is used for composite development. And the history of this is that surveys used to be shorter. So they used to cover just one concept. So the alpha statistic was used to determine the reliability for that entire survey measure, whereas now surveys are longer, so that we're looking at multiple themes within a survey itself. And so the alpha is actually used to put together series of questions to develop composites. So I already covered this a bit, but so composites are there to, for, for a number of reasons, it improves interpretability. Instead of reporting single questions, you can report themes that people can latch on to. Customer service opening hour software and other things I mentioned already, I'm sure you can come up with more. It's easier to track changes over time. Since if you substitute one question for another, if you change the wording, and your crown box alpha still establishes the statistic establishes that it should be a part of that, of that same composite, like you wanted it to, then you can safely report through the composite results without having to, you know, talk about individual question wordings as much, since you've already established that this composite is testing the theme that you wanted to test. So, which is really an improvement in construct validity. The key thing is, I want to point this out, you have an idea of what you want your composite to be when you're doing your survey design, but these composites are generated after data collection. So when I put together a survey, you know, I might have four questions that I think are customer service questions. And then I find out that maybe only two or three of the four here are related to each other, or thematically related to each other, using the alpha statistic. And those three two or three questions will get reported together as a customer service score. And so the alpha statistic itself, it's not a statistical test. So it's not, you're not doing a test for significance. You're looking at the variance among the questions that you that you think are in a composite. So you might put together 10 questions you think are in a customer service composite. And the alpha statistics statistic, which I've calculated in SPSS. Next slide. We'll give you a statistic that tells you the degree of variance amongst those questions. So the similarity and variance. And it's really you are heuristic to select questions for thematic composite. So you'll typically a statistical program will give you a an overall alpha score for the the questions that you've identified are what you think are in that composite. So as if it's a mini survey itself, and then it will give you a list of the alpha scores if you eliminate a question. And so on the example here that I've provided, you know, it's a five, I thought that six, six questions would be in the composite. But it turns out if I eliminate question for the alpha statistic increases to a pretty high number, almost point nine. So I'm willing to drop that question from the composite, since it's not as related to the other questions. And so I would probably have a pretty solid five question composite here. And so the heuristic that I'm the statisticians usually go with is anything over point seven is good. But if there's a major drop in your alpha, that that sort of that interferes with your your alpha statistic, then you can drop that from the question. And so the alpha is really looking at so the correlation between the questions, but it's a function of both the number of questions in the composite itself, and their inter item, their inter item correlation. So the larger number of questions you have, the more, the more leeway you have. But if there's only a few questions in the composite, you know, there'll be a large change in the alpha if you drop one, compared to if you have ten you only drop one. So what's it what's important for me? I like to do Pearson correlations of my my survey research project is to differentiate these these two, these two topics, these two concepts. So crownbacks alpha is a measure of reliability, it's not really it's not used for hypothesis testing, you can't run a significant test on it and it's used for making a comparison amongst multiple questions. You'll get a single metric for a series of questions, whereas the Pearson correlation is I'm trying to determine the covariance amongst two questions. It is used in hypothesis testing, so you can figure out if two questions are significantly correlated with each other. And it's key that it's only two questions. So this is important for my work, because I want to figure out typically how different concepts in the survey are related to overall satisfaction, which aspects of the services that my client is offering are most important to driving overall satisfaction, since that's what they want to pay attention to the most. And something that's important to stakeholders is to find out, okay, we have these resources. And we have this amount of time, what can we focus on to improve our scores? What can we do to improve our respondents opinion to make them more satisfied with what we're offering them? So I'm running a little bit of time. So I may come back to the scoring methods. I think you're familiar with the T test, but I do want to show the Pearson coefficient. It's mentioned, yeah, it's useful in hypothesis testing, but I want to show you what it looks like when I'm running my correlations. So each question, I've given a name that's related to the concept that it's getting at. And in this case, it's these are individual questions. I've just given a different name, although you can do a Pearson correlation for the composite scores as well. And so if I'm interested in this typical, what is the driver of overall satisfaction? I want to see which question has the highest Pearson coefficient. And so in that case, that'll be customer service is the strongest driver overall satisfaction with opening hours being farther behind and software availability being among the least important. But you can also see other correlations between the questions. So the answers to visit frequency are similar to the answers on opening hours and also for study rooms and opening hours. Okay. So I'll touch on some scoring methods since we have a little bit of time, but I do want to leave some time open for questions. So there actually are different ways to approach scoring answers from your respondents. Probably the most intuitive ways to do proportional scoring. And this scoring is dependent on the number of answer choices to your question. So in this case that I'm showing you, it's a never sometimes usually always question. So there's four answers. And so you just divide them into the zero to one scale, zero for never and always for one. And so this is to translate the words into a numeric valence. So that's a pretty intuitive way of doing it. However, this is also pretty common. It's called top box scoring. And this is when you're prioritizing, when you're sort of, you're dichotomizing your responses. Say at the end of your survey, you realize that a lot of people are answering very satisfied, satisfied and either dissatisfied or satisfied. You're not getting a lot of other responses. You may want to separate those two and you're kind of, you've looked at your responses and you realize there just isn't enough variance in those answers. So instead, you're doing top box scoring. So you can tease out the difference between a satisfied respondent and not very satisfied respondent. And this way, you can just assign a one to the two top scores as if that's a perfect score for you. Whereas everything else is a zero, which is the lowest score possible. And that way, you're, you're teasing out minor differences and sort of making up for an imperfect question design. Because you didn't anticipate in advance. And it's not always possible to that your answers would be skewed from your respondents in a certain way towards being very satisfied and satisfied compared to not satisfied. And so top box scoring is a way to separate those respondents. So composite scoring. So the most intuitive way to do composite scoring is simply to take the proportional score for each question. So for the example here, you can see at the top, at the top right, there's the answer in the numeric equivalent, and you're just taking an average based on the number of questions in the, in that, in that composite. Problem is that, you know, some respondents will skip a question. And so you have to figure out what to do with that skip question, whether it's to simply remove that question from the average itself and then just put the, the total over, over three, because that because of that one question. The problem is, is that when you do that, what you're doing effectively is imputing the answers to those three questions, the fourth question, so that when you're reporting the composite score, you're effectively reporting that all the questions, when you're, when you're doing an interpretation and someone's reading your report, it's going to look like all four questions or were answered for everyone, and that each question had an equal contribution. So that might be okay if the respondents are all answering the questions in similar fashion. But if there's one question that's getting skipped a lot, you might want to try a different method. So what you can do is actually just, you can wait to the questions in a different fashion. So if one answer, so you can actually wait the question that hasn't been answered as much more, so that to compensate for the fact that it has been omitted so many times. So that's one of the corrections that you can do, or you can actually do some imputation methods, which I will actually get to in the next, our next workshop to assign a to assign a response that's been omitted based on their responses to other questions in the composite to the survey itself, which is something you can do if you're allowed to, you know, if you, if you want to learn more about your respondents, but are also allowed to impute responses sometimes for compliance reasons, it's not possible to, to do that. Okay, okay, so we just have a few minutes left. So taking you through the quantitative analysis, which is the approach you use when you have your, your survey data collection, you've established what is a complete survey, you've done your deduplication, you've done your exploratory data analysis to find trends, and to reformulate your hypotheses. You form a composite to improve your, your survey reliability and your validity and also to improve the interpretability, especially if you're doing a repeat survey over time, longitudinal analysis, and you're doing hypothesis testing based on your expert knowledge of your, of your population itself. So if you have intuition about how different subgroups are answering, you can answer those questions with some, with some work with, with the Pearson core, the Pearson coefficient statistic. So just cover what I wanted to talk about for quantitative analysis, there will be a workshop in October on survey reporting, which we'll get into regression analysis, survey preparation for future cycles, future iterations of your survey work, report structure and development, and then developing reports and dashboarding for stakeholders who are interested in your work. So just want to thank you for your attention and for your questions, but we do have some time for, for questions now. So feel free to, to jump in. Just clarify, you said you were using SPSS here for, for all of your examples. For some of the, some of the graphics are in R. R, okay. Even the, the, the race tables from R or, or SPS, but I've used data a long time ago, and I know it used to give you some pretty crude graphs too. All the, all the visualizations are from, from R, just the, yes, the, the statistical output is from SPSS. Okay. Yes, I just have a general question, Kevin. And there's a lot of decision making in terms of handling the data as, as we've talked about, you know, what's complete, what's not, are there ways that you found to kind of counter any bias on your part when you have to make decisions like that? Yeah. Yeah. That's an important point. I think it helps to, you know, to use your, your professional experience to sort of write out what it is, the analysis that you want to conduct, and then to evaluate which of those analysis could lead, analyses could lead to, to bias. So if you're interested in demographic subgroups, you may actually decide that you, you want to limit your exploratory data analysis to, to other topics, so that you're not, you're not developing preconceived notions about what you see in the data. You know, at the same time, when you're doing your exploratory data analysis, you're, you're still early enough in your work that you're not, you're not doing the sort of regression work and the reporting work, which will come in the next step of the process. So there's still some separation from what we're doing here to reporting. So there can be another, there can be another time period if you think about the work that you've done and to see, you know, how bias could have entered into your analyses. But that's when we're, that's when I'm typically talking to other colleagues in my, in my group in the cert, who are, who are working with me on the survey project itself, to see, you know, how, how, how else they would be conducting those analyses. There might be a different method for mine and to work out, which would be the best approach as we move towards the reporting period. So I will, I will go ahead and add my, my email address in case you all don't have it. If you have any questions about, you know, the presentation today or any questions about survey research that I could, I could assist you with, whether it's dashboarding or anything you heard in earlier presentations. Great. Thank you, Kevin. I would also encourage our colleagues here to reach out to Kevin, because he is available to, to provide some consultation service and advice, especially to our teams as, as you begin to analyze your data and begin to think about how to interpret what, what you have collected. So please do reach out to Kevin. Kevin, it looks, it looks like we may be at the end of questions. So I would like to add my thanks again to you, Kevin, for leading this workshop. And I look forward to seeing you in October for the other workshops that you'll be offering. And just as I said before, we'll send materials out to everyone from today and encourage everyone to sign up for the next set of workshops. So thanks everyone for attending. We really appreciate it. Thank you, Kevin. Thanks so much, everyone.


Tina takes the snap, swings it out, intercepting tights, 40, 30, 20, 10, 5, in zone, and touchdown!