Let me welcome everybody. I'm delighted to see so many of you here today to discuss wonderfully deep and important topic with a fantastic guest. My name is Brian Alexander. I'm the Future Transforms creator, host, and Chief Catterer, and I'll be your guide to the next hour of conversation. I am absolutely delighted to be able to welcome Richard Aron. Richard is a professor and a dean at UC Irvine. He's the author of one of the most important and controversial books published in the 21st century about higher education, academically adrift. He has been researching and exploring how to measure and think about student learning, and most recently he is finishing what I believe is the first phase of the next generation undergraduate success measurement project. If you'd like to learn more about that, well, we'll be doing it together, but the bottom left corner of your screen, you should see a kind of yellow or tan-colored button. Just press that button and it will pop the homepage of that program. Now, let me welcome Professor Aron and bring him on stage. Greetings! Thank you Brian. Really grateful for all the work you're doing with this forum in general and for inviting me onto it in particular today. Well, thank you.  That's an honor. We ask people to introduce themselves using a particular method. I'd like to ask you what you're going to be working on for the rest of this year. What's the, what are the big projects and the big ideas? They're going to be top of mind and soaking up most of your time. That's great. Great question. Yes, others just to be complete, there's two projects I'm working on at currently. One is on a local project in Orange County. How do we help educators and social service agencies better serve foster youth and housing insecure youth? I'll set that aside. It's really not higher ed focused, but the other project that is my primary project, one we'll talk about today, the next generation undergraduate success measurement project that was funded by the Mellon Foundation to set up a state-of-the-art measurement system to track undergraduate experiences, outcomes and trajectories in a much more holistic, comprehensive manner than kind of the existing measurement that's occurring in the sector. Which is one of the main reasons I wanted to bring you here today to dive into that. I'm happy to jump in and explain some of the project goals, the way we're going about the measurement work, our holistic framework for thinking about student success and then you also share some findings today from this project that we can kind of learn from and help inform policy and practice moving forward. Fantastic. Friends, I have a couple of very, very quick questions to get the ball rolling, but the purpose of the forum is to be a platform for your own questions and comments. So again, just look in the bottom of the screen and press that raised hand button if you want to join us on stage, bearded or not, you're all welcome, or just type in the question mark and type in your question or comment. One quick question to start off with is where are you in the process? I was trying to guess that you'd finished your first phase and that's a very good attempt. So we are, let's see now, we are almost 18 months into data collection. We're just about to launch the second set of these innovative performance assessments we developed so to measure growth in some competencies we think are aligned with 21st century workforce needs and liberal arts education values. So let me share, since we started talking about the performance assessments, let me talk about that and then let's just put a little flag in the conversation that we're 18 months into this, that means we started data collection before the pandemic and also have this kind of unique opportunity by unfortunate accident to kind of understand the effects of the pandemic and moving to remote instruction on the lives of undergraduate students and outcomes as well. But let me say since we started talking about the performance assessments, that's one of many different data streams we're using in this project, but since I mentioned it, let me explain a little bit what we did there. So we had the good fortune of partnering with ETS, the big assessment firm, to not just use their off the shelf standardized assessment items, but they were so interested and excited about our project that they partnered with us to co-design new assessments that we thought were better aligned with the goals ahead. So we did use their off the shelf measure of critical thinking, it's called the Haydn assessment, it's a commercial product, it could be used by anyone, but then we designed a suite of other performance tasks for students to undertake. First we designed a collaborative problem solving task for students to engage in. Four students are put into a virtual space, they're given a set of documents to examine, they solve the problem first individually, then they're given an opportunity to interact with each other to share information from the documents they have and solve the problem again after the collaboration. We're able to observe the actual collaborative process as well as the extent to which the collaborative problem solving result at the end is aligned with the desirable outcomes. So collaborative problem solving, we also very interested in perspective taking, can students develop the competency to take the perspective of someone different than themselves. So they're given these scenarios, complex social scenarios where they're asked to take the perspective of diverse social actors in the scenario. We're also very interested in the 21st century at this historic moment around confirmation bias, individuals change their opinion when new information is presented to them. And so we have a confirmation bias task to assess students ability to change their opinion when new information is provided. And finally we partnered with Sam Weinberg at Stanford, a history project at Stanford, he's doing excellent work on online civic reasoning. Students are given something that they might see in Twitter, Facebook, social media, are they able to make sense of it and determine the reliability and validity of the data. And so again, we did these initially with students in fall 2019. We're going to assess them again on these competencies in spring 2021 and next month. And then we did a whole lot of other data collection, other than these performance assessments as well. And I'm happy to talk about that those other strands as well. Well, this is great collaborative problem solving, taking other perspective online civic reasoning, grappling confirmation bias. And that's just one set of what you wanted. Exactly, exactly. Tell us about a couple of the other strands. Yes, so we've got one strand, the project we called, did you see the administrative data strand, that has all the, you can think of it as the conventional data that colleges and universities often routinely collect, course taking, grades, credit accumulation, choice of major, the college admissions data. So we have some background information on them. And then some other administrative data that some universities like Georgia Tech and others make use of, others don't, data on what, when and where students are interacting with student support services. When they engage in academic advisor, when they engage in tutoring services, when they engage with the career service office, and so on and so forth. And so we also integrate all this administrative data into our project. That's again, the administrative data strand. We also make use of something that is kind of rare in higher education, but I believe an incredible opportunity. We make use of data generated in the learning management systems. So today, everyone is interacting through these systems, Canvas is one of them, Blackboard. There's a few different ones, but much of the interaction between student and faculty are increasingly mediated by these learning management systems. That was true before the pandemic, obviously, even more so since. And so what we've been able to do with that data is generate academic engagement measures for every student in every course at UCI, how many hours they're spending in the course, how online, how many times they go on the site to access materials, the extent to which they interact with faculty on those sites, and they interact with peers on those sites. We also have access to products that they deposit there. So again, an authentic way to measure how students are engaging in the academic enterprise, and we use that data to integrate it into this measurement system. And then the third and final strand is extensive survey methods and experiential sampling. And so the administrative data, the LMS data we look at for all undergraduate students, there's a 1200 students in a cohort, freshman and juniors, we take on each year. And for those students, they get the performance assessments and they also get extensive surveys. Those 1200 students will receive eight surveys over the course of each year, a background survey about their orientation, their goals, their values starting college, but then also at the end of each term, how they're experiencing their classes and what's occurred in their classes in terms of the type of instruction they've received, as well as again, their goals and how their goals are evolving over time. A subset of a third of those students, they get even more surveys. For those students, they get weekly surveys and also two weeks during the year, we do experiential sampling with them, where they get buzzed on their phone, what are you doing, who are you doing it with, and what's your emotional state. And so we've got this very fine-grained, detailed data on what students' lived experiences are at our college, at UCI. And this is from everything from their interaction with their peers, which is key to the holistic development you would want on a college campus, to experiences of discrimination and microaggression. So we do our best to capture the full range of student experiences and we do it for the following reason. If we are serious about serving our students well, we have to understand, in a broad, holistic, comprehensive framework, how they're experiencing undergraduate life. How these experiences track with outcomes and their trajectory. We can't intentionally design without that. We can do it on the basis of ideology and hunches and so on. But that's not, every other industry in the world today uses data to better and intentionally design how they interact with the clients they serve, the consumers, the customers. Or if Netflix does it, my gosh, in higher education where we have something much more serious in our charge, the responsibility of supporting student growth and development to face the incredible challenges that we're going, that we're facing in society, moving forward the environmental crisis, the current public health crisis, the political polarization in our country. My gosh, do we have a responsibility and a need to figure out how to best serve students so that they are developing in a way that they'll be prepared to meet those challenges. That's a very, very stirring call and by God, that's an enormous amount of research being done. I've never heard of anything anywhere near the scale. This is true, but let me get out of the way. Thank you for giving us that introduction, the rest to the work. And again, remember, everybody on the bottom left of the screen, there's a button which will take you to the website for the project. But I wanted to share some of the questions that have come up so far and again, invite you all to add your own questions and thoughts. As you can tell, Professor Aram is definitely happy to share with you. So to begin with, we have a question from Paul Walsh at USM. Does the project give students a live trending outlook so students can see how they are progressing in relation to their efforts and to previous students? Yeah, that's a great question, Paul. So one way to use data like this is to personalize instruction and to use the data to feedback to students so that they can better understand and take ownership of their own growth. Those are things we might aspire to get to down the road. It's not something we had the resources or ability to build into the system at the current stage of the project. Our goal is let's collect this data so that we can inform institutional improvement efforts, continuous institutional improvement efforts, as well as use this data so that the educational research community, the social scientific research community, can better understand the value of liberal arts education broadly conceived and also identify what works and what doesn't work in terms of these underlying educational processes. So this data that we're collecting, it's going to be de-identified and deposited at the University of Michigan in the data archive ICPSR for the larger social scientific community to make sense of. Let me say one more thing about that because if you've been reading the newspapers recently, Biden's talking about infrastructure. Let me tell you what the most important infrastructure we need in this country today is. We need infrastructure about how to deliver, measure, iterate, and improve higher education. I can think of no greater infrastructure need than that because individuals alone can't do this. We need to build the infrastructure so that we can improve higher education, not just to do better by the students we currently serve, but to increase access dramatically so that our country is prepared to meet the challenges we're going to face going forward. Shame on us in this sector if we aren't articulating that need to leaders in Washington and demanding that these kinds of investments get made. Well said. Paul, thank you for the great question and Richard, thank you for the multi-pronged response. Again, if you're new to the forum, this is a great example of the Q&A tool so you can just definitely use that. We have a question from the excellent Steve Ehrman just published his new book and he asks, how do your measures attract students to work softly enough to illuminate their current capabilities? Yeah, great question, Steve. One of the key challenges around performance assessment is motivation. If the students aren't motivated to do well, the reliability and validity of the measurement gets called into question. Going back to the academically adrift work, one of the critiques was that the reason why students didn't grow on these on these instruments was because they weren't motivated to take them seriously and the longer they spent in college, the less motivated they were to do something you put in front of them. So they're fresh, but by the time they got to be the end of their sophomore year, senior year, they were turned off to assessments and academics in general. And in fact, I hate to say it, from the Wabash study of liberal arts, longitudinal study, their measurement suggests that again, the longer students spend in college, the less academically motivated they are. If you're a committed educator like myself, really challenging finding from the Wabash study that really we need to do a lot of thinking about how academic engagement is structured and student esteem is relevant to students and motivated. That's a long-winded way to get back to Steve's question. So one is kind of on a basic level, how do you get them to do these surveys and assessments in the first place? We provide some limited economic incentives for students to take these surveys, something like $50 for the year to take the surveys and the performance assessments. For the ones that are doing the weekly surveys and the experiential samplings, they're signed up for an independent study course. And so they get course credits, essentially one class over spread out over the year to engage in these surveys, which are really about reflecting on their own education. And so they're motivated in that way. Finally, in terms of the performance assessments themselves, I can't say this about the off-the-shelf critical thinking measure, the heightened measure that we use from ETS, but the ones that we co-designed with ETS and the online civic reasoning assessment that we borrowed from Sam Weinberg at Stanford, those tasks are very engaging. The tasks themselves are kind of very engaging and motivating. So you're put into a group and you're, which of these three candidates should we hire? Here's some information on them, interact with each other. It's an engaging task. Thinking of yourself in this complex social scenario, engaging, confirmation bias a little bit less so that that task. But the online civic reasoning is, I don't want to reveal the items, but again, quite engaging. You can imagine scraping the social media to get a sample of what's out there and then asking students about reliability and validity. It would be hard for that task not to be engaging. That sounds like a lot of fun, a bit of role-playing. We have more questions coming in and thank you for that one. And I'm kind of building on that. We have one from Ray Garcelon. He asks, do you think there are any useful strategies or guidelines to effectively integrate external measurements of student success as a part of a holistic system of student learning, i.e., a nursing program? Yeah, thanks. So that's great. I think a lot about that. Ray also, in my day job as a dean of an education school, we have a teacher education credentialing program where there's always an interest to see not just how are they performing in the program and the competencies that they're developing in the program that are assessed there, but also to track them into their jobs as teachers and get feedback from the districts and from the field at the extent to which they're actually able to engage that work. So I think it's a very useful thing to do in general. Where we are in the project, we're going to get to some of that in the next two years. So I mentioned we started data collection in fall 2019. The design of the study was that the 1200 students in the sample that are with the extensive measurement, they are half freshmen, half juniors. Some of those juniors, essentially half of the juniors, a quarter of the sample, continuing juniors, they had started at UCI as freshmen. The other quarter are junior transfer students. In the UC system, for every two entering freshmen, you take one junior transfer student in, primarily from the community colleges. So we've got, one way of saying, from that fall 2019 cohort, roughly half of them were juniors, they are graduating, the ones that are graduating on time, will start to graduate in May this year, sorry, June this year. And we are, the Mellon Foundation has invited a proposal from us that's at the June board meeting they'll decide on. So I don't want to get ahead of the foundation. But if all goes well, we will have two years of additional funding to track those students after college into the labor market, into grad school, into life. Because college is not just about labor market outcomes. It's about a broad set of human development. And so we want to, we should really be interested in how they are faring in, after graduation, in terms of, broad set of outcomes, in terms of human development and general flourishing. Well-being, general well-being. That's an ambitious answer. I really want to see that go forward. So let also, let me be clear, I haven't had a chance to say this yet. It's not just me, Brian. So we've got an interdisciplinary group of working on this project at UCI. There's 12 faculty involved, leading people from developmental psychology like Jackie Eccles. She's the foremost national expert on student motivation. She's very involved in designing our surveys, taking a leading role in that. We have Mark Warshauer, National Academy of Education on Ed Technology, guiding the learning management systems analysis. We brought in a three dozen leading experts in the country for convenings at UCI prior to data collection to advise us on developing this measurement system. So let me assure you this, Brian, if this wasn't left up to just a sociologist like me, because I wouldn't have the ability to kind of put in place a comprehensive project of this character that's really interdisciplinary at its core. Well, my congratulations to the entire team, which sounds like a swarm of people. Yeah, and then imagine the doctoral students and we also involve undergraduate research assistants. It's several dozen folks on campus that are actively working on the project. Oh, let me say one other thing about that I'm really proud of. The undergraduate senate at UCI formally endorsed the project. We brought it to them. We said, you know, that we're measuring undergraduate student experiences, trajectories and outcomes for this reason. We'd like you to consider it and weigh in on whether or not this kind of work is in the interest of students. It's formally endorsed by our undergraduate student senate because again, this is work that every college and university should be doing in one fashion or another. We've got questions that actually address several different parts of your answer. Let me bring in our friend Tom Hames who books at one of these. Is this data being used to form part of the instructional redesign process and which isn't the same? Is this being shared with faculty real-time? Great question, Tom. So it is our goal to develop what you've called real-time ways to share this data. One can imagine a set of dashboards, data dashboards that faculty could have access to where they were able to observe outcomes generated from our project. Some of that is indeed happening right now in the following way. These kind of dashboards are being set up by our division of undergraduate education using data to inform instructional redesign. So it's kind of some standard things like where are the roadblocks, courses with roadblocks where students are getting DWIF. You're able to identify those in different majors. In real time, you're able to track student outcomes of track a dashboard of what courses students take subsequent to a particular course and what are the grades in those courses. And they're building into those systems. The measurement we're doing is informing the types of measures they're building into these dashboards. So the learning management systems, academic engagement measures that we identified are going to be part of that internal data system at UCI. So that effort is underway.  We have the good fortune of accreditation happening next year. So it's a joke because everyone in higher education don't people dread having to fill out their forms and all the exercise. But at its heart, it's about using data. It should be reflective on practice and improve practice. And so coincidentally, our accreditation is coming due next year. And if you're an administrator at UCI trying to think about where are some learning outcome datas on student experiences to see whether or not we're accomplishing this goals, lo and behold, we've got the state of the art measurement project that can inform that effort and hopefully be used in a way that accreditation is at its best to inform instructional redesign, not just to be a sham compliance exercise. Yes. Well, what a great coincidence. And Tom, thank you for a question true to your approach to education. Going back to the LMS, we have the awesome James Shulman question from the American Council of Learning Societies. And he asks, from your early work with the learning management system, what do the prospects look like for deriving measures of engagement that can be built into the very software? And what are the politics of the surveillance issues? Yeah, yeah, so very, that's a very, yeah, very good question from James Shulman. And I will take a moment just to out him here. He was formerly at the Mellon Foundation when this project was in early design stages. And so he helped inform kind of our original thinking about this project and engaged the foundation and consideration of it. So I'm very grateful for his prior efforts and his ongoing interest in the project. So our original project goals were to improve measurement. I mentioned around institutional improvement at UCI. I mentioned informing larger social scientific knowledge generation. But another goal of ours always from the start was developing innovative measures that could be disseminated throughout the field of higher education more broadly. So in the case of the LMS data, right? It was a major lift for us to make sense of that clickstream data, right? Every click that a student engages in the LMS system, we had to, we don't get too technical here, but we had to purchase, you know, space on this huge Amazon web service server to park all this data so that you could analyze it in, you know, it would have otherwise just overwhelmed even the data, the computing systems of a major research university. And then you had to have the scientific expertise to code it, make sense of, again, to translate all this big data noise into coherent measures. So we were able to do that at UCI because of a lot of our research capacity at the campus. But every college and university in the country certainly doesn't have that ability. You know, UCI, we're kind of exceptional. It's a major R1 university. It's also, as a dean of the School of Ed there, I'd be remiss to not say that it has a great school of education ranked 15 nationally as of Tuesday this week. It's the best public school of education in the country. Send your potential PhD students our way. So back to the LMS system. It's, we were able to generate this data, but not every campus can do that. The way to do it at scale is there's three majors, LMS providers that control 90% of the market. And so it's very, it would be very low cost for them to take measures that have been scientifically generated and validated in our project because we've got this data. We can triangulate it and make sure we're identifying things that track with other measures of academic engagement and the like. And then build them into the systems and automatically provide them to other colleges and campuses. So that is our goal. We're still, you know, honestly, we're still, you know, we're barely 18 months into data collection. The pandemic, you know, created opportunities for us, but also challenges. So I'm really proud of what the teams generated so far, but we haven't yet gotten to the point of identifying measures that we would recommend be provided for the field as a whole, but we are going to get there. And so I think that's coming. We haven't lost sight of that. Now, James also raised the issue of the surveillance data. Like, what do you mean you're measuring academic engagement in my class?  So the, I'd say two things around that. One is we have to have real conversations about what is the purpose of measurement. We have to say loud and clear, it's not about accountability, right? That's, you know, we know from K-12 that attempts to use measures that are imperfect and limited to enforce accountability on schools and K-12 teachers has been disastrous. It's been counterproductive. It wasn't scientifically justified in the first place. The measures weren't able to do what some well-intended stakeholders thought they were doing enforcing these accountability regimes. So measurement can't be about that. It's got to be about something else. It's got to be about using it to intentionally design programs to better serve students. And so one is, you know, let's be clear about what the goals are and the goals we can share and agree on and reject those that we, that, you know, are not useful ways for it. The other thing I would say is you build into the system safeguards so that individual privacy is maintained. So all the data we're using at UCI, by the time any researcher including myself gets to look at the data, it's been de-identified. There is a person, a literal individual hired for the project that takes the data from the system, merges data together in different strands, but de-identifies it. So by the time a faculty or a grad student or a researcher gets to start working on it, it's not identifiable down the road. And so again, I think there's ways operating procedures to protect privacy, to allay some of the fears of faculty worried about too much surveillance. And then I think we have to be really clear about the purpose and value of the measurement. And then I'll say one more thing, you know, like, you know, we have to put the students, we have to put students first, we have to put the needs of students first. The, that, you know, 50% of students starting college in the U.S. don't graduate in six years. It's probably a little bit closer to 40, 45% today. But come on, there's no other sector in this country that would tolerate that level of underperformance year after year, decade after decade, and not engage in a broad-scale social movement to do better by those, the students. And so part of doing better is to take student experiences seriously and use this measure for good in ways that are, ways that are good and can help us address these kind of problems of student success, retention being one of them, but holistic development and growth, you know, another. Here, here. James, thank you for the great pair of questions. And again, thank you, Richard, for the very, very deep and passionate answer. We have, we have one quick question, clarifying question from an awesome person, and then we want to hear about your findings. So this is from Roxanne Risken, a longtime friend and supporter of the program who says, during the two-year tracking student mood is mentioned. Can you talk about how a student's mood will be tracked and how this information will be used to help students during a two-year study? Hey, great, great, great. Well, that'll get that gets us into some findings. So thanks for that question. So when you use the term mood, you know, I think of our experiential, the best data on that is the experiential sampling, where we ask about emotions, right? What are you doing? Who are you doing it with? What's your emotional state? And, and so your, that data is, you know, really captures mood in real-time lived experience is not retrospective. How did you feel last week? It's how you feel now 50 times. Remember, they're, they're asked 25, one week, 25 another. Interesting. So here's a finding of throwing my first finding for you, for the, for the group. We had the by accident more than design, we, we deployed this experiential sampling method last year in February and there was no pandemic. So we had a people, you know, there was, for people following the news, you know, there was some concern about a looming, but camp in Orange County where we were, there was literally a handful of cases, no, never, no case on campus operations. And so we have people's, what are you doing emotional state? They're all on campus. They're all feeling pretty good hanging out as undergraduate students. And then we asked them again in April, experiential sampling. A lot happened in March. The, the World Health Organization declared a pandemic. President Trump declared a national emergency. The governor of California issued stay-at-home orders, stay-at-wide. The campus essentially moved to remote instruction and told everyone they should go home out of the dorms. So what's their, what was their mood? What was their emotional state after? Hypothesis, right? There were a lot's changed. They're worried, they're concerned. Turns out it was, it was remarkably stable, remarkably stable. We had very little emotional changes in emotional mood before and after.  Well, maybe it was still too early in the pandemic.  Who knows. But, but the, the, the empirical fact is their emotional moods changed very little. Now, the mood is different from, we also track psychological stress, mental health issues with more standard battery in the social, the social surveys they get. Stress does increase during the pandemic. It was particularly elevated in the spring term right after the pandemic's onset when people move to remote instruction. It's, it's, this year it has lowered people's, this is, you know, we've gotten more used to the kind of the realities we're facing. So the stress has been lowered, but not yet lowered to the levels that existed pre pre pandemic. So we, we, we look at stress, mental health, but we also look at it, look at this kind of emotions, which is what Roxanne calls moods. That's great. Roxanne, thank you so much for the great question. And Richard, thank you for sharing that finding. What else, what else can you tell us? What else have you managed to discern? Yeah, so, you know, I'll try to come, I'll try to share some surprises, right? Because that's, you know, so going into the pandemic, we were, we expected, we're worried that not just about student stress, but students' ability to continue to make academic progress. They're being at higher risk of dropping out. You know, colleges and universities have thought for a long time, best way to keep students from dropping out is have them socially enmeshed in college life, residential facilities, extracurricular active, the more socially integrated, socially engaged they are, the less likely they are to be dropped out. So all of a sudden, they're dispersed to their homes. You know, it's hard to do this stuff on Zoom. Many of our students at UCI are 50% of first generation, 85% are from non-white Hispanic, non-white non-Hispanic backgrounds. So we could, many of the communities they lived in were deeply impacted by the pandemic. So really worried and concerned about academic progress and academic engagement, as well as the mental health thing. So one of the big surprises is, in fact, they've done incredibly well academically. The retention, there hasn't been elevated dropout rates at UCI. In fact, the retention, if anything, seems to be improved a little bit. And the grade point averages are up. The credit hours accumulated are up. And our measures of academic engagement, both from surveys and from the LMS system, when we look at the LMS data, we had 15% of the courses at UCI were already online. So when you compare those online courses before and after the pandemic, the academic engagement in the LMS system goes up even in those courses. So students academically are doing well, flourishing and thriving. I don't want to, those seem to flower the descriptions. But we were worried, a lot of our worries turned out not to pan out. Now question is, why did that happen? So here, this is somewhat hypothesis more than empirical findings. Well, the university invested a lot in encouraging faculty to be accommodating and flexible in how they interact with students to move whenever possible to asynchronous formats that are much more convenient for students. Students weren't spending time commuting, trying to get to campus. Students weren't spending a lot of time socializing with friends, which can be developmentally productive. And it can also be developmentally counterproductive, as well as academically distracting. So these are some of our hypotheses about why with our particular type selection of students, in a particular institutional response at UCI, so I don't want to generalize this to schools nationally, we saw this phenomenon. That's enormous. That's really striking. And that actually triggers a connection to another question that comes to us from across the Atlantic. Brian Mulligan, the University of Technology Sligo, asked, what if we find that nothing works really well, but you didn't find that? But also, or that these practices only are affordable by elite institutions? Yeah, well, that seems too pessimistic for my take on things. I think my take on undergraduate education and if you go back to academically adrift, you'll see it there. People might remember that book as being about limited learning and all the students we were underserving. And I think there's about a third of the undergraduate student body in the US that really isn't academically engaged, really poorly served. But there was also in that earlier work a sense of success and flourishing and students that were applying themselves and taking advantage of their education and were academically engaged. And is that easier to do with more resources? Yes, I would say that resources are helpful. But I don't think it's all. I mean, I really don't believe it's only about resources. I think it's about a pedagogical design. So let me give you on pedagogical design. One of the things we're tracked in our in our study pre-post pandemic is the extent to which students were the percentage of time spent in lectures in their classes, right? How much of their classroom time was in lectures? And boy, did that drop going from in person to remote. The remote was much more interactive, much less time listening to lectures, much more time through poor class discussions and, you know, connected to kind of practices we know people call active learning, progressive pedagogical practices, what different terminology around it. But that increased a lot. That's not, you know, that's not just about money. That's about kind of intentionally instructional design. We all as faculty, we all love hearing our own voices. And I say that kind of tongue-in-cheek because I'm talking a lot in this past hour to you. But, you know, there's a big disconnect between what we, you know, with us being interested in our own performances and what students are experiencing as learners. And yeah, I think we can, we can, that central insight has huge implications for how we redesign programs to better meet students and student needs. Richard, that's a great moment to, I'm afraid, pause on because we have reached the end of our hour. And I hesitate to go further than that given everyone's commitment. And of course, your time with the roughly eight jobs that I think you do, that's a great note about our focus on students and about what we can do well. Thank you so much for sharing this extraordinary project and for being so great, grateful as to share your findings right now as it's beginning to advance. What's the best way for everyone to keep up with you and with this project? Yeah, so I don't do Twitter, I'm sorry, I don't have Facebook, but we do have a web page for the project that you were so gracious to put, I believe, in the bottom corner of the screen here. And we're doing our best to keep that updated with findings that we're presenting papers, the media coverage that's around the project and so on. And so that's a great place to go to kind of keep up with things. And also people, of course, can email me and I'll do my best to respond and be responsive and to any questions that came in. Fantastic. Well, thank you so much. Good luck continuing this project. We're really looking forward to seeing what comes from the next. And thank you for being here. Thanks for having me. Thanks for this great forum you're doing, raising all these important issues with this larger audience. Thank you so much. Thank you so much. But don't go away, friends. Let me just point you to where we're heading next. Just to remind you that over the next few weeks, we have some great topics coming up ranging from accreditation to educational technology, to equity by race. If you want to learn more about that, just go to forum.futureofeducation.us. If you'd like to keep talking about this, if you'd like to keep discussing questions like how do we measure student learning? What do we do with the data? Brian Mulligan's question about their applicability. We have many venues to continue this conversation from LinkedIn, Slack, Facebook, and, of course, Twitter. If you'd like to go back into the past and take a look at some of our previous programs to touch on these issues, like last week's session about data analytics or previous sessions about student engagement, pedagogy, the LMS learning design, just head to tinyurl.com slash FTF archive. We have about 249 videos there. And before we go, let me just thank you all for a stack of really good, very thoughtful, very probing questions.  I'm delighted that forum can be this kind of base for all of your thinking together. But even more than all of that, please take care and be safe, all of you. We'll see you online next time. Bye-bye.


Thank you Jonathan and thanks for moderating the moderators have just been excellent throughout the whole conference. Hi everybody, my name is cable green I'm the director of open education at creative Commons I see some familiar faces here in the room. First, before we get started, if everybody could just introduce themselves in the chat, that would be great maybe your name where you work and where you are. And I am in Olympia Washington today at my home and I was telling Jonathan i'm I and all of the other Pacific Northwesters in the United States are happy today because it's not raining and at this time of the year it usually is raining so it's a good day. It's got somebody from Kansas. All right, Natasha's in India welcome Natasha. Where's everybody else from at North Charleston South South Carolina welcome Jessica. Excellent. Most people come in go ahead and introduce yourself oh there they come we got Canada Brazil. Colorado Springs for Jonathan it's one of my favorite places. All right well welcome everybody i'm going to go ahead and share my slides, let me say right up front I will be sharing these slides with you right at the end of the presentation. So you don't have don't don't need to take notes don't need to take screenshots i'll make i'll give you a link and these will all be available. I am going to go ahead and share my screen and ask Jonathan can you see the slides. We're good to go all right I see a thumbs up fabulous. Today we're going to talk about the UNESCO recommendation on we are and specifically what can we all do as open education advocates and people who work in open education to create the resources and the support services and the advocacy and the discussions around this recommendation with. Our national governments with our state and provincial governments and with our education institutions so that's what this session is about today. My information is at the bottom of this slide if you'd like to talk more about this topic i'd love to speak with you. My email is there i'm cable at creative commons.org, and you can also catch me on Twitter at see green C g R E N. Okay, all these slides, unless otherwise noted are under a creative commons attribution license so feel free to reuse them as you see fit. So a couple bit of context here creative commons is now 20 years old we're actually having our 20th anniversary this year. And we've been working on open education for that entire 20 years i've been at CC for just over 10 years, and we've always been in the open education space and will continue to do so. During that 20 years we've worked with many others also working in open education. Today we're going to focus a little bit on UNESCO. And so we've worked with UNESCO and OECD and other idios and national governments and lots and lots of civil society organizations or NGOs around the world for this full 20 years. UNESCO is many of you have probably heard of UNESCO know of UNESCO UNESCO is one of the wings of the United Nations, and UNESCO specifically is the part of the UN that focuses on education, science, and culture. And so when, when the United Nations gets involved in conversations around open education it's led by UNESCO. What's really cool about UNESCO is that they have these various legal instruments that countries around the world the UNESCO members are national governments from around the planet. And they come together and they talk about the big issues that affect not just their individual countries but the world. And then UNESCO has these various legal instruments to to formalize the the wants and desires and the commitments of these countries. And so they have things called conventions that are at the highest level. And so like the convention on human rights would be the highest level of of a recommend of what UNESCO is talking with national governments about and then commitments that national governments are making on the on the low end or something called declarations declarations or more. Here's a here's a topic it's really important, something that we should start to pay attention to. So in 2012 UNESCO actually passed what's called the UNESCO declaration on OER in the middle are something called recommendations, and I've got the detailed script here from UNESCO, but recommendations are legal instruments. Etc etc etc if you look at the very bottom and bold. These recommendations are intended to influence the development of national laws and practices.  They are discussed and debated and written usually over a couple of years, lots of meetings happen around the lead up to passing a recommendation. And these recommendations are exactly what it says here they're intended for a national government that's a UNESCO member to say. Okay, this is a serious topic what national laws do I need to change what policies do I need to put in place. What practices in my country do I need to support.  In 2019, there was the 40th UNESCO General Conference and one of the items on the agenda that was voted on was a UNESCO recommendation on open educational resources. It was unanimously adopted all countries voted yes by the 193 UNESCO Member States and Jonathan, would you please share in the in the chat the link to the UNESCO recommendation so everybody's got that. Thanks. And so this is now of an official recommendation it's been adopted by countries around the world. And then the United States just a quick addendum here. The United States is one of the very few countries in the world that is not a formal UNESCO Member. So long story if you want to learn more about it, you can, you can Google why that is, I'm happy or I'm happy to describe it, but they're not an official member that said United States did actively participate in the conversations and the, the editing and the comments around the lead up to the recommendation. And so they were very involved in this they just didn't have an official vote because they're not an official member of UNESCO at this time. I say that because there's still an opportunity for the United United States to fully participate in the implementation of the recommendation, just like every other country. The recommendation, as we said before, this is a missile a bit more about it here it's a specific UNESCO instrument that gives national governments a list of recommendations and a list of actions for them in this case because this is a recommendation on OER to support open education in their countries, and then it actually calls out and says don't just do this alone. You're working with other countries, as you implement this and that's true of all recommendations. This particular recommendation on OER has five major areas so if you look at the document that Jonathan sent you, you'll see that the recommendation is has these five major sections in it. And that is building the capacity of stakeholders to create access reuse adapt and redistribute OER. So this is all about capacity building professional learning opportunities, getting certified, having, you know, various trainings having conferences and bringing together a lot of people in your country to learn about open education, and why it's important so it's that that awareness raising and then skilling up of lots of people in country. The next area is developing supportive policy for OER. So this is everything from from funders including governments, putting open licensing policies in place, where they say you know whatever's publicly funded is has a creative comments license on it that's quite common we call those open license requirements, foundations and institutes and others oftentimes adopt those. But it's also things like we're going to have a policy to regularly provide funding for universities. We might have a policy to support faculty with release time who want to work on converting a course to OER converting a whole degree program to a to a zero textbook cost degree. It might be a supportive policy to to adjust the rules around promotion and tenure, so that when some when a faculty member or an educator goes for promotion or tenure that they're, if they're publishing and open access journals, and or if they are contributing to the comments that that's viewed as a positive thing and you get more points in your promotion tenure. So there's lots and lots of different types of policy that the recommendation calls out that we should look at. The third area is include encouraging inclusive and equitable quality OER. This really goes right at the conversation of what type of what kind of content. Are we building for whom who gets who gets to be involved in creating that content. Are we truly being inclusive and equitable as we're creating educational resources and then openly licensing licensing them. So this one's absolutely critical and you've seen at this conference, many examples of some really great work happening in this area. The fourth area is nurturing and creation of sustainability models for OER. And this really goes at the primarily at the business models at the flows of money, how policy might interact with this. The commercial sector might have to say about this and what the public institutions that are that support education, new ways of them thinking about how they can ensure that when they use public funds that the resources that they procure or produce or commission are open and I'll talk about that in just a moment. And then the last area is to promote and reinforce international cooperation so there's no need for any country or any state or province, or any education institution to do this alone. The recommendation has a very detailed set of actions, and we can all work on those actions together. So that in a nutshell or the broad areas of the recommendation within each one of these categories you'll see a lot of detail and specific subcategories and actions that governments and others can take to implement the recommendation. So I want to emphasize how important this is this is a real milestone in open education, open education as a field is really only 20 years old, maybe a little bit older depending on who you're talking to. That's not a long time when it comes to politics or laws or the focus of in this case the world's nations. And so this is a, this is a milestone this is something to be celebrated and most important in my opinion this is a unique opportunity to advance open education for all the reasons that we work in it globally. And so some other points here when we're working with national governments or provincial or state governments, it's an opportunity to have scaled impact. It's one thing for me to work with my community college or university to have, you know, promotion and tenure policies that favor the production and use and adoption of open education. It's another deal when the Ministry of Education in Belize says from now on, everything that we fund and procure for educational resources will be openly licensed it's just a different level of scale. And we can with this instrument with this recommendation, we can have new conversations with governments because the governments have signed on to this thing the government's built this thing that we couldn't have before. And so that's, that's very exciting. Those of you who know me or heard me speak in the past know that I'm very keen on thinking about public funding and how what the rules are on public funding of educational resources of research in science etc. And this is this is my firm belief that publicly funded resources should be openly licensed when the public pays for something the public should have access to what it paid for. And we shouldn't restrict those resources just to one country, if the public in one country paid for something. Yes, they should all have access and there's no reason not to let the rest of the world have access as well. One of the new ideas. So you know this is an older idea we've been working on this for a long time this should not be new to anybody. This is an idea that that I started to put forth maybe four or five years ago, and whereas open policy is very top down so it's you know the funder could be a foundation or a government saying if you take this money. What you produce in your grant or your contract will be openly licensed that's that's an open policy usually, whereas open procurement is really bottom up. So open procurement might be a school district buying educational resources or it might be a university's math department that maybe Jonathan works in and that math department saying. Okay for entry level pre calculus textbooks and educational resources let's have a conversation as a faculty and decide what we are going to use so what are we going to assign our students. And so the idea of open procurement is at the point that you still control the money. So if you're a, if you're a school district in primary or secondary schools.  What open procurement says is, don't do the traditional thing, which is, I'm going to give a bunch of money to the to the market probably to a for profit publisher. I'm not going to sell me educational resources at a high price and the model used to be that I got paper, and then I kept that paper for 1012 years and amortize the cost over that long period of time. And then I bought new resources, well that was problematic because resources got old and stale and students couldn't keep the resources, and the books ripped and you couldn't update them and it was a bit of a mess. The new model that publishers are putting out is a digital lease so now you don't even own what you buy, you lease digital access to the resource for for the duration of your lease and then when you stop paying, you lose access which is also bad right this is what we should be doing with digital resources over procurement says if you are procuring if you're buying resources, that's fine, build by commission you know pay money for what you need. If you can build an in house, even better, but it's okay to have somebody else build it for you or buy what they have. But number two, ensure that when you procure or buy something that you own what you buy. Right so I don't like I don't lease cars I go buy a car and then I own it and because I own my car I can sell it or I can loan it to somebody or I can let my kid drive it when they get their licenses I have all these things I can do with it so if you're going to buy or commission copyrighted resources, make sure you, the public education institution owns the copyright to what you buy, and then hopefully you'll have an internal open policy that says you're going to share and CC license, what you own. I'm trying to put this in a catchy phrase, you know, build or buy what you need, own what you buy and share what you own, and that's the idea of open procurement. I bring this up because we wove this into the recommendation and suggested that that that countries and education institutions and districts of schools and primary and secondary schools, take a good hard look at how they spend their money and ask themselves is there more a more effective efficient way of procuring and owning and sharing educational resources that is not beholden to the market forces which are taking us down a path that are not very healthy. Okay, so that's a lot about the recommendation. Now, what's what's happening like who's working on this is anybody taking advantage of this milestone and starting to put together some help for governments. And the answer is, yes, and we're just getting started. There's a long way to go. So there are at least three collaboratives that creative Commons works with and not just us many other organizations are part of these as well. The first one is the UNESCO oer dynamic coalition so Jonathan if you could share that link that would be great. This is a group that's led by UNESCO and specifically led by the UNESCO staff which, which facilitated the whole creation and passing of the recommendation. So this is a really important group. What they've done so far is they've hosted webinars with experts from all around the world on the different areas of the recommendation. And there's a lot of sharing of ideas going on and challenges and how we can collectively overcome those challenges. And so that's one group. The second group, we call the network of open NGOs. This is lots of different non governmental organizations or civil society organizations that work in the open education space, who have gotten together and said look, all of us are willing to do something to support governments and in education institutions around the recommendation. Why don't we work together. Why don't we coordinate our efforts why don't we pool our resources so that we can, we can have a more united set of services and and communicate, you know hopefully with with one voice to governments that we're working with. So this is a great group they meet once a month. It's facilitated by by OE global they do a great job pulling us all together and keeping us on track and and we are Africa group or we are Africa and Neil Butcher and associates as part of that and they're leading us now in an in an effort to build case studies. So we're talking about short one to two page or case studies that are easily digestible by government officials to demonstrate the success of OER. So all of us have, you know, ideas and philosophies about why open education is good and we've got anecdotes we've got data from oftentimes our own countries or our own institutions. And that's great. And governments, you talk to them about that stuff and they say show me the data. And so we're putting together research backed case studies of various OER successes from all around the world. What we found with this group was, we had a lot of ideas, but what we really needed to do was to stop, pick something focus and do that one thing. And then we're going to move on to the to the next thing. That's coming along nicely. The third group is a group that that I facilitate it's called the creative commons open education platform. Think of this as a working group for open education topics that are global in nature. So this is a big group we've got over 1100 people from 94 countries and it's growing all the time. You're all welcome to join this it is free to join. Jonathan, could you drop those three links in for me please. The first link, Jonathan will share is a link to the open education platforms web page on the CC website. There's information about how to join if you have any problem joining just send me an email and I will add you to the list. We meet every few months we also have a slack channel and various forms of communication. The next link Jonathan will share is our 2021 work plan in the platform. And the reason that I want to share that is that one of the areas of work that the platform members decided that they wanted to work on was helping governments implement the UNESCO recommendation on we are so you'll see some more detail in there. And then the third link that Jonathan will share is a grid that we're going to use in just a minute. We're going to have some interactivity and work together. There are other possible actions to to operationalize or help governments implement the recommendation on we are and so this this grid is multi column thing that Jonathan's sharing the column on the far left is the recommendation. So that's actually the copy and paste text right out of the recommendation. The next column over is what can governments do I think that's what it is I don't have an in front of me. So what could governments do so what are the specific actions that we think governments can take that fulfill those parts of the recommendation, and then the other column is what can institutions do and here we're talking about education institutions, universities, colleges, schools, and other formal education institutions. So let's go back to that document in just a moment. So as I said please feel free to sign up and join the CC open ed platform we'd love to have you it is open it's free. All are welcome to join. Okay, so now we're going to come back to that last doc that three column document that Jonathan shared. And what I want to do is I'm going to stop talking, we're going to open it up for questions first, and then we're going to engage in this activity together for the remaining time. And that is what I'd like to do is have us all think about how can we work in our countries with our governments, either at a national level or a state or provincial level, and or your education institutions that you work with. And help them implement parts of the UNESCO recommendation on OER. When you look at the recommendation, it is huge there's a lot of stuff to do in there. We can't do it all at once with any government or any institution and they may never implement the entire thing. And that's okay. The question that we're asking at this early stage of implementation is, what can we start with what's a natural fit for your institution or for your government. What is their opportunity. What skills do you have what connections do you have that you might be able to help your government or your education institution make some progress on implementing parts of the recommendation. And or what might you produce so as you can see in these groups what we're starting to do is to produce resources that advocates will need as they're engaged in the support services. And these studies that we're building with the network of open NGOs, those are going to be licensed CC by and we're going to share and publish them openly, and anybody in the world can take those translate them modify them and use them in their advocacy efforts. And we're going to be, we're going to keep building in these groups more and more of those openly licensed assets that advocates can use. So that's another area where you as individuals or you as groups of open education leaders can contribute as well. And so, so I'm going to stop sharing for just a moment actually let me just share my contact information here again cable at creative commas.org or at see green on Twitter. I'm going to stop sharing and then we're going to come back to this document in just a minute. There we go. And so at this point. Jonathan were there any questions along the way that we should circle back to in the chat that people have already asked. I think there's only one Jody Bailey asks, says I subscribe to the CC open email list. Does that mean I'm already a member of the CC open education platform. Yes, that is the email list. Okay. At this point there's really three ways that people can can chime in one is feel free to unmute yourself and turn on your video and and ask your question or just suggest how you might brainstorm or how you might contribute and and or how you might help your government or your education institution. If you've got an idea, please share it. If you have a question about how to do it let's talk about it. Hopefully if you you can also chat in the chat window and I'll keep an eye on that. And we can also use this Google document everybody has comment permissions for it. And feel free to look through it and add comments about where you think you might be able to contribute. So I'll stop talking the floor is open. And then you feel free if anybody wants to ask a question or talk about how you might contribute just unmute yourself and turn your video on if you'd like. Hi, my name is Elena McNabb and I work at the College of DuPage as an adjunct faculty manager. I'm sitting at this list, and I'm seeing a couple things just because I specifically work with adjunct faculty here in our institution and since that our institution is a community college it's a lot of our resources end up being public anyways. So I'm just thinking about how, like our library can incorporate. And I guess, advertise it to the community but then also just some things that I can do within my own department would be to organize a workshop which we do, but make it focused on oh we are, but then also I like the idea of installing oh we are we're doing adjunct faculty boards, which is a great incentive for faculty. And eventually, you know we can start with adjunct faculty and move it to full time and incorporate those two groups but yeah I thought that was a really great opportunity to develop that within my own department. Those are all really great ideas and thanks Elena for for sharing. I think the first section of the recommendation right it's all about building capacity building awareness. There's a ton of good ideas in there for, you know, for an institution that's maybe just getting started, or you're in the early stages of building.  Yes to everything you said, and you know you might see some other things in this first section that might be useful as you're looking to build the capacity and the skills at your institution. Another quick plug that I'll make is that Creative Commons also has something called the CC certificate, which is a 10 week online course, we also do it as a one week boot camp. And it's an opportunity for people to really learn at a deep level about copyright the public domain CC licenses, open education, a bit of advocacy, more and more about open pedagogy we're weaving into the curriculum. And so if that's something that's of interest, Jonathan maybe you could drop the link in for the certificate page that'd be great. We also just, there's a there is a fee to that the it's a $500 course fee for the 10 week course these are facilitated Jonathan's actually one of our long term long time facilitators. These are really great courses people love taking them they. But thanks Jonathan for putting the link in. We also just struck a deal with the regional compacts in the United States to provide a 15% discount off that price, which is new. So if your institution is a member of SRE be your witchy your neppy or any of the regional compacts which it probably is, then it's not $500 it's whatever $485 it's with that discount. But that's another I bring it up Elena because that's another way to build capacity at your institution is you might want to get two or three people CC certified at your institution. And then you've got some some real experts in open licensing who can serve as resources to other faculty and instructional designers etc. Yeah, no that's great and I can already think of a couple people that I can plug in there to see if they'd be willing to do that so yeah thank you. Okay. Oh, I see a question here. Has there been any pushback against the recommendation at any stage, maybe from commercial publishers or do you anticipate future pushback. Yeah, there was a lot of pushback at every stage and they're. And then what's happening right now is different than pushback but I'll describe it. So yes, when the recommendation was being worked on. And by the way, UNESCO is on track to pass a recommendation on open science this November at the next general conference which is awesome, and we hope in the future that UNESCO will have a open culture, which is kind of its third area that it focuses on. And so we'll be nudging suggesting helping UNESCO had that direction as well because open glam or galleries libraries archives and museums is another space that creative Commons works and we've got a program around that as well. So to answer Jonathan's question did we have pushback. Yeah, there was there was pushback from. There's pushback from publishers who don't like, you know, rarely like discussion about open education, full stop. Because you know the creation adoption of open educational resources usually means less market share for for them to sell expensive, all rights reserved copyrighted materials. And so they don't like that and they raise their voice. We also had, you know, there was, I would say, quite a bit of lack of awareness and understanding by several countries around the world, as the recommendation was being debated. And so you I'm sure you've all encountered this as well as you've had conversations about copyright and open licensing and and we are with with people that you've worked with.  That's not the case right you're the beauty of CC licensing as you keep your copyright, and you're adding a license for the public that provides some, some rights and some permissions, but the author still retains her or his copyright. And so there was that confusion. And so there had to be some education of those representatives along the way. And so of course debate about the specific language countries, when it comes to that these UNESCO instruments countries of course like to have full authority to make decisions in their countries about their countries. And they're always very careful about ceding any authority to outside entities including international governmental organizations like UNESCO. You know there was a big debate about whether or not countries would be required to do these things or if countries would be even required to report every few years about the progress they were making against the recommendation. And so you'll see language in it which is maybe a little bit softer than some of us would have liked where it's encouraging countries to do these things and it's recommending that countries report publicly on their progress. Honestly, that's the way of international agreements right you there's a lot of compromise to get these things across the line and to get them passed. And those those compromises have to be made. And that's, that's not abnormal at all all international big international agreements encounter that kind of discussion and debate. And I, and I believe that to be a healthy thing that's that's the way diplomacy works. What's happening now is different. So this thing is now passed. And, of course, it passed in late 2019. And we all know that shortly thereafter we were all blessed with a global pandemic. And so, as you might imagine, national governments around the world are a bit distracted at the moment. And as you imagine, and then as a whole. And then if you look at their ministries of education or their national departments of education. They are also very focused on a specific set of challenges, right so as schools closed. There was this massive move to online learning. And in many countries, including the United States and Canada where most of the conference participants are from, you know, there were a lot of teachers and a lot of both in primary and secondary education but also in tertiary and university education that were not well prepared for that. There were a lot of universities and schools that did not have the technology in place and there was you know, if you watch zoom stock right that we're using right now zoom stock went way up because there was a massive purchasing of that and other technologies to go online. And so that that grabbed the attention, as you might imagine of ministries of education around the world. And also, a lot of concern right now and a lot of effort, looking at kids that and I'll say kids I've got two kids who are in primary and secondary school, who, when their schools closed, they went home. Many in many many countries around the world they do not have good internet access or in many cases even stable electricity they certainly don't have one to one laptop programs in many countries. And so closing school didn't mean moving online closing school men closing school like I don't go to school anymore. And so that was a known factor we knew that was going to happen the problem is is that as schools are reopening. A lot of those students don't come back. They've been put into labor situations where they're now working and helping to support their families because coven took a hit, or the parents took a hit and their employment and their ability to earn a living and so there's just lots of challenges and problems right now in the education space globally that are drawing the attention. And so, as you might imagine things like the UNESCO recommendation on OER which is, it's not mandatory. This is a, you know, an optional thing that governments have signed on to that they've lost focus a little bit on that. And rightfully so, right we got to make sure that students can safely get back into school. That being said, there's also, so that's the, that's the down major downside for education around the pandemic we know we hope that that will get better as vaccines continue to spread as in the United States anyway they're about to approve vaccines for five to 12 year olds I believe that's coming in the coming days. But that's, we all know that globally we've got a real problem with vaccines is incredibly inequitable, what's happening in terms of vaccine distribution. I think of all the vaccines given in the world, most of the countries in Africa have, you know, have received something like 2% as compared of their population have been vaccinated as compared to many countries in the global north which are, you know, 60 70 80% of those are actually inequity problems there that affect students in education that are going to have to be worked out. The silver lining which is interesting if you saw there was a presentation at this conference about about, what's his name, Bayview analytics that does surveys every year about OER. So that's the question just looking at the United States. What happened in terms of the adoption and use of open education resources when courses went massively online. And the answer was not a lot like we didn't get a big bump in OER, but to be determined. And so they're kind of keeping an eye on it. And so, thanks Jonathan gave me the one minute warning. So that'll be interesting to watch. But that is an opportunity for we advocates that work in open education that as, as schools engage more and more in digital because they have to go online or they're having more hybrid. That that is an opportunity to talk about OER raise awareness and help them adopt open educational resources instead of all rights reserved expensive commercial resources which they probably have to lease and not own. I'll stop there and I'll take my last 20 seconds and say two things. One is, if you'd like to talk more feel free to reach out to me anytime. I'm happy to connect you with these groups I'm happy to work with you if you've got ideas. Second, if you want to add ideas directly to this document you've got comment level access feel free to throw your ideas on here and our questions and I'll review those and reach out to you. And then last if anybody does want to stay after this Jonathan's going to close the recording, but if anybody wants to stay and talk for a minute I'm happy to do so. Back to you Jonathan. Yes, thank you so much we I'm going to stop recording now.


Prophet Isa A.S. was this symbol of humbleness and asceticism. No man in that time and in many parts of history was less materialistic than Prophet Isa A.S. He was the least materialistic man on earth because he knew how to prepare for the year after. He knew what's good for us, what's bad for us. In a beautiful hadith which Imam Ali Ibn Abi Talib A.S. narrates from Prophet Isa. He says Isa A.S. once told his companions, his people, he gave them a beautiful examples that farmers or those people who have a nice garden, you know the meaning of this example. He gave a beautiful example. He says the materialistic aspect of this life is like the weed that grows in your garden. Have you seen those weeds, right?  He says that's the likeness of this materialistic world. When you become attached to this materialistic world, it's like weed that spreads, that flourishes in your garden, in your farm and it starts destroying your crops, your plants, the beautiful turf that you have. That's the likeness of this world. Once you allow your heart to submit to the materialistic aspect of this world, that's it. A point will come in which you cannot do anything. Like when the weed takes over your garden, first of all, it's very difficult to uproot the weed, right? To get rid of it.  There is no point. And that's what happens to the heart of the human being. When you allow the materialistic world to captivate your heart, that's what happens. And you lose that spirituality.  Has no appetite for food. You'll put the most delicious types of plates in front of this person, but this person has no appetite. Why? Because this person is ill. This person is sick. That's why. Otherwise, the food is delicious. It's delicious. On a normal day, this person would be so happy to be eating that food. But when you're sick, you lose that appetite. Prophet Isa A.S., he says the one who falls in love with this world, what happens? He'll be struck with a type of spiritual disease. What happens? He has no appetite to worship Allah. He has no appetite to repent. He has no appetite to prepare for the Akhara, for the hereafter. Just like when you're sick, you have no appetite for food. The same happens when you become inflicted with the disease of this materialistic world. You have no appetite. Some people sometimes wonder, say it. I'm trying to pray. I'm trying to do dua. I'm just not interacting with it.  I need to make room for spirituality. Start by trying to remove the materialistic aspect from your heart. Then you'll make room for spirituality. Then you'll make room for Allah SWT. Because Allah says in one hadith, I am the best partner. If you give me 99% and you give the world 1%, I relinquish all of my 99%. I relinquish my share. The heart is for Allah SWT. Hence, another hadith says, Al-Qalb-u-Haramullah. The heart is the sanctuary of Allah. It's a sacred place for Allah SWT. Therefore, do not allow anything besides Allah to sit in your heart. These are the lessons that we learn from Prophet Isa AS. Otherwise, look at the world.  People go to war with each other because of what? A brother will turn against his own brother and stop talking to him because of what? Because of this materialistic world. In one hadith, Prophet Isa AS was walking with three of his companions. They stopped by three blocks of gold. They were there in the desert. Prophet Isa looked at the gold. He noticed that it caught their attention. He told them, you see, these three pieces of gold, they caused people to die. They kill. This gold has the power to kill. Then they left. Those three went back to that gold. They all wanted it, right? So what did they do? Two of them conspired. They said to the third one, go and get us some food. And they decided when he comes back, we'll kill him and we get the gold to ourselves and we don't have to share it with him. Well, he went. The shaitan came to him. What did he do? He's like, let me poison the food and get rid of those two and I'll get all the gold to myself. So he brings the food. He poisons it. He gives it to them. They kill him. They attack him. They kill him. He dies. They eat the food.  All of them die and none of them benefited from that gold. Prophet Isa AS walks back. He sees them dead. With the permission of Allah, he revives them. He brings them back to life. As the Quran says, he would bring people back to life.   This is the lesson that we learn, brothers and sisters from Prophet Isa AS.


What's happening guys so as some of you know I got pneumonia recently and I have to say that that was not a fun experience. 2 out of 10 would not recommend but anyways I'm finally starting to feel normal again and yes I've seen your comments saying that I look tired but anyways let's talk about college majors and the only degrees worth going to college for. Now I've already made a few videos on this subject but I just keep getting more requests and if you're new to my channel then welcome and if you're coming back to my channel then welcome back and you might have seen some of my other videos on the subject and basically if you look on YouTube most business or career related channels say that you shouldn't go to college and it's a complete scam. Then if you ask your parents or your grandparents they're probably going to tell you that college is the golden ticket to success and just like with most things in life the truth lies somewhere in the middle and in my videos I always try to teach you guys how to find out for yourself and all I ask in return is that you Hulk smash the like button. Smash. And as long as you always remember to do that I'll keep on making good content and replying to all of my comments but in this one I'm just going to straight up name the degrees that are worth going to college for and then towards the end of the video I'm going to show you a comprehensive list so that you can go and check for yourself. So first of all we're going to start off by assuming that if you're thinking of going to college you only want to go for four years and you're going for a bachelor's degree and later on I'll talk about how if you decide to go to grad school get your master's or doctorate there's some degrees where that's a good choice but for many of them it isn't. Now average wage right now in the United States is around 48 to $52,000 a year and this is the overall average of all people those with degrees and those without. So I think it's safe to say that you want to aim for a degree that makes at the very least $48,000 a year especially assuming that you're going to have to pay back a bunch of student loan debt in order to get that degree. So for instance according to a recent pay scale study gender studies degrees are only making about $42,000 a year and that ranks $772 out of 835 possible degrees. And also remember that this is all self-report data so it doesn't count the ones that weren't able to get a job as many of them aren't. So I think it would be safe here to conclude that getting this degree is not a good financial decision as the average person is going to be going nearly $40,000 in debt. So this is the graph that was made based off of the data in that study and it's really great to look at you should probably pause it and just check out the graph really quick and this is a really great representation for some of the most common majors. But if you want to look through the whole study all 835 majors I'll link that down in the description remind me if I forget but overall when you're doing research and you're looking at this data you want to make sure that you're making at least $48,000 a year with the degree that you're getting. Now next we're going to look at unemployment rates as well as future growth. Now this one is really tricky but overall just make sure that you can actually get a job after you graduate. Now the average unemployment rate right now is about 3.5%. So if you look at statistics from the NCES and you see that your major has an unemployment rate that's greater than 3.5% that could potentially be a bad sign. Then another thing you can do is get on the Bureau of Labor Statistics or BLS and look at the future job growth potential for your particular career that you're looking to get into. So if your dream is to be a meter maid I have some bad news for you. So again this isn't an absolute but you want to be very wary of something that has a negative average growth over the next 10 years because that means that the job is probably dying. Now the next thing you want to think about is whether you actually need that degree in order to get into whatever job you're going for. And a great example for this one is a criminal justice degree. For most law enforcement jobs you actually don't need this degree and there's only a few that require it. So make sure that the job that you want actually requires a degree in order to get into it or else you're just wasting your time. That would be like paying for a course on financial education when you can just watch my videos for free. Now one exception that I hear a lot about this is if you're going for a graduate degree. And there are examples of bachelor's degrees where if you just get the four-year undergrad degree there's not that many job opportunities but if you bump it up to a master's or a doctorate there is a lot of opportunity out there. One really good example of this one is data science where there's not that many opportunities for those with a four-year degree but if you get a master's there's actually a ton of job opportunities out there. But whenever you're thinking about this just make sure that you realize when you go to grad school first of all it's a lot more expensive and the interest you have to pay on those loans is a lot higher plus you have to start paying it right away. So if you do decide to pursue a degree where you're thinking of going to grad school afterwards it's even more important that you plan ahead and make sure that the career or lifestyle that you're going for is going to lend to you getting that degree. Now all of these are great points of course but go ahead and check out the study for yourself. I'll link it down in the description and then check out my videos right here because I made them just for you. Go ahead smash the like button hit the subscribe button ring the little notification bell comment down below any ideas you have on the video thank you so much for watching and bye for now


There are two prominent techniques for representing text documents numerically, bag of words, and document embedding. Now today we'll be talking about embeddings. So, just as a quick recap of our previous videos, we've learned that embeddings are numeric representations of words, and that after embedding, each word corresponds to a location in the embedding space. There, words with similar meanings, such as mom and dad, will lie close together, and unrelated words, such as car and butter, will lie far apart. Okay, now let's get started by setting up some data. We can use the data sets widget to load the BBC3 data set. This is just a collection of BBC news articles. Then we pass it through the corpus widget to ensure we use the text from both the title and the content. Now, let's see what we have in the corpus viewer. Okay, the first few articles are from the business section, with topics varying from jobs in the US to dollar values and greenspans. Now, I also want to take a quick look at the boxplot widget to check out the distribution of document classes. There are about 500 articles on business and sports, and just short of 400 on entertainment. Okay, now we'll embed the documents with the document embedding widget, just like we've embedded words in the past. Here, there are two embedding models to choose from. Espert and Fast Text. Now, Espert is a sentence-based transformer model that works for over 50 languages.  So, since we're now working with more extensive texts, not just individual words, we'll use the default Espert. Also, a nice feature about Espert is that it doesn't require any pre-processing, it just handles entire documents. The data is now embedded, meaning each document has a vector representation, and we can examine those representations in a data table. There they are, each document described by 384 features. Now, let's try to classify the documents using their embeddings. Remember, classification means we want to predict a class of data instances from its description. So, in our case, we have some embeddings that describe our documents, and the classification task is given these embeddings to predict a document category. So, we will use the set of 1,400 embedded documents to train a logistic regression model, and then feed it to the predictions widget. Then, I can go to the BBC website and copy a part of some article on sports, and place the text in a new Create Corpus widget. And just for good measure, I can take another piece, this time on business, and make a new entry in Create Corpus. Okay, I'll have to embed these two articles as well. Then, I can feed the resulting embeddings into the predictions widget. And, there we go. Both documents are correctly classified as sports and business, respectively. Now, since we use logistic regression, we can even inspect the classification model in Anomagrap. Hmm, this doesn't look very informative. The business category seems to be represented by features with odd names, like Dimension 73 and Dimension 322. Now, this is a document embedding issue. While text embeddings can certainly represent text, interpreting them can be difficult. All we get from document embedding is a set of numbers, to which, at least at this stage, it's very hard to associate any meaning. Let me show you a quick way of characterizing a group of documents with associated words. We can project all the documents into two dimensions, using something like T-SNE. In the T-SNE visualization, each document from our data is represented by a single point. The documents with similar embeddings and therefore similar semantics should appear nearby. So, using T-SNE, we should be able to uncover groups of similar documents. Now, let's color the documents according to category. Great, we can already see how the clusters represent different news categories, but there is some overlap. For example, business and entertainment are pretty close and some business documents lean far towards the sports. Still, we can attempt to explain the documents in the T-SNE plot. I'll feed the output of T-SNE to the annotated corpus map and make sure all the data is passed through by rewiring the connection. Now, to find document groups, I'll use a Gaussian mixture model. Annotated corpus map determines clusters in the visualization and labels them with keywords. These keywords tell me that the sports section, for example, mentions England and Wales a lot. The entertainment news covers the best films and music, and the business news tends to talk about prices, percentages, and the economy. Now, I have no idea why I is specific to sports. I would need to investigate this further or just instruct Orange to remove particular words from document characterization. Now, coincidentally, we will be learning this stuff in future videos when we cover bag of words. Document embedding is a great technique to extract numeric representations of documents. It enables us to use any machine learning technique downstream from clustering to classification. But its main downside is that it's difficult to interpret. But as we saw with the help of some interactive maps, like T-SNE, we can even try to alleviate some of those issues.


Welcome folks, we have the Dow Industrial Straighten Up 75, Nasdaq Stuff 2, S&P's Up 7.5, Gold Contract Up 5.70, Straighten Up 2,004 and Outs. We have Silver Down 15 cents, $24.80 an oz, Light Sweet Crude, Up a Buck 48, $77.13, a barrel, notes and bonds. A 10-year note, Up 2 ticks, Straighten 1207, the 30-year Up 4 at $126.10 and King Dollar, $King Dollar, up $196 ticks, Straighten 101.076, Euro at 111 Yen at 141 British Pound at $128-1 U.S. Dollar. Now we're going to have out here, we have Option Expiration, you also have the rebalancing of the Nasdaq, okay, because the bottom line is that everything's going to be rebalanced as of the close today, so you can expect some real action coming into the close. And if we go over to the NQs, we take a look at the NQs first, the NQs do have a high volume low from this morning that I expect we're going to go after. So let's take a look at this baby, because it's going to get so interesting, man. So whoops, not that far, here we go, there we go. Where's that volume? Put this up again, okay, so what you're going to see right here, that low that's established out there at that $549, we're at $574, $549 has volume, and there's no volume up above, okay, so we're going to be solidating all day long, so that's telling me we're going to go after that at the close. We go over to the E-minis and you get the same setup inside the E-minis. We take a look at this, and what you're going to see, there wasn't a huge amount of volume on the way down, but that is still your high volume bar, it's laying out there like a sore thumb, that's $49,000 on the E-minis, that's $4561, so that's $74 right now. So it's going to get intriguing coming into the close, no doubt. The dollar did get the bounce, and the bottom line looks to me like the dollar has a little more to go on the upside because what you have out here is that if we take a look at this and we just put this on a weekly and you line this up as to where ICE is, you know, you're laying out about, there's not too much action anymore, if it gets inside the range it's going to be interesting. You got 101, basically 500, we're at 101, 08 right now. Stay right there folks, come right back and expect some volatility, and I mean down the volatility coming into the close out here. We have the Dow up 65, NASDAQ down three, S&P's up seven and a half, we'll come right back.


The Appalachian Advantage plan is the simpler, better way to get the phone you actually want instead of paying hundreds of dollars up front.  It's called the Appalachian Advantage available at Appalachian Wireless. The Paintsville Veterans Referral Center will be holding their 13th annual Veterans Day Telethon at the Paintsville Recreation Center on Sunday, November 6 at 12 p.m. It is free to come up there and join the crowd, join the fun. We have some entertainment as you walk in. The Veterans Referral Center relies almost solely on donations and the Veterans Day Telethon is their biggest fundraising event. In order to kind of keep the lights on for our center, we have to do fundraising and this is our biggest event for it.  The Veterans Referral Center 13th annual Veterans Day Telethon will host many talented musicians, dancers and more. The event is free to attend. The numbers to call to donate during the Telethon will be posted to the Veterans Referral Center Facebook page or you can call<PHONE_NUMBER> to donate. For Mountaintop News, I'm Kelsey Dean.