At this time, it's my pleasure to introduce Dr. Victor Benassi, Faculty Director, Center for Excellence in Innovation and Teaching and Learning at the University of New Hampshire. Dr. Benassi graduated from the City University of New York with a Ph.D. in Psychology. He has been a faculty member at the University of New Hampshire since 1982. He has a professor of psychology in the Department of Psychology and with a professor of college teaching in the graduate school. During the early 2000s, Dr. Benassi served in the Office of Academic Affairs as Vice Provost for Undergraduate Studies. He is currently Faculty Director of the Center for Excellence in Innovation and Teaching and Learning. Please join me in welcoming Dr. Benassi to the Naval War College. Good morning, everyone. It's so great to be here. I am looking forward to our time together and I've already had a full day of experience of being shown around the college by Tom Gibbons. Again, thank you so much for being here. This is the third involvement I've had with the Naval War College about speaking at this event. Three years ago, Tom asked me if I would be able to come and I had already been booked and recommended someone who came and then last year he invited me to come and I had already been booked and someone else came. And I told him last year that if he asked me early enough, so he did in March. So thanks, Tom. These slides are already available at the University of New Hampshire Center for Excellence in Teaching and Learning website. So if there's anything on here that you want to be able to access or link up to, you'll be able to do it there. We're also videotaping this presentation that will be available. So here are the people we thank. The Davis Educational Foundation from out of Maine provides support for the science of learning of work that we do at UNH plus the Provost Office. And here's a few of my collaborators. So in thinking about presenting to this group today, I had a particular challenge. And that is that nearly all of the work that we've done over the last decade in science of learning has focused, not entirely, but it's focused on undergraduate education. And the kinds of learning outcomes that one is interested, particularly at lower level courses in college and university, are different than the kinds of learning outcomes that you folks are interested in the courses that you teach at the graduate level. So that got me to think about how we can try to match up the kind of learning outcome that we're trying to produce with what we, mainly psychologists, know about how human beings learn and how people learn in academic settings, which then we think auto inform the kind of instruction that you do. So linking the kind of knowledge with the learning processes involved informs the instructional methods. And of course it's an iterative process. So let's pull this together. So the kind of learning informs the kind of instruction and learning processes which auto inform the kind of specific interventions that we use in our courses. So at the most basic level, we want people to learn facts. Of course, the people that come to your courses already know a lot of facts about a lot of things and a lot of the topics that you're teaching them about. But my understanding is that the students that come here come from many different backgrounds, from different countries, from different areas of emphasis at different levels in their career and of course in some areas they know more and some areas they know less. So although we're not primarily interested in fact learning at the graduate level, of course it's important. So what we know is that principles about spacing of practice and learning opportunities and trying to engage students in retrieval of information that they've studied is important learning processes. And that any kind of activities that prompt retrieval, for example, quizzing can promote fact learning. The person that I recommended to come here last year, Nate Cornell from Williams College. I chatted with him briefly before coming here today. And he told me that one of the topics that he focused on was the so-called testing effect in the role of retrieval practice. He's a world leader in that area. So anything that we can do to get students to engage in retrieval will promote learning. We think of quizzing and testing as a way to assess student learning and it is, but it's also a powerful tool in promoting learning. Moving up the hierarchy a little bit, you know, many of the outcomes that we're interested in graduate education is teaching students to make discriminations, to be able to classify information, to be able to put things into categories. I'm going to give an example of this later. What we know in terms of a memory learning process is that the, what we call the interleaving of study opportunities is a very potent and powerful approach to promoting learning of things that go into different categories, things that get classified here versus there. I'm a beginning student of the Battle of Gettysburg and one of the things that I had to learn early on was how the Army of the North and the Army of the South were in fact put together. My friend used to laugh at me all the time when I'd get regiments and brigades mixed up and so on. So I had to learn to make those classifications. The interleaving of practice is a powerful tool in doing that and I'll give an example of where you can use content review and quizzing to promote interleaving. What we're interested in here primarily is higher order thinking, sometimes generally called critical thinking. So what we're interested in is our students learning general principles, making generalizations from specifics. We're interested in creating a mental model about whatever it is we're studying. So across your courses, students are learning many mental models about how things are organized, how people are organized, how events are organized, how they interrelate with one another, creating a model of the operation of whatever it is we're studying. We're also, of course, critically interested in problem solving. So I'll talk about two memory processes, one involving what's called self-explanation. And the other, which is a type of collaboration. I've studied a little bit about your seminar structures across your courses. And I believe and hope that the collaboration approach that I tell you can apply to any of the seminars that you offer. So we'll talk about using self-explanation prompts and an approach called team-based learning. So the facts. I'm not gonna give you an example of a study about the facts. I mean, your speaker talked a lot about that last year. There are literally hundreds of studies that have been done and dozens and dozens of studies that have been done in real academic classrooms that show the benefit of spacing of practice and retrieval practice. So in any event that actually generates the retrieval, like quizzing can produce positive effects. So I'll just cite two chapters from a book that myself and two colleagues edited a few years ago. One, Shana Carpenter has a chapter on spacing and interleaving of study. And Mary Pike and colleagues have one on what's called test enhanced learning. The testing effect. Both of those appear in the book that I'll show you later. Then any of you can access, for free, the American Psychological Association, Division Two, Teaching of Psychology, publishes books about a whole variety of issues related to teaching and learning. And this book appears there and any of you can access it for free. So let's move up the hierarchy and talk about discrimination classification. So in looking at your theater security decision making course, what I see is a number, all your syllabi are great by the way. They're models, I wanna show them to UNH faculty as great examples of how to structure the connection between what we want students to know, what we do, and how we assess it. So theater security decision making, I think a lot of political science people were involved in that program. And so you talk about the levels of analysis approach, different kinds of analyses. In psychology, we do this as well. We talk about the biological basis of behavior, the social basis of behavior, the organizational or community basis. And those different levels of analysis require certain kinds of classifications and discriminations that are made. So I hope that what I talk to you about, you'll see it as applicable. So interleaving of practice, and retrieval, and content review. So unfortunately, I've not done any work with faculty here at the War College. So I can't give you an example of the studies I'm going to talk about that would be directly from this kind of approach and content that you teach here. But I can tell you that there is a growing body of literature across numerous academic fields and disciplines that show that these principles, if appropriately applied, work. By work, I mean they improve the learning outcome that we're trying to achieve. So the example I'm going to give you is from a course in statistical reasoning. This is a course that undergraduates at UNH, in fact, across the world take in learning about statistical reasoning and the appropriate statistical tests to use to analyze data from certain kinds of research designs. So what we did, in a statistics course, we've done this study maybe five times. So I'm going to show you results from one study, but we've done it about five times. So in a statistics course, this is the typical way that the course proceeds. Is that a statistical test is covered. So what does that mean? It means that students read some material, typically from a textbook. And then they go to class, or it's online, they go to class. And the instructor provides further instruction. And usually in statistics, like many sort of quantitative oriented courses, provide some work solutions of how to do that test or when that test is appropriate. What we do is after that work occurs, and we've done it with face-to-face courses and totally online courses. We just finished a totally online course that did this approach. After it's done, what we do is we have students go to our learning management system. Currently, we're using Canvas, like you. We just finished using Blackboard for several decades. As they go to this online module within Canvas, and they read about a particular statistical test, the one that's being covered at that point. They read about it. The critical point about statistical test is you need to know what assumptions are required in order to apply that test to a particular research design. They learn that. They go through the module and review it. And then they take a quiz on it. So if it's the first statistical test that's being covered, what they do is they take a test that covers that statistical test, end of it. What we do is we do experiments, studies in our courses. The way whenever we do a science of learning intervention, and if we just did it, we'd get an outcome. But of course the question is, compared to what? If we did this and we get an outcome, does it mean anything as if we didn't do that at all, or if we did something different? So what we do is we do these studies in which we have control conditions within the studies. So in the non-interleaving condition, what students do is they go through a module on a statistical test. They go online. They learn the material. They study the material. They take a quiz. Then they go to the next test in the course. They do the same thing.  But the quiz now asks questions about the current statistical test that you're studying. But it also asks you questions about the test that you already covered. And you can see that if you go through NRK's eight statistical test, what you do is in the non-interleaving condition is you cover a test, you move on. You cover a test, you move on, et cetera. In the interleaving, you cover a test, you move on. And now you're assessed on this new one, but you're also assessed on the one that you already covered. That's why we call it revisiting the past. And we repeat this in our case through eight statistical tests. These are the eight statistical tests, if you care. Any statisticians in here? All right. Look familiar? All right. So our experimental manipulation is within a course. So we had about, well, we had 38. Danny Rasko was the instructor of this course. We had the interleaving and non-interleaving condition. So I'll show you a figure here. On the vertical axis there is how students performed on a comprehensive test that was given at the end of the course. And that comprehensive test had 24 research designs, 24 of them. There are eight statistical tests. Guess what? There were three questions about each test, all scrambled up. And what students had to do is read the description of the research design. And then choose among the eight statistical tests which one was the appropriate one for that research design. So that axis there shows what percent they got right. If anybody's interested, I've got confidence intervals around those means. That's the non-interleaving condition. What we always do, of course, is save the best for last. And so this is how they performed in the interleaving condition. So what we had here was about a nine point difference in performance on the final exam. Now again, nine points, 10 points, 10 percentage points. Is that a lot? Well, I mean, if you're a student, I know maybe less so in graduate education, but in undergraduate education, would you rather have a 70 or a 79? But that's in terms of grades and so on. But in terms of learning, if you think about using this approach and accumulating it over many instructional opportunities in many courses, I mean, we haven't done this. I'm not even sure how we would do it. But to look at the net effect of this over, say, an undergraduate or graduate education, it could be substantial. Not so much in terms of the grade students get, but in what, in fact, they've learned. So how might you incorporate this principle into your course? Courses that you teach. I mean, I think it should be straightforward. And it could be done in a relatively easy fashion. So what I'm reading in your syllabi is students do this work outside of class. It involves usually reading papers, articles. There's usually some kind of what I might call learning assurance or readiness assurance where they post the discussion post, maybe are required to comment on other students. And then they come to the course, to the seminar, and then the discussions and activities that occur in class are informed by what they've already learned outside of class. It would be very easy that once the course gets moving along, that you spend a short period of time at the beginning of the class asking questions about material that was covered before. It's a very straightforward, simple recommendation. I have other recommendations, but I don't have time. And you could think about how you might be able to revisit the past and what you do. If you want to do an assessment of this, what you would be required to do is at the end of the seminar, I noticed that I think in all of the courses that I looked at the syllabi, there was an exam at the end of the course. If not all of them, I saw it frequently. They had papers that required, but an exam at the end of the course. That exam, you could do an assessment by, in fact, asking questions about the concepts that were covered earlier, and if you wanted to do a study, an experiment, you could do this across different offerings of the course to see if you bump up performance. So moving up the hierarchy, principles, mental models, problem solving. In the strategy and policy course, one of the learning outcomes that I saw was skilled in applying naval perspective through the use of analytical frameworks. I mean, that sounds like critical thinking to me. It involves mental models. It definitely involves problem solving and learning of general principles. So what I want to do is talk about two approaches that I believe would be directly applicable to any of the seminars that you offer here, whether they're the required courses or the electives. One involves self-explanation and the use of self-explanation prompts, and the other involves a type of collaborative learning called team-based learning. So making sense and meaning of new information. Again, the studies are at the undergraduate level. Thomas told me that many people involved with the work college have backgrounds in STEM fields. We're doing most all of our work now for the last two years in STEM education, particularly in the biological sciences. So this is just one course in which we've done studies in using self-explanation. So self-explanation. Self-explanation is a type of constructive learning strategy.  They're listening to other students talking in a seminar. The constructive part of it is, is they're not just reading it, not just taking it in through say visual auditory senses, but as they're reading, listening to the material, they're actually engaging in a constructive process. And so you'll see I think in a second what I mean by that constructive process. Self-explanation involves self-monitoring. Psychologists call this metacognition. I'm sure you've heard that phrase probably more times than you want. And what this self-monitoring involves is as you're reading new material, you're reviewing it in relation to what you already know, to your prior knowledge. That's critical for developing critical thinking skills, integrating what you're learning now with what you already know. But also when you're reading or listening to material, you're generating questions for yourself because you may be thinking, some of you may be thinking there's, I didn't really get what he was talking about this being a constructive process. So you're engaging in that kind of thing. So what happens is as you engage with new material and we do things to get you to engage in self-explanation, you're putting this together in an organized fashion. The mechanism that is involved in this from the psychological point of view is that self-explanation is an excellent tool for helping you to identify you being the student, the student to identify gaps in your learning. In other words, when you're reading this new text, there are going to be concepts and points that are not crystal clear to you that do not fit with what you already believe. In other words, you have a mental model. You always have a mental model. Might be very shallow and new, but you start off with a mental model of something based on prior experience and self-explanation helps you identify gaps in your learning and it helps you the process to modify your flawed existing mental models. All right, so how does this work in practice? So in this biological science course, what we do is we have students read and what they're reading in this course is our textbooks chapters. And so this textbook chapter is broken up. You just get a mental image of textbooks, right? There's a chapter on mitosis, but within it there are sections, right? Different sections. What we do is, before it was Blackboard, you could easily do this in Blackboard. Now it's with Canvas. Is what we do is when students read a particular section of a chapter, they then switch over to the Canvas platform and we have some questions that we ask them. And they, we have them type their answers. Sometimes self-explanation is just talking out loud. That is, people may be reading and just talking out loud to themselves or if it's a laboratory study, somebody's tape recording it. But what we did was we had them type answers to these questions, which I'll show you as they're reading along. So they read, they answer these questions, they go to the next section of the chapter, they read. In your case it could be they read the introduction to an article and they get some of these questions. They go to the next section on an analysis of the problem. They ask some questions, what does this mean in terms of the bigger significance or the application of these principles to whatever your topic is. So what are the kinds of prompts? I mean, there are many different prompts that could be used. These are three prompts that we've used. We've done others as well. What, so they read the section in the biology textbook. So we ask them, what information is new? What have you learned in this section? How do the new ideas work with what you already know? So now we're trying to get them to connect with prior knowledge. And then we always ask them two I wonder questions. In other words, now that you read this, what are you wondering about? Are we wondering like how this mitosis stuff is gonna work with, what's gonna come next or think of your own courses and what some of those I wonder. These are fun to read because students write all kinds of interesting things like, I wonder when this is going to end. They don't all do that. But one of the reasons, by the way, that we have them type these up is that we currently, literally this summer, for this course, we are actually going in and doing content analysis of the quality of their responses to these. And what we find is, in fact, the higher quality responses are associated with better learning outcomes later on. Surprise, surprise. All right, so what we do, again, an experiment, we're gonna randomly assign about half of these students to a self-explanation group and the other to a summary group. So the self-explanation group is just what I showed you. They read the sections and they answer these questions. In the summary group, what we do is they read the same exact material section. Once they get done, they go to the Canvas site and the question that they're asked is, or it's a statement is, please summarize what was covered in this section. Now, we could have had some students not do anything, but what we wanna do is we don't wanna be able to just to say, students do better when they write stuff down and it really doesn't matter what they write down. We wanna be able to say that it's the self-explanation process. So what we need is a control condition that's something more than not doing anything. We actually pick summarization because it's not a bad strategy. It's not useless. It's actually pretty good. So the point I'm making is that if we get a benefit above summarization, which research has shown actually is pretty good, then we're showing that we can actually improve performance. See, one of the things, see if I were sitting out there in the audience is, I'm thinking, why don't you know the answer of what he's gonna show us because he wouldn't have just gone through all that if it isn't gonna work, right? Not gonna set us up and not have any result. So again, on the vertical axis there is, in this study, they had three chapters of the textbook and then they had an exam. And then on that exam, they were asked conceptual questions related to the material was covered in those chapters and this is how they performed in the summarization group, about 60%. By the way, we have about a five year record of how students do on the first exam in this course. This is one of the things we've been trying to improve and it's about 60%. It's a difficult course and a lot of students, they've come out of high school, they've had high school biology, they get thrown into this course. These are science majors, STEM majors and it's a challenge for them. They get about 60% right. So again, now what are we gonna see? What we're gonna see here is that we see about a 5% bump. Now statistically, if we do a t-test for independent groups on this, it's a significant difference. It's 5%, not as big as what I showed you before which was almost 10 for the other but I gave you a little heads up about why we might expect that this result wouldn't be bigger. What Mickey Chi, she's the person that developed this approach is what she found in many of the studies she did under very controlled circumstances is that she could get a 20 to 30% bump in performance from self-explanation. I mean huge, I mean anybody would take that. Here, our students are doing this activity on their own. There's a lot of noise, I mean in many ways, they're at home, wherever home is, they're doing this and who knows what else they're doing. So there's that piece and what we see is in the quality of the self-explanations, if the explanations are not very good, the learning outcome is not as good but what we also find is if the learning outcomes, if the explanation is stronger, that the learning outcome is stronger. What Mickey Chi has also shown is that if you instruct students on how to do self-explanation, you get a bigger bump. The study that we're going to do in the fall, starting in a couple weeks is exactly that study. We're gonna compare this condition here with one in which students are pre-trained to use self-explanation. A lot of the studies we do, we wanna see whether the interventions are as effective for students with different skill levels and we've done two kinds of skill levels. One is we look at say how they performed earlier in the course or we look at their SAT scores upon entering the university. And so in this case, what we did is the students in the class, we did a split just to show you visually. So students who scored above the median on the SAT verbal in the high SAT group, students who are below the median for the class were in the low. This is relative low and high. Just split at the median, midpoint. What we wanted to see is whether the interventions were as effective at those two different levels. Let me give you a little general statement here. We've now done this kind of analysis but also looking at how they did on prior exams in the course. What we find almost always is that the lower skill students benefit more from the intervention than the higher. So one way to talk about that is the highest performing students, highest skilled students, they've figured out ways to do their study. I mean, how do we know that? Because they're high performing students. So what we find is that the interventions for those students tend not to be as potent but the low skill they do. So here are the people in our high SAT group and voila, what we see is that there's virtually no difference in performance between the two groups for the people that are in the low SAT group. What we see, the blue there on the left is that they perform significantly higher than the ones that are in the summary group. And in fact, if you look across the students that are in the lowest, they're performing as high as the high SAT group. We see this pattern of results time and time again. So how might you incorporate this kind of principle into your class? I mean, it would be very, very straightforward. Again, I know that, I think maybe Tom and talking with you is that you have students do their reading outside of class and they then do discussion posts. So one way to do self-explanation that could improve the quality of those discussion posts, which we would believe would improve the quality of discussion and activity in class, is that as students are reading these articles, is that there would be these appropriate self-explanation prompts that are given after different sections of the material. It would be very easy if one were inclined to do a study where you could actually assess the impact of these interventions on the discussion posts themselves and on the quality of the discussion in class and if there's that comprehensive exam at the end there. Anybody's interested in this kind of stuff, I'm the guy who likes to work with people for fun doing this kind of thing. All right, the last approach is team-based learning. This is an approach that actually developed not independent of, but somewhat parallel with the growing body of research and science of learning. In fact, Michelson, who developed this approach back in the 1970s at the University of Oklahoma, is well ahead of the science of learning people getting involved in education. That is, the science of learning people have been around since the 1800s with Herman Ebbinghaus and not learning nonsense syllables. I mean, we know a lot about how human beings learn, but the cognitive scientist really got into how to apply science of learning and education really when the federal grant money at NSF dried up for basic research and then all the cognitive psychologists became cognitive aging as the Institute for Health went into that area. That money dried up and now through the Institute of Educational Sciences is money for doing education. So Michelson's work is completely compatible with and in fact, we won't do it here so much, but we could contextualize all of the components of this approach within a science of learning framework. I picked it because this approach is particularly relevant and appropriate for graduate education. In fact, although there are a number of studies that are done have been published on this approach at the undergraduate level, most of them have actually been in graduate education. So what we do is we develop what I'll call permanent teams and a permanent team is a team that lasts as long as a project lasts that you're gonna do in a course. Project could be what we're gonna do next week in the class. The project can be we're gonna cover this topic in the course for the next three weeks or it could be a course long project that you start at the beginning and it culminates at the end with a paper or presentation or both or some other product. So when I say permanent, I mean permanent with respect to whatever the projects are that you're going to work on. So Michelson talks about readiness assurance and what readiness assurance involves is some kind of pre-training. So pre-training could be that out of class students watch a lecture capture, the teacher talks and presents information. It could be some kind of professionally made video. I'm gonna give you access to some videos on science of learning at the end of the talk here today. It could be reading material. You all have reading material for your seminars. People have actually identified that this next component is important as a way to boost performance. After the material is studied, read, is there some kind of assessment? So it could be a quiz, an actual quiz. It could be a discussion post. It could be both. It could be coming into class and the teacher asking questions randomly to different students. So what you're trying to do is you're trying to increase the probability that students do what it is you want them to do is read the material in a deep way. But it also is, as I said, giving these quizzes is more than assessment. It actually boosts learning. I mean, that's the big point about the so-called testing effect. And retrieving material improves learning. We could do these assessments individually. Students do them individually and they can get feedback. What Michelson does is at the beginning of a class session, he gives them an individual quiz. And after they're done, then their team gets together and they go through the quiz collectively and argue and discuss about what the right answers are and then they get feedback at the end of that. He's done, and others have done studies showing both of these interventions boost the quality of student learning. So application activities. What could you do? What kind of activities? What kind of projects? Case studies, of course, are everywhere here at the War College. Problem solving activities. Of course, you have a big focus on gaming and gaming scenarios. I mean, I don't know a lot about that and my understanding is I probably won't get into that building over there. But I know enough about the kinds of conceptual requirements and the involvement in teamwork and the involvement in competing ideas and how do you structure things so that somebody who is a dominant individual, may well be a dominant individual, but may not have the best ideas doesn't overtake the process and so on. So lesson learned activities and looking at your syllabi, I see references to Napoleon. I see references to the Civil War and I can imagine that with the kinds of readings and discussions that are going on there, which are lessons learned. And as we were talking about, I go to Gettysburg a lot and I see groups coming down from Carlisle from the Army War College that are doing a lessons learned tour with their teacher. So each team member has a role to play. And again, these slides are available. I don't expect you to digest all this. I just want you to see that this isn't, you just put groups of people together and that's the end of it. What I'm trying to get at is that each member of the team has a defined role. This approach here is based on what's called problem-based learning, very similar to team-based learning and there's a reference there if you wanna learn more about it. So a scribe, a tutor, a chair and the members of the team. Michelson says, I'm not sure how much research behind this, but teams of around six to eight are ideal. If your seminars are about 12 to 16 or so people, you could think of two teams within a course. So what kind of problems do we deal with in team-based learning? The problem, what kind of problems? Well, the best kind of problems to be worked on are the ones that you work on. The ones where they're open-ended with no clear, defined solution. In other words, you can't go look up what the right answer is. Is critically, the problems build on prior knowledge. So if you do a project that say last three class periods is that everything you do across there builds on what happened before and builds on your prior knowledge. Collaborative work to find solutions. I have a lot of background in addition to cognition in social psychology. And this is a social psychologist's dream to study small groups in this kind of situation. And of course, we're people and we have personalities and we have approaches and some of us are less dominant, more dominant. And so developing a structure where collaboration works is a critical part. And if you're interested in this, you'll need to go to Michelson or someone else to learn exactly about how to structure things so that you have an effective group. I mentioned this before, is these kind of problems create cognitive conflict. And the cognitive conflict could be in terms of conflicting with what your prior beliefs are. We all enter all kinds of areas having very firm beliefs about the way things are in that field or discipline. And in fact, the more we know, the more we have those beliefs. But your beliefs and your beliefs may not be the same and on a same academic topic. So this could create cognitive conflict. Team-based learning has an approach which in fact can allow you to take advantage of that conflict which actually results in deeper learning. There's not a single right answer. I mean, when you do gaming, right?  And of course, the critical component of this approach is to try to promote reflective thought or what we generally refer to as critical thinking. So the team-based process involves preparation and that there are clear learning objectives. Your learning objectives in your syllabi are outstanding. You have usually some in-class pre-training where the instructor sets up the problem that's going to be studied over whatever the period of time. And then there's out-of-class exploration which could be doing many scenarios. That somebody is a team member, you're assigned to read that article, you're assigned to read that article and that has to be sent on Thursday night to this person right here who's going to integrate it so that we come back to class next Tuesday. This is what's going to happen. That's where the chair of the committee comes in. All right, the application process. In class, we have this group work that occurs on the project. The teacher has usually, not usually, always developed a well-defined plan for what's going to happen. So this is what's called a learner-centered approach, but it's an instructor-generated approach. So this is not an instructor sitting back saying, meet as a group and solve it. And critically, and this falls in closely with all the science of learning at work, is that assessment is absolutely a necessary component. The assessment approach could involve these items as well as others. There's usually, at the end of the project, a group debriefing where the members of the team usually are almost always led by the chair of the committee, does a debriefing, not only about what solutions were generated, but also about the team process as it worked for the group. Now, if you're gonna have a series of these projects across the term, and you keep the groups across, is that group debriefing could critically inform improvements in the group process moving forward. Usually, in team-based learning, there's a team project which could be a team-based paper or a team-based presentation. There's also peer evaluation and self-evaluation. And that that peer evaluation that occurs from other members of your group is shared with the members of each member of the team to help improve their process as group members. Again, this is critically important moving forward. Now, you have a final exam in your courses. You have, usually there, the students write two or three major papers in your courses. Those could be the ultimate learning outcomes. That is, by engaging in this process, even though it's a team outcome, you can still have the individual accountability in terms of the major paper that's written and or the final exam. So our in-class group work and our out-of-class exploration, they go hand in hand. Most of these projects last over several weeks of a course if not the whole semester. I'm gonna give you one quick example here from UNH. Again, not in an area taught here, but it is a graduate level course, has a few undergraduates in it. About 60 students, pretty large graduate course. Beth Stewart was the faculty member. And I'm not gonna go into details for the purpose of time of the study, but basically what we did was we had four projects in this course. And what we, the researchers, did was we randomly selected two of them to be team focused and two of them solo. So what happened is, teacher would introduce a project in class, clear set of instructions, students on their own would go do their readiness assurance. They would come to class, they would take a quiz. We would score the quiz, they would get feedback. If it was a solo project, then what students got was the worksheet that had all the tasks to be performed. And then each student individually worked through the worksheet. When the worksheet was done, that was submitted. In the team-based approach, so let's see, there were say 60 students in this class, there were about 10 groups of about six each in the class, then they would come for the two projects that were team, they would come to class, they would take their quiz, then they would get into their predefined groups, and they would get their worksheet, and then collectively that team of six would work through it, produce the product, and one report would be handed in. So that's the approach. What we were interested in is on a major exam, how did students perform on conceptual questions that were related to the material covered in those four modules, all right? And so here's what we found. Is in for the solo, right? And by the way, these are the same students now, two of them were solo, two of them were team, and the solo, it's about 78 or 79% correct on this midterm exam, and there we go on the team. There was a benefit of about seven points, and that's pretty good. Again, these are graduate students, the level of performance tends to be somewhat higher, but there's no ceiling effect here, so this is something that I think holds up. We've done a study like this in maybe four or five classes so far, and this is what we get overall. If any of you are experimental in design, research design people, any one study like this is potentially problematic because there are alternative explanations that might be for this or that. What we do is we manipulate different things in different studies so that collectively, if they all show the same kind of result, we have greater confidence. In other words, this is not a laboratory study, but it does show a benefit, and we're not the only ones that have shown this. Other people have shown it as well, all right? So how might you incorporate this?  What you need to do is simply to think about how you would structure a group activity in a systematic way and that has these in-class and out-of-class components to it. Not, wouldn't be that difficult to do. Again, if anybody is interested in this, I'd be happy to communicate with you over the phone or by email or whatever to give you some ideas. We're doing projects all the time now with faculty at UNH using this approach, so it would be pretty easy to connect with you if you have an interest. And I think in the kind of seminar structure you do, it would be relatively straightforward, but also you could see good learning outcomes. So here's a bibliography that you're not gonna write that down now, but you can get these slides. Michelson's book with Knight and Fink up there, 2004, it's still a classic, but there have been lots of more recent work that's done, including in graduate education. Just to conclude, do you already do this kind of thing in your course? Are you sitting there thinking, well, well, that's nice science of learning. People said that. I've been doing that for years. If that's the case, that's really good. I'm impressed. When I started doing this kind of work about a decade ago, I thought, that makes sense. I do kind of that. But I also know that a lot of what I did, hopefully less now than before, a lot of what I did was not very well informed by science of learning. And to be sure, when I was a student, the kinds of activities that I engaged in as a student would be like I'd be a poster child for how not to do it, right? I mean, just one example here. You read something, you don't understand it. What do you do? You reread it. What do you do when you still don't understand it? You reread it. What the research shows is unequivocally is that rereading material has limited benefits. Mark McDaniel had published a study a few years ago where they did about 15 experiments pretty much across a whole bunch of designs. And they found in one of them rereading had a benefit. In one case, rereading had a decline effect and all the others had made no difference. And so, but none of you reread, I know that, but I'm engaging in my confessional here. You can incorporate these evidence-based principles informed by the science of learning. I hope you see in a relatively straightforward manner. I mean, for some things, having the benefit of working with somebody from a teaching and learning center or somebody that knows about this kind of work, especially if you wanna do an experimental design to really tease apart what the effects are due to. But by and large, you can do these sorts of things. And I'm gonna give you some references. And you can expect strong learning outcomes. So I've only scratched the surface here. There is so much exciting work being done in science of learning over the last five or 10 years that if you had a serious interest in it, like I do, you could spend a lot of time learning about it and still only scratch the surface. But not all of you, maybe not any of you are interested in doing that, but you do have an interest in your students' learning and there's some things that you could access, some sources that would be benefit. So I mentioned before the book that we did, make no money on this. This is free at the APA website, Society for Teaching with Psychology. What we did, this is my co-editors, Catherine Overson and Chris Hackler, what we did is we asked about 30 of the world's leading science of learning experts, would you like to write a chapter, not for the other 15 people that read your stuff, but for faculty that really don't know anything about this stuff?  And guess what? Every single one of them said yes. So this book has the people that are doing the really ground-baking work, and they all wrote a section at the end of it about how can the everyday teacher use these principles in their course. So the book is like three years old, but still quite up to date. Make it stick. Some of you might have, this is a very popular book. Henry Rotiger and Mark McDaniel are the science of learning people. Turns out that Paul Brown is the brother-in-law of Henry Rotiger, and he's a novelist. And over Thanksgiving dinner, they were talking about what they did, and they decided, Brown said I can write a book that people will actually buy, and that's it right there. And then my favorite person in science of learning is Richard Mayer from UC Santa Barbara. This is a very nice, thin book that's accessible to students, but also to faculty. It's not a technical book. Some user-friendly websites. There's one here called Retrieval Practice by Pooja Argawa, who was a doctoral student of Henry Rotiger's. The learning scientists, these are, I think, two faculty from UMass Lowell, and here's their website. This is an excellent general site. Megan Smith and Yana Weinstein. And what about some videos? Deep Learning. That's Mickey Chi. I mentioned Mickey Chi before. She's the self-explanation person. She's got a video where she talks to us, not to the other technical people. And there you can access that. Henry Rotiger has how people learn, very accessible, and I did a presentation two years ago now that focused on teaching, learning, and technology. I know you have your distance education component. I could have talked about a dozen or more studies that we've done in using technology to promote learning based on science of learning. So if you're interested in that, you can go to that YouTube video there and watch that as well. So thanks very much.


All right, everybody. It's 340, so I'm going to get started. How are you doing? My name's Jason Fritcher. I'm a principal infrastructure engineer in Semantics Cloud Platform Engineering Group. And I'm currently involved in the middle of rolling out Barbican for a lot of services we're using in our OpenStack Cloud. And just wanted to share with how we're using it. So I've worked in the internet industry for almost 20 years, mostly in development and operation roles running internet-facing services. I've always had an interest in security and kept an eye towards developing trends in the security field and have been one of the people that other people on my team come to, asking for advice and reviews and whatnot.  This is the agenda I've put together. So for those in the audience who may not be familiar with Barbican, I'm going to give a quick five, maybe 10-minute introduction to it. Beyond that, I'm going to discuss what kind of problems we have at Semantic and how we're using Barbican to resolve those problems. I'm going to move on then to talking a little bit about what we've done to improve the out-of-the-box configuration of Barbican and make it a little more secure. And then I'm going to cover some improvements that Semantic's working on in Barbican. And then if things go well, we'll have about 10 minutes for Q&A. So an introduction. Back in the old days, this is what a Barbican was, was a defensive structure outside of a castle or city, and was in a very effective defense until about the 15th century, where more advanced siege tactics and artillery became prevalent, and they kind of fell out of use. Today, Barbican's arrest API designed for the secure storage provisioning and management of secrets, such as passwords, encryption keys, and TLS key pairs. Barbican basically has three types of resources that it manages. You have secrets, which actually contain the data that you want to protect. These are typically small bits of arbitrary data. Barbican really doesn't care what you put into it, but out-of-the-box has a 10 kilobyte limit on what you can store in there. Really, you don't want to use this as a large encrypted object store, because it's really not meant for that. If you have large objects you want to encrypt, encrypt them, throw them into Swift, and then store the key in Barbican. Barbican also has containers. These are basically logical groupings of secrets. So if you store a TLS key pair in there, the container will have a reference to a secret with the private key and a reference to a secret with the reference to the certificate.  The last resource type it has are orders. These are more longer running tasks, like when you ask Barbican to generate a key for you. Asymmetric keys can take a long time if your system doesn't have a lot of entropy on it. And you really don't want a client hanging around waiting for that request to complete, which could potentially take minutes. So you submit an order, tell Barbican what you want, and then you can pull Barbican to find out when it's done and then get the result back. There's two basic components to Barbican. There are API processes, which are basically the front door to the system. All client interaction to Barbican occurs through API processes. These processes also completely handle all of your requests for secret and container requests.  And when you make requests for orders, the API processes will take those requests, stuff them in the database, and then send the request off to an asynchronous worker for further processing, which brings us to the second type, which are the worker processes. They do the longer running tasks behind the scenes, and then when they're done, they update the database with the results so the API nodes can see them. And again, those longer running tasks are asymmetric key generation. It does symmetric key generation. And it's also how certificate management is implemented. Here's some potential dependencies that you can have with Barbican. If you want to do user authentication or authorization out of the box, you have a keystone plug-in. If you need something more scalable than SQLite, which is the default configured database, you can interface with Postgres or MySQL or any of the other popular databases in OpenStack. If you have standalone worker processes, you need a message queue for the API nodes to talk to the worker processes. In OpenStack case, that's RabbitMQ. And then if you want true security in your system, you need hardware security modules. And these basically provide a secure environment for cryptographic operations and key storage. So moving on to how we're using it, basically, some of the problems we're having is, I'm sure, like a lot of organizations, you'll find in your version control system you have secrets stored in there, be it credentials to access systems for scripts to run or API keys for API services, or any number of other secrets that you might find in programmer code. In addition to that, you have issues of, how do you start TLS keys? How many of you here have a central machine that you go to generate your certificates and key pairs on? And then that's the only copy of it you have other than what's in service? Or a developer who generates that stuff on their local machine and that's the original copy. Barbecue can help solve that. Another problem you might see are encryption keys. If you're doing per object encryption, how do you handle the keys? Where do you put them to securely store them until you need to use those objects again? On top of all those problems, automation is the key word of the day. How do you enable automation in your deployments and deploy secrets like service account passwords and stuff like that in a machine accessible form so a human doesn't have to go and put them in? And finally, it'd be nice to be able to automate more of the certificate management lifecycle. The capability is not there today, but it'd be nice to have a system where you store a TLS key pair and the system notices, hey, this expires in two weeks, how about I go renew it? And then after getting a signed certificate back, it goes and notifies the service, hey, I have a new key for you, and then it's deployed automatically. The use cases we're looking for here is our LBAS team really wants to use Barbecue for TLS provisioning. You want to be able to have other semantic customers come in through some interface say, hey, I need an LBAS instance. It needs to support SSL. And here's the certificate and key rep for it. And then the LBAS instance can go and stash the key pair and certificate in Barbecue provision the LBAS instance, and then that fetches the keys for provisioning there. This also enables future scaling in that if you need to scale up your LBAS instances, the key's already there in Barbecue to provision. We also have in our configuration management system tying back to version control, service account passwords in there and API keys we'd like to get out. We'd like to pull stuff out of our puppet manifests and chef recipes to stuff that into Barbecue and then have a plug into those systems so that at runtime they can go and fetch those secrets for deployment on the systems. And finally, we're looking at deploying key management as a service for different product groups within Symantec. They've been really interested in this service because there are products that really want HSM backed security, but they don't want to have to manage HSMs themselves in the data center, HSMs being hardware security modules.  For how we've deployed things, basically we've got a Calera MySQL DB cluster that is doing all of our data storage needs. This is really nice because it gives us a multi-master topology where we don't care what database node we write our data to. We just write it and then the clustering software takes care of replicating it around to all the other cluster members. This gives us really good replication support and it also enables us to do cross data center replications so that we can have one consistent Barbican view across data centers. This is particularly important because one of the things we're looking at doing is if we hit an issue with Barbican in a data center, we can just fail it to another data center and the client shouldn't notice anything other than some increased latency. We've got a rabbit MQ cluster for handling all of our messaging needs. We're using SafeNet Luna hardware security modules for the root of our security trust. I'll get into more on those later. And for the actual API nodes, we're basically running Barbican in a UWSGI container with Apache and Mod Proxy UWSGI in front of it. We wanted Apache up there so that we had a convenient place to insert filters into the request pipeline. We also offload all of the TLS processing to Apache itself. One of the things we're considering doing is inserting mod security into that pipeline and writing some web app firewall rules for Barbican so that we can, at the Apache layer, detect bad requests and reject them without ever passing the request back to Barbican. This also provided a nice point to insert metrics emissions. So we generate metrics for Barbican transactions at that layer and we didn't have to go dig into the Barbican code to enable metrics emissions. So moving on to some of the hardening stuff we've done. One of the first things I looked at was how to secure database access.  No offense to any MySQL people in the audience. Enabling TLS within MySQL is pretty easy. On the server side, you just insert the SSL key and SSL cert directives into your My.com file. After this, MySQL will advertise that it has TLS support to clients. If you want to ensure that TLS is always used, add a require SSL directive to your user account and then the server side will reject connections that don't come in over TLS. On the server side, activating TLS is as simple as adding the SSLCA parameter to your SQL connection string. This enables the client to do validation of the MySQL certificates. But as I said, there's challenges here. Basically, MySQL does not implement intermediate certificates properly. So if you throw an intermediate certificate into the PEM file on the server side, MySQL won't read it and it won't send the intermediate to the client, so you won't get a proper certificate chain on the client side for validation. You need to look at adding your intermediate certs to the CA PEM so that the client can generate a trust path. In addition, in the MySQL client libraries, their hostname validation is extremely poor and naive. I ran into an issue deploying it where Semantic being a CA. We use EV certs for everything and the subject DN in an EV cert is longer than a buffer that the client uses for parsing. Basically, it expands the DN into a text string and then searches it for CN equals and extracts the hostname after it. We overflowed that and truncated it and hostname validation failed, so it's something to watch out for. And also, depending on what your mix of OpenSSL versions are, you could run into issues with Diffie-Helman key sizes. So a recent version of OpenSSL added code to reject all connections that use Diffie-Helman keys below a certain size, which MySQL does. And if you run into this, basically, the only solution you have to fix it is to specify a cipher that doesn't use Diffie-Helman for key exchange. It sucks, but that's the way it is. So moving on from there, we did some work on hardening the Barbecon nodes. With the type of data that Barbecon has, you really wanna restrict access, the pool of people who have access to these machines to as small of a pool as possible, to narrow the pool of people who have access and you can actually vet the people who have access to all this potentially sensitive data. Depending on your security needs, it could be as simple as just restricting SSH access. If that's all you need, then in the SSHD config page, the allow users and groups and deny users and groups directives should be sufficient. If you're looking for broader restrictions, like say you don't trust your data center personnel and don't want them logging into these machines on the console, then you could look at the PAM list file or PAM access modules to enable system-wide restrictions, including console and x-axis and anything that uses PAM to authenticate. Automation brings challenges as well. If you use automated systems like Puppeter Chef, then that opens up the scope of who has access to your nodes to anyone who has access to modify your Puppeter Chef code. If you don't have fine-grained controls on who can commit changes into those repositories, you may need to just disable automation all together, which is the route I took. If you're not using community packages, be aware of file system permissions. Really, you want all of the Barbican code and configuration files to be owned by some other user than the user that Barbican runs as. Really, the only things Barbican should have right access to are a log directory and attempt directory. This is to prevent issues where if someone manages to get Barbican to execute arbitrary code, they can't go in there and change Barbican to allow greater access. With read-only access to its own code, an attacker can't do anything to it. And finally, something to look at is host-based firewalling. I would really advise to implement IP tables to restrict access to the Barbican service to just your load balancers and your monitoring systems. This way, if someone happens to break into your network on an adjacent node, they can't attack Barbican from that node, and you force them to go through established paths to get to Barbican, where hopefully monitoring or intrusion detection systems or something like that will pick up their activity. Now we get to hardware security modules. So for cryptographic security, these things are like the gold standard. You're not going to find much better than these in the civilian market. Hardware security modules are specialized devices. They provide a environment that is tamper-resistant and tamper-evident, and it's an environment you can have secure storage of key material in and perform cryptographic operations in. And it's extremely difficult for an attacker to remove secrets from these things that they're not authorized to get. You start tampering with these, and usually they'll either freeze the cryptographic modules until you reboot them in the case of minor attempts or just outright erase themselves in more drastic attempts. They come in different shapes and sizes all the way from USB thumbstick-sized processors, which mainly just hold keys and maybe give a little bit of on-chip operations to PCI cards that you add to your servers, which can give you much better performance, to network appliances, which are basically hardened servers with PCI cards in them for the secure processing. One thing to note is performance does vary greatly between these devices, and when it comes to just raw symmetric performance, like the type that Barbican uses, general-purpose CPUs will beat it for performance, but those general-purpose CPUs don't give you the security environment that the HSMs will. One thing to keep in mind is these are not easily jumped into. There are challenges here. I've run into this myself with my production deployment in that data center personnel who are not familiar with these things are very hesitant to put them in their environment. I've lost two or three months of time fighting with our data center people trying to get my devices put in. Finally, they caved, but it was a horrible fight to get them in there. Depending on what your security risks are for your production devices, you may need to build a physically secure environment within your data center. In our case, we are, because our data centers have access to third-party vendors, contractors and such, and it would be a shame for someone who had data center access to be able to walk in, pull a hard drive out of my Barbican box, and have credentials to everything else they would need to steal all the data. It would also be a shame for someone to be able to walk in and tamper with the HSMs or potentially just walk out with one. But building that secure environment comes with challenges of its own. You want it to be secure, so you end up having to implement auditing processes and compliance processes and policies to ensure that they remain secure. And when the auditors come knocking and ask you to prove that your environment is secure, you better have the evidence on hand that it is. Something else to consider with these things is you need a good separation of privilege model. In the ideal case, you don't want your HSM administrators to have access to your Barbican servers or to have access to the database. Because if one person has access to all of these things, one person could go and dump your database, get credentials to the HSM, and then go decrypt everything that's in there and walk off with all your sensitive data. You don't want that. Now granted, that ideal case takes a lot of resources and knowledge. So it might not always be possible to have the ideal case of separation. I know in our case, we don't yet have three-way separation there like we should, but luckily we have another team within the company that's going to manage all of the production HSMs for us because we have a whole lot of those already that manage our CA business. So it was a good fit there. So now if we could find a way to get DBAs that don't have server access, that would be great. So let's move on and talk about some improvements that we're doing. So the first one being performance improvements in Barbican's PKCS 11 module. The module that is in there right now has a lot of room for improvement in performance. There are a lot of round trips to the HSM per transaction. A lot of this is because it is opening a new session, authenticating that session, doing what it needs to, and then closing that session for every transaction. And it's not doing a lot of caching of some things that could be cached. So part of what I'm looking at for improvements is reducing the number of operations that are being done against the HSMs by caching data where possible. So Barbican implements a multi-layer encryption scheme where you have master keys that will live inside of the HSM and be protected by it. Each project within Barbican has its own top level keys that are then used to protect the data within that project. So the way things are currently, the project keys, every time you go to encrypt or decrypt a secret, it loads the encrypted project key out of the database, sends it to the HSM to decrypt, and then sends the data into the HSM to operate on. Doing this over and over and over is really not efficient. So we're looking at caching those project keys such that after they're used the first time, we leave the decrypted key in the HSM and just hold a reference to it for future use. We're also looking at, as part of the caching, holding a session open so that we don't have to make future authentication requests for future sessions. In PKCS 11, sessions inherit the authentication state of already open sessions. So by having a session already open that's authenticated, any new sessions we create inherit that and we don't have to log in again. Save some time. And then later on down the road, once we can do some profiling and performance testing on this and we can ensure that there aren't any resources leaking on the HSM side, we can reduce that down to just a single persistent connection to do everything through and that'll save some more overhead. The community does have an effort going on with her looking into improving these things on their own and I'm looking here at the summit to sync up with them, compare notes and hopefully reduce this down to just one new improved module. So the next improvement I'm looking to make is basically classes of service in the PKCS 11 module. So the existing modules, it's either all or nothing. Either you do everything through the HSM or you don't use them at all. And this becomes a bottleneck if you need to do high transaction rates. I haven't been able to generate performance numbers of my own but in previous conversations with people the current module does not scale very well. So if you want to scale this out in the current model you need to throw money at the problem and buy lots and lots of HSMs. I'm sure the companies that make those things would love that solution because these things tend to be expensive but for I'm sure our usage, we don't wanna throw gobs and gobs of money at them. So what I'm looking to do is basically create a split class of service. The default class would be exactly the same as it is right now. That way we don't have default behavior differences between the existing module. But what I wanna implement is a second class of service that relaxes the encryption standard. Not everything needs the full level of security that the HSM plugin provides. So what I'm proposing here is a hybrid between the PKCS 11 module and basically the simple crypto module that's in there. The simple crypto module is the one you get out of the box. It's a development module and it's not secure because the master key lives in a configuration file in plain text. So don't put anything sensitive in there. What I'm proposing to do is basically take a lot of what's in simple crypto and instead of having the master keys in a configuration file, use the master keys in the HSM. They will continue to protect the project level keys the way they do now, but instead of wrapping the keys, they would encrypt the keys and then hand the plain text key back to Barbican and then Barbican can do secret level encryption on the API node itself. Combined with the caching from the previous change, this will take a lot of load off the HSMs and enable high transaction rates for services that don't need the top tier of security. I would love to contribute this back to the community as well and something else I hope to get feedback on while I'm here. We haven't yet started implementing this functionality yet so now's the perfect time to get feedback before we do. And finally, we've got one other improvement we're researching at this point and that is Barbican doesn't have a whole lot of database integrity protection. All of the encrypted data that's in there, so your encrypted secrets and the encrypted project keys have integrity protection on them but all of the metadata associated with those objects and all of the other data in there does not. So if an attacker can manage to get right access to your database, they could go in there and play with the metadata and do not fund things like take an expired key and make it not expired anymore. Or if you're using per secret ACLs in there, take the encrypted content of a secret they don't have permission to access and swap it into a secret in the same project that they do have access to and now they have the plain text from the secret they shouldn't have. This is a challenging problem though because we don't want to absolutely kill performance just to add that integrity protection and there's a lot of other hurdles that go along with this like every time you make a schema change in the database you potentially invalidate all of your stored signatures. So we need to find a way that we can add this integrity protection and have a migration path in the face of database schema changes. I mean I'm open to feedback from anyone in the community who wants to provide ideas here and we'll keep the community up to date on what we find here. That's it for my improvements. We're gonna move on to QA now and see what you got. There's a mic back there for questions so that it's picked up on the recording. Anyone? Hey, I'm gonna get a rower. So I was wondering what objects you're encrypting and you talked a little bit about your transaction rate. So like which ones have the highest transaction rate? Encrypted objects right now are the same ones we've had in Barbican forever. So just the secret data, those are the only objects that get encrypted in the database. Beyond that you have your encrypted project keys but like your containers, they're just objects of references to secrets and orders are mainly references as well to other things so there's no encrypted data there. As for transaction rates, we're still in the process of building our production environment so I don't have real performance numbers myself yet. I just have numbers that other people have generated. Anyone else? What do you use for high availability of the REST API endpoints themselves? For API HA, we've got Barbican behind hardware load balancers. They've worked fairly well except for being able to do health checking because the entire API basically sits behind authentication. It's been a challenge to do a real deep health check on Barbican. I've got a service I wrote that kind of works around that by sitting on the side that knows how to query Keystone to get an authentication token so it can make an authenticated request but that's another improvement that I would like to make but haven't actually started working on yet within Barbican. So you mentioned in your non-HSM rooted crypto approach where you cache the project level keys in the HSM itself. Are you worried at all about the HSM capacity being a limiting factor of how many of those keys you can actually store in the HSM? It very well could be an issue. At the scale of my deployment right now, it's not but it'll be a simple matter to add code to the plugin to periodically sweep the cached keys and find ones that haven't been used recently and expire them. Okay, thank you. Going once, going twice? All right, I'll give you seven minutes back. Have fun. Thank you.


Thank you very much. Thank you. Well, thank you all very much. I'm very happy to be with you today to talk about what we've accomplished these past eight years and to look forward to what there is yet to accomplish. The Business Government Relations Council stood with this administration as we fought the established wisdom that once ridiculed our ideas about economic growth and taxation. The companies that you represent recognize that they would not be able to succeed in the increasingly competitive world marketplace without major reforms at home. And that's why you supported us as we fought to cut taxes and eliminate unnecessary regulations and restrain the growth of federal spending. It was quite a battle, but the battle was worth it and the facts bear this out. Six full years of uninterrupted economic growth, the longest peacetime recovery in history, an unemployment rate of 5.4%, and almost 19 million new jobs created since our recovery began. Today more Americans are at work than ever before in the history of this great country and a greater percentage of the total workforce is currently employed. Now that total workforce, I had to come here to find out what they meant by that. That's everybody in the United States, male and female, 16 years and up, and 62.6% of them currently employed. Well, let's take a look at our successes from a Euletide perspective. According to a Philadelphia bank, the index of the cost of giving your true love, the gifts mentioned in the carol, the 12 days of Christmas, declined by 6.5% this year. The price of pear trees fell. The price of partridges was unchanged. However, I must sadly report that cost rose for pipers piping and drummers drumming. So you see, we still have some work to do. We understood the key to prosperity was low inflationary growth, and we achieved it. Our other aim, cutting the budget deficit, was hampered by a budgetary process that can only be called insane. And so I continue to support two measures to stop runaway federal spending, the line item veto and the balanced budget amendment. George Bush needs them and I hope you'll help him get them. Now I know Clayton will be talking to you about the Uruguay round midterm review in Montreal last week, but let me just say this. We've made remarkable strides during this decade toward our goal of free and fair world trade, not only in the GATT, but also through our passage of the U.S.-Canada free trade agreement. There are many who said that we could not stem the tide of protectionism, that the only way to respond to unfair trading practice was to close off our own market. Well, we didn't want to succumb to this defeatist attitude, so we launched the Uruguay round against all odds two years ago. And the midterm review, we reached agreement on a framework to move the negotiation forward in all but two most difficult areas, agricultural and intellectual property. We remain committed to an international trading system based on the principles of freedom and fairness, and we'll continue to press for the end of agricultural subsidies and the protection of intellectual property. We're confident these aims can be achieved, and when they are, I believe the people of the world will know a prosperity of which we have only the slightest glimmering. So as I take my leave of you, I ask you to continue the battle, the battle for the line item veto, the balanced budget amendment, and free trade throughout the world. I was one of the 43 governors in this country that had the line item veto. I used it 943 times in the eight years and was never overridden once. When you line item those things that may be passable when they're all buried with other things and then send them back for them to vote on them standing out there all by themselves, they don't vote the same way. Well, thank you all, and God bless you all. Mr. President, we in the Business Government Relations Council are deeply appreciative of your leadership of our nation for the past eight years, and I agree with you. I think one of the greatest achievements has been that phenomenal economic growth that we've seen sustained for a little more than six straight years now. You are leaving an amazing legacy, and we thank you because you've made our jobs as business representatives in Washington just a little bit easier. Our organization has encouraged a greater participation in terms of an interchange of information between government and business and an increased dialogue, and you and your wonderful staff have really helped us make that come about. So as a small token of our thanks to you, I'd like to present this plaque in recognition, as it says here, of your enlightened leadership to enhance the principles and purposes of the Business Government Relations Council. We thank you. Well, thank you very much. All of this and all I did was get government out of your way. You did the rest. Thank you all, sir.


Due to his extreme talent and wealth of political knowledge, Trump Jr. was offered a seven-figure deal by Rumble. And best believe, he took that deal because it's a very good deal for Trump Jr. And a very bad deal for Rumble, given that Trump Jr. is not a very prolific creator on Rumble. Yes, he posts consistently, but in terms of engagement, not worth the seven figures, but we'll talk about that. But first of all, here's some details about the deal, courtesy of MediaEye. This is very interesting to me. Trump Jr. signed a seven-figure multi-year deal with Rumble, the right-leaning alternate video platform to YouTube, according to a new report. Trump's new podcast will be titled Triggered, with Don Jr. pulling its title from the Republicans' 2019 book, also titled Triggered. He's very creative. According to Axios, the podcast will include Trump Jr. reacting to topical news stories and guest interviews. Rumble CEO Chris Pavlovsky has plenty of faith in Trump Jr. predicting in a statement that video podcasts from Rumble will be able to compete with traditional linear television at some point. And I mean, you've got to have faith in somebody to pay them seven figures. It can't have anything to do with the fact that this is the son of a former and possibly future president. And, you know, meritocracy be damned, this is definitely all about Trump Jr. because he's just, he's that talented. And Rumble was like, this dude who's already posting for free on our platform, we've got to lock that down and get Trump Jr. exclusivity by paying him more than a million dollars. Now some additional details about the platform courtesy of Axios. The company has made strides in user growth, but continues to burn through cash as it launches new features and spends on marketing. The Toronto-based company said it had 71 million global monthly active users on average when it reported its third quarter financial results in November. Of those users, 57 million were based in the U.S. and Canada. The company which raised money from VC backers, including billionaire Peter Thiel, U.S. Senator-Elect J.D. Vance and Colt Ventures, raised some 400 million in gross proceeds when it went public via its blank check merger. Now for sake of comparison, Twitter gets approximately 450 million active users while YouTube gets 2.6 billion. Yes, billion with a B. So they are nowhere near competing with any other social media platform, but that's not really what this is about. This is meant to be a right-wing ecosystem where Republicans and conservatives essentially have a safe space where their bubble isn't going to be penetrated with alternate views. And this is antithetical to what they purported to support years ago, this free exchange of ideas.  Now, Trump Jr.'s show is going to launch on January 23rd and it will air twice a week, so not even a five-day obligation from him. And the question that's on all of our minds, which is a bit of a rhetorical question, is, is this worth it? Is Trump Jr. worth seven figures? Well, I'll let you be the judge of that. This is a rant that Ron Filippowski shared on Twitter from one of his recent videos. And I've got to say, my man was really hyped up on caffeine or Coca-Cola, something, because he was very hyper here. And I just, I can't even give you any more context before you watch it, just watch. They want to take those phones and donate them the less this causes, rather than giving them back to investors and all of that other stuff. But this is a person who was wrapped up in that scandal. She ran a fund that was involved in it. A close colleague of Crypto King, Sam Bankman-Fried, and listen, is now pleading guilty, saying she knew what they did was illegal. So she knew what they did was illegal, but the Washington Post is still trying to make her into a victim. Okay, I mean, think about that. The Washington Post owned by Jeff Bezos. Okay, so it's the Amazon Washington Post, other notable headlines from the Washington Post in the past. Abu, but my dad took out, it was leaning ISIS.  This is the Washington Post. This is another article from the Washington Post before the FTX collapse founder, Sam Bankman-Fried, poured millions into pandemic prevention. He's such a good guy, we should make sure he gets off easy. And guess what, folks? In our system, he probably will, unlike you or certainly not me, if we actually did the things that he did. Most of those initiative have come to a sudden halt, according to the Washington Post. I wonder why? Because it seems as though he defrauded billions from people and poured them into leftist causes. And they're shocked. They want to put it back in there, because guess what, folks? The legacy media, they took their money to write headlines for them, in my opinion, right? And by golly, they're going to write them, whether they deserve them or not, whether it's current or not. They paid for those headlines. Rumble paid seven figures for that. Seven figures. And I'll be completely honest. I wasn't paying attention to anything that he was talking about there. I couldn't focus on the substance, because I was at the edge of my seat the entire time, just wondering when his heart would literally explode in his chest. Because, I mean, goddamn, he snorted more cocaine than an elephant could handle. His girlfriend was like, Trump Jr., sir, how much cocaine would you like? And his answer was, yes, all of the cocaine in the world, please. All the cocaine. I want it. And it's deeply, deeply cringe worthy how he's trying to emulate his father's cadence and hand gestures. It just, it's cringe that makes me want to jump out of my own skin. So they couldn't have, you know, opted for the seven figure deal because of Trump Jr.'s talent. Obviously, I was being sarcastic at the start of this video, but maybe you're wondering, well, what about the engagement? Maybe he's getting in a lot of views. Perhaps if he's drawing enough eyeballs to people to rumble, then maybe that is something that they could overlook if he's just this talentless hack. Well, no, in actuality, because I looked at his video catalog and we get more views than most of his videos. Most of his videos don't even hit 10,000 views. And while some do better than others, anyone with the brain would be able to realize that this is not seven figures worth of engagement. You're not going to pay anyone seven figures when they can barely get 10,000 views on a video that they post. We all know why this is happening. This is another classic case of failing up in the United States where somebody who is the former child of a president won the Lucky Sprim Club and they get a really sweet deal that nobody else would get. It's not about the views and the engagement. This is about them getting Donald Trump. And even if it's Donald Trump and you want to say that you have this exclusive deal with the former president's child, still, I mean, he was already posting on your platform for free. So to just give him money inexplicably for no good reason, it just goes to show you that rumble is not going to be a platform with longevity because they're almost putting too much money into courting these right-wingers and not enough money into bettering the infrastructure of rumble and making it a platform that is drawing in more people and not just right-wingers. I mean, if you want to make money, if this is a business that they want to be sustainable, you can't just exclusively court right-wingers, right? But that's what they're doing. And we'll see how it works out for them. But either way, it's working out really well for Trump, Jr. Because to get so little views and have so much money by working two days a week, Jesus Christ. I mean, it must be nice to be born into wealth. You just see more wealth come on to you when you do nothing. Many people are saying this. Join today, folks. You won't regret it.


So we're here with the PlatySense, and what are you showing here? Yeah, this is our product because it's sealed. This is a sensor for swimming to improve your swimming techniques. So here, for example, the swimmer would wear it? This is another product, actually. We have two products. This is one product. This one we are talking about. This is another one. I'm talking about later. So what can this do? This one, it measures your force. The force you press against the water. And also your hand's motion. We have some pictures. Okay, for example, this graph. Our sensor can measure things so in this graph. They measure your force, and here we use the color to represent different amplitude of the force. And also your hand's motion. After the swimmer wears this for swimming, and after the exercise, you can upload all the data to a mobile app. And then the app will run some algorithm to analyze your show and give you some suggestions. Can the app show this nice graphic like this? Yes, yes. It shows like this? Yeah, we aim at this. Is it for professionals or for amateur? No, it's not for professionals. It's for amateur, a serious amateur swimmer or triathlon. It should be for everybody? Yeah, anyone wants to swim faster. Anybody who wants to swim better? Yes, yes. Anyone wants to swim better? Just to let myself... How heavy is this? You can try. This is quite light. It's quite light, how long is the battery? Three hours of continuous swimming. Three hours of continuous swimming. Yeah, continuous swimming. Is that a price? We target for $150 USD for a pair of sensors. A pair, retail, right? Yeah, retail price. And what's the other one you were showing over there? Oh, the other one. Maybe you come here. We call it Marlin. Basically, it tells your left hand at the speed. How does it know? That we have motion sensors inside the device. For example, for pool swimming, after you make the turn, there's a human voice because we have an earphone.  Something like that. So you can swim continuously without stopping to look at your watch. Fantastic. Yeah, and also it works in open water as well. You can also wear it in the sea because we have a GPS receiver. GPS sensor? Yeah, GPS sensor. It's just like a running watch, but it's for swimming. And also it lets you know the time in real time. You put it on the back of your shoe then? On the back of the goggles. And GPS built-in. Yeah. And it knows when you change direction? Yeah, when you make the turn for the left. For indoor use, it just detects the motion. But if you're really fast swimmer, then the headphones are going to get out of your ear, no? Yeah, that's something you have to work on. Yeah, to try to improve. How? Strap on the ear. Strap on the ear. And we are also trying to use the bone to transmit the voice, rather than an earphone.


The National Sound Archive is based here, an extraordinary collection of some six and a half million recordings, growing all the time I might say by voluntary deposit from the music industry and by the extraordinary work of oral historians and other sound collectors. It's on 40 or so different formats and some of those formats will be illegible now, we reckon, within about 15 years. The technology is simply fading and dying in front of our eyes. This is an instance where digitisation means preservation. It's an expensive, difficult business, there'll have to be triage, we won't be able to protect everything. But nonetheless, we are building a partnership not just of us, but of regional archives around the country to campaign on this, to raise public awareness, to fundraise so that the amazing work that our teams do here can actually be amplified to the benefit of the UK.