Hi, my name is Tracy Tagohama Espinosa, and I teach a course at the Harvard University Extension School called the Neuroscience of Learning. It's an introduction to mind, brain, health, and education, and in this video, we wanted to share with you 12 steps that we use to create this very successful instructional design. Now, this is a realistic video. We want to begin just sort of by framing this. This is not an easy thing to do, but it's absolutely effective. And so we want you to think a little bit about whether you want to invest the time and actually have that paradigm shift of thinking that it will take to actually create a course like this. We'll do this within the worked model, this framework of our course so that you can see exactly how we put it together. We'll go through each of those different steps, and then we're going to put everything in the balance, you know, the pros and cons of using this particular instructional design methodology. So there's a whole lot of ways to take your courses online, and they range from things like webinars and MOOCs, things that have pretty much no contact between participants to this very high-touch environment of doing online learning in the structure that we're going to recommend. So there's a huge gamut of things, and you have to decide where you want to be on that spectrum. Now this video is really out to feed the curiosity of those people who want to challenge themselves to actually make the best possible online course. Many of you watching this video have already seen the tour of our online classroom and canvas. Please feel free to do that to get a sense of how things are laid out. So what we want to do in this video is give you a behind-the-scenes look at how all those pieces came together. This course is taught 100% online. It has a balance of synchronous and asynchronous activities. We use Canvas as our LMS, and we use Zoom for video conferencing. It is a flipped course. We meet synchronously once a week for two hours. We use a lot of things that are both teaching as well as evaluation tools, discussion boards, quizzes, reflection papers, semester project, but we don't use them in exactly the same way that you might be very familiar with. We'll explain a little bit about how we use those a little bit differently. For example, discussion boards are not used to check people's reading of a chapter, but they're meant to build community, right? Quizzes aren't meant for giving this end grade or summative kind of evaluation, but rather, more formatively, people can take them as many times as they like. We just want them to have vocabulary and core concepts before they come to class. So, similar tools, different take on them. We typically have a luxury of having people from several different states in the United States, as well as from many different countries around the world, and, of course, caps at 50 students. I'm the main instructor, and we usually get two teaching fellows. And the evidence we've been gathering has been done since 2014, documenting these last seven iterations of the class, and to measure the impact of different changes we've made to tweak the course to be its very best. We, of course, have all of the official feedback that we were given over the years, that we would love to have you stop the video and go ahead and read some of the things that people say about the course, but they really go into the instructional design, and they celebrate the ways that that facilitated their ability to learn how to learn within the structure of this class. We're really pleased when we get a bunch of comments that say things like, this is one of the best classes they've ever had in their lives, or that they felt that this was a highly personalized experience, and I invite you to have a look at those comments as well. So we're going to pick apart this structure that helps students sort of get a groundwork idea of basically what is a class about through the pre-class video. We invite them to explore using a huge variety of materials that we put into the bundles, which are many libraries curated for each topic. We have a lot of opportunities for them to talk to each other, especially in the small group work in the synchronous classes. And we have policies that encourage students to rewrite any assignment they'd like, using the feedback to improve their learning outcomes. This means we have several things that are individual activities that students do on their own, other things that are group activities, some things that are online, some things that are offline, as we try to leverage all of these tools in the best way possible. So let's look at these 12 steps. First, we have to decide on the course objectives. And some of you may call course objectives things like educational competencies or learning outcomes or learner outcomes, right? These are the things we set up in our syllabus by saying, at the end of the course, learners will know or be able to do or value. And this hovers around the big question, what knowledge, skills and attitudes will learners have at the successful completion of your course? So keep that in mind as kind of a north as you plan those objectives. And also keep in mind that the OECD definition of educational competencies is accumulation of knowledge, skills and attitudes that an individual can have. And knowledge are all the things that you could basically get off your phone, Google-able knowledge, dates, facts, formulas, theories, concepts, historical dates, right? Then we have skills, which is the application of the knowledge. It's one thing to know A squared plus B squared to C squared. And another thing to know when you use it, solve what kinds of problems. And then at the innermost core of educational experiences, our attitude and value shifts that occur. For example, when we begin to value teamwork or that we are people who are in solidarity with one another or that we appreciate perseverance, want to be resilient in the face of challenges. We know that anything that a student learns within your classroom context is some form, some combination of knowledge, skills and attitudes. So as you do this first step of deciding what it is, where you want to be at the end of the semester and you decide your list of objectives, just keep in mind this longer list of knowledge, skills and attitudes. And we have some example of course objectives, the way we've articulated them, for example, in our syllabus. They look like this. They're generally started with a bunch of verbs. We're going to explore, examine, highlight, learn, consider and translate different information. And we also give students that hook of an overarching goal. What is the overarching goal of the class? Maybe you'd like to try that as well. This leads us to our second step. So once we've decided on the objectives, we have to decide what order we're going to cover, what topics. Some people consider the topics to be the curriculum of the class or the way the syllabus has been laid out or they call them weekly topics. This is what we might indicate in our syllabus is being something like at the end of this week or at the end of this module or topic, students will have the prerequisite knowledge for the following least topics. So basically the order of the topics is important because they respond to a kind of constructivist design that has a learning trajectory that has logic to both the student and to the teacher. And the main question we keep in mind as we do this is what is the logical order of the topics that builds off this neuroconstructivist design or this neuroconstructivist trajectory of learning that it makes sense not only as we read through this list but basically in terms of the way that the themes build off of one another. This means that once we have our list of objectives we really have to break down our course structure and we do this by weeks. We have 15 weeks in our semester and we break them down in this way and we have identified a topic per week. So we think about well at the end of the day the last topic I might want to consider is neuroethics but at the beginning I definitely need to have my introduction and that means I have to have these fundamental basics of neuroanatomy but oh yes, I better stick in affective neurocentering that actually fits in very well at the smack middle concept because it has so much to do with neurophysiology but it also has to do a little bit with psychological terms. We know that there's pillars of attention and memory that need to be talked about, neuroplasticity. So we fill in the main topics in a logical order so let's talk about how we express these topical orders. What we do in our syllabus is that we list by week the name of the topic for example so we have each week listed out 1 through 15 and then we indicate what the main topic will be for that particular week. Additionally we also indicate there some of the other section topics so we're going to go into much deeper detail. In order to decide those weekly objectives we have to do a teeny bit of research. So weekly objectives, some people will call them unit objectives or module objectives. In our case each module is a week so we can call them weekly objectives but this is what you might put in your syllabus that says something like at the end of each week learners will know or be able to do or value very much the same structure as the understanding by design money that we use for the global objectives the same thing now at a weekly level. And this pretty much responds to the question you know what knowledge, skills and attitudes will learners have at the end of a successful completion of this week's topic. So this means once we have our structure we have it laid out in these 15 weeks we know what our topics are. If we take the example for every single topic we're going to do the same thing but for example in the mind-body connection we then look at it at this level of knowledge, skills and attitudes what are those sub levels. So we have a global objective for this week but we also have these specific objectives which are these subtopic areas. For example in mind-body we look at sleep and dreaming, physical activity and nutrition but also for example with a nutrition we have another subtopic there of the gut brain access. And we came to know this because by defining these weekly objectives we had to do a lot of research and that research also helped us build up bundles. These are the many libraries we use to curate information for students on that specific topic for that week. So going back to this layout in our syllabus not only do we have this level of information in the syllabus but within each of the different classes when we begin the class we also explain those subtopics for example on plasticity. We're going to be looking not only at the definition of plasticity but questioning this idea of critical periods looking at some case studies and understanding protective and risk factors that are related to plasticity. So we indicate that in the live class slide deck which we'll get to in just a second. So I mentioned that we do a lot of research to figure out exactly which subtopics and what information is most up to date. Well we also use that as we are researching to create things that are called bundles and in our course we create at least one bundle per week. For example in the week of plasticity we have plasticity and epigenetics so we have these two bundles of information. If you click on any of those within our course it takes you to another page in which there's a long list of resources that we came upon as we researched the topic as we updated the topic for the semester. And what we've done is take the time to hyperlink each of these articles or videos directly to their source so that the students don't have to do a lot of fumbling around to find the right resources. We do this because we want to differentiate homework. Students get choice. They're asked to come to class having reviewed at least one item within the bundle but they get to choose what and what's fascinating is that we can see that many students dig into far more than just one thing. They also use the bundles for example as jumping off points for the semester projects which we'll talk about in a minute. These days for K-12 teachers we also want to motivate you not only to think about bundles as being scientific articles and videos of Nobel laureates speaking about their topic but you can also include things like podcasts and links to other platforms like for example Khan Academy or Teacher Tube. Or you can include links to apps free apps that students can use like Duolingo to improve language or to educational gaming resources which can also enhance skill sets. The fourth step is to create the pre-class slides. So since this is a flipped class these are the slides that are going to be used to make the video pre-class. So we have pre-class slides some people just call them power points or slide decks or the flipped video slides and a key step as we're emphasizing throughout is to keep in mind learner outcomes as we do the stage of planning. And so our guiding question here is really what is the best use of my synchronous time. So the idea here is to offload some of those dates facts formulas that knowledge based information that's vital but you don't want to spend a lot of time just dictating that to our best use of our face-to-face time or synchronous time might be better used in debate or in discussion or for clarifications or going deeper into the information or using or applying skill sets right. So we offload the other important information but not necessarily something they have to do in synchronous time with us to an asynchronous activity as homework. And we create this slide deck based on what we have decided are the objectives of that weekly topic. So we create that pre-class side deck directly from the information we've gotten from our research on the bundles and those knowledge skills and attitudes for each topic. And depending how you might want to flip your class there's different ways or links of slides. Sometimes people will decide they want to flip an entire class lecture so they have a video that might be 20 to 60 minutes long. Other times people choose to do little bite size nuggets of highlights. So maybe these are three to eight minute videos having two or more of those. And yet other teachers just make a quick video introductory saying, hi kids this week we're going to be looking at dinosaurs and they're eating habits and you're going to be watching a video that's been made by whatever. And you just use yourself as a bookend to introduce somebody else's video which means that your video itself is really really short. So depending on your choice of how to flip different types of slide decks.  But you have some choices to make. You should not flip everything that would be in a regular class. And so if you ask yourself what is the best use of my face to face time you need to offload the things that aren't the best use of your face to face time. So if we want to do things that are more application based or if we want to introduce a new problem or collaborate together or to rehearse a skill set that's great use of your synchronous face to face time. So we need to give all the prerequisite information in the video beforehand. This means we're trying to leverage the use of this technology to support our learning goals not just sort of shove whatever we can into technology and make some good choices here. One way to think about this is to use Bloom's taxonomy right. At the basic level of Bloom you have a lot of definitions and facts and just being able to understand core concepts. I would suggest using that taking all of those dates, facts formulas, theories, information that's really important for students to have but they just have to memorize it right. Use that stuff at the bottom of Bloom and use that to flip. But also remember your pre-class video shouldn't be like full of like just that boring stuff. You really have to use that hook and engage students in the information. Sort of frame things perhaps as a problem, case studies as some kind of a narrative to give your videos a little bit more life. So once you have the pre-class slides we make the video. So the pre-class video is sometimes called the flipped video or just the video or the weekly video. And again keep chanting to yourself this understanding by design concept what are the objectives? How am I going to evaluate this? What are the activities I do? And at the end of viewing this what is the knowledge skills and attitudes my students will have? And again we do this by asking what is the best use of our synchronous time and then we offload the things that might not be the best use of that synchronous time. So after that slide deck the slide deck is what creates the pre-class video. So the pre-class video based on the pre-class slides highlights those key ideas that basic elemental knowledge of the topic of the week. The sixth step then is to use that same slide deck that you created to make the video to create a discussion board prompt. Now some people just call this the discussions or the chats or the chat forum or the class forum. Remember however you should still think about this in an understanding by design structure. What will the students be able to know, be able to do and value after they participated on that discussion board. And for us the discussion board is a space to create community. So a guiding question we might use is how can I get students excited around a topic that stimulates debate and encourages a learning community. So this means from that same information we can create this discussion board prompt. And the discussion board prompts have the same instructions every single week so students don't have to use a lot of cognitive load there but rather use their energy on the prompt which we try to keep pretty short. So they look at the prompt, they see what they're asked to reflect upon and what they're asked to give evidence for. And they make that first attempt at their prompt, they write out their ideas and then they're asked to respond to two other learners prompts. And so this creates a back and forth between concepts and between ideas between individuals in the class. So similarly based on those slides it came from those specific objectives we create the quiz questions. So a quiz might just be what you know as a test or a knowledge check or a section review or a chapter review. The idea is that we still ask ourselves why would I have somebody do a simple multiple choice true or false test? I mean that's very superficial knowledge isn't it? It is however it is fundamental to being able to apply that information. If you're unclear about if one theory said one thing and another theory said another it's very hard for you to be able to discuss and use that terminology. So we use the quiz in our class because we know that frequent low-stakes testing is a fabulous way to enhance memory and it also gives us the assurance that everybody shows up to the live class with a certain level of vocabulary so that we can have deeper discussions in the class. So a guiding question here might be what is the basic knowledge and vocabulary I want to be able to use with the students on that topic of the week. So again coming from that base of knowledge knowledge skills and attitudes we come up with our quiz. And these quiz are not rocket science basically you know they're true or false there are multiple choice and the beauty of it if they're well conceived and thought out and definitely test these out before you put them up but they are self-graded so you don't have to it's pretty hands off kind of an experience. You just think of some great questions put them up there and then hopefully you ask the students go ahead take it a million times just get a perfect score but try to take it once before you do any of the work to get a baseline what did you know before you've been coming to the class then take it again after you've watched a video and after you've come to the live class and then try to get that perfect score on average our students take each weekly quiz about four times and they have a very high average very close to a hundred. Once they've done that we use the discussion board replies and we use any frequent mistakes in the quizzes we look at the discussion board content questions that some students will submit and we look at the three two ones the reflection papers from the previous week and that helps us develop the live class slide deck so the live class is what we do in our synchronous meeting time so you might call it the live class powerpoint or the synchronous powerpoint or the live class presentation or the live class slide deck all of those would mean the same thing to us and this does not mean creating a totally different presentation this means taking the core of what you used for your pre class slide deck and then enhancing one or two of the key ideas that are there that you're going to go into depth on based on the feedback you got from the discussion board so after they come to the live class what knowledge skills and attitudes would I expect of them so the main guiding question here is you know what is that in-depth understanding I want students to have after they come to this live class so this means that based on the replies to the discussion board and based on the pre-class slide deck and those mistakes that might have come up in the quiz we decide what is important to do in the live face-to-face class we want to remind you that every time you lay out that live class slide deck that you should really frame it you know this is what we're going to do today and also offer that summary in order for people to keep their heads around where they are going with the information and how far along we are with that process so now the ninth of these 12 steps is when you actually have the face-to-face synchronous class with your students now this means that the students have already done quite a lot of work before they get to you that's the typical nature of flipping they have to do a lot of work before they come to the live class most of the work is before they come to the live class however when they get to this live class or the synchronous meeting or this face-to-face encounter here what's really important is that you have created a class protocol this gives students a regularity of expectation of what occurs in the class which reduces their cognitive load they don't have to prepare for something totally different each time they understand the basic structure of what you're going to be doing in that class encounter which allows them to be able to pay attention to the right things at the right moment so this means that within the live class we do several types of activities one important one has to do a small group work and the discussion and collaboration on activities and application of the information so our protocol looks like this don't feel obliged to do absolutely everything but we get in the class a few minutes early and as people come in start to come in little by little we greet them we say hey Jenny good to see you again how's everything going we use their names and we also allow them to sort of you know vent slash transition for more of it is they're coming oh no fine I have I'm just settling down it's been quite a busy week great well tell me a little bit about your week so that they can feel this very natural move from being in one space to coming into your space we engage in a lot of short dialogues they can be about the class or not we let the students sort of drive that and if they have any class related comments we are sure to jot those down and be sure to integrate them into that live class the point is that we want them to feel recognized as members of our community as they come into that virtual door so as we begin the class we basically announce that we're going to be recording and we start the recording right and we thank people for arriving on time I know that sounds counterintuitive it's your class they better arrive on time actually it's their life and it's busy and they have made a way to get to your class and we got to thank them for being there we also use this opportunity to make sure that anybody who doesn't have their cameras on turns them on and make sure that their names are not their emails or whatever that they have written their name the way they want to be called by others and then we do a very quick question in the chat for example write down one thing that really struck you about this week's pre-class video that you want to be sure we talk about today or something like that why do we do this this also helps us take role it's a really fast way to make sure that people were in the class on time and responded but it also helps us put our finger in the pulse of any big issues that might have come out of the videos that we didn't already incorporate into our pre-class slides we also want to remind you to use the chat extensively it's one of the greatest disruptors that we have in educational practice it's like inviting students to pass notes behind the teacher's back but use it wisely and it really does have a whole lot of benefits for example it offers immediate clarifications to students on questions it also encourages this kind of a back-general conversation that dives deeper and personalizes some of the examples for students and of course the teaching assistants are in charge of manning the chat basically reminding me every few minutes if there are some key points that are coming up that we need to discuss in the larger group if you don't have an assistant go ahead and ask any student to do that it's an incredible skill set that they can develop to learn to synthesize and to scan that information and to be able to pull out those juicy points that would be of interest to other students so take advantage of this wonderful opportunity to get those clarifications and use that chat throughout the live class presentation so let's look at how we begin we begin by showing the basic slide deck which will include the objectives of the day and so we explain them and we also display them on that PowerPoint and we ask students if they are happy with the agenda of the day or if there's anything that they really want to make sure we cover and after we've gone over the agenda the very first agenda item is always your questions and ideas and those are things that we have to hold from responses from the discussion board so as we start with students ideas we begin by sort of summarizing the way that we group them or these patterns of responses you know three of you you know Jenny Anna and Tom you really called out this one issue but I thought it was very unique the way that you approached this Tom can you elaborate a little bit more on what you wrote in the discussion board now that gives Tom the microphone and the chance to synthesize some of his main ideas which calls out points from other students and what's so special to see is that you'll often time see them congratulating each other within the chat and you know what a terrific idea Tom or I hadn't thought about it that way or excellent point and this is a wonderful way to give peer feedback throughout the course so for each of these summary points we usually choose one or two students to speak and really celebrate what they have taken the time to write in their discussion board posts. After we've gone through all of the things in the discussion board which can take very easily a half an hour or so to go through those key ideas and to make sure that we've sort of vetted the most important patterns of replies that have come out in the discussion boards then we turn to our pre-class by deck which we've now taken one or two of those big ideas and we've chosen the things that we think need to be highlighted even further and we contextualize that information and we remind the students of that core vocabulary that was in that pre-class slide deck and we talk about the controversies that might occur within the field around that particular topic then we go into breakout rooms in order to get deeper into a group discussion we normally put about three or four people into a breakout room in order to have a really good chance for everybody to have a talk we normally go to the breakout rooms between eight and ten minutes although every time we come back students say oh we could have spent a lot more in there because they get really into it and they have the opportunity in such a small group that everybody gets to talk but before you send them to the breakout rooms make sure that you frame it with a kind of a prompt we use a lot of the visible thinking ideas you know I see I think I wonder oh we tell them that they're signing this come back one thing that they all agreed on and one thing that was controversial or something like that so that they come back they're in a similar framework if you just send them off and you say okay how did everything go you can get all kinds of different perspectives as opposed to something more unified so we recommend that you send them off with a specific task that they return being able to articulate their findings in a certain format and then after they come back from the breakout rooms we have them share in this larger group and when they come back I'll call on just about anybody somebody who has probably not spoken yet already and I'll say so Anthony who was in your group with you now this really forces them after the first or second time they realize she's gonna ask me their names I better know people's names so they get to know each other better and they'll say well I was in the room with Anastasia and Victoria okay terrific and tell me what was the one thing that you discovered you had in common what was the one thing that surprised you or for which you had great differences and this is fabulous for their power of synthesis very interesting you'll see that some people are naturally already good at this they already know this they've taken notes in a way that they summarize their ideas and in 30 seconds they give you the whole 10 minutes encounter then there's other people who have a really hard time synthesizing their ideas and they'll get stuck on what they thought on what they thought on what they thought and it's very important to subtly cut that off and to say I hear what you're saying that was really an important point to you did Anastasia and Victoria find the same things make sure that they realize what they're doing is reporting for the group and can they synthesize other people's ideas that's a huge level of thinking so after you've done that one person two people three people depending on the size of your class you may be calling on seven or eight people to give that summary that can take quite a while but it's well worth doing allowing people to share back groups and to hear each other and always at the end try and say well I hear at least two patterns of things coming out of what you guys did in your small groups what patterns are you guys seeing and that will prompt them to think about globally what is this group saying how are we understanding this information this is fantastic because it goes from that individual thinking to that small group understanding to this larger group synthesis to try to understand where are my thoughts in context to the others you basically repeat this until just about everybody has had a chance to talk also always create a space in the beginning middle or end of this exchange with the students to say by the way if there's additional ideas that came out of your group that you'd like to be sure are also heard please put them in the chat that's a nice way of saying if the guy who's summarizing for you isn't doing a good job add on to that so you can create that space in the chat so everybody feels that their voices are being heard and taken into consideration now something to always remember as you are doing this live class slide deck and deciding what are those topics you want to go into remember it's depth over breadth when you flip a class this means that this kind of more superficial definition information is given in the flipped part so that in class we can go deep we can get clarifications we can get to those subtle nuances of the information so the idea is to give that space where students can go deeper into their own understanding of those concepts as opposed to adding in totally new ideas onto the class so the synchronous time remember should be devoted to helping students get any clarifications they need and to seeing real life applications of the information and always remember let the students lead they generally know what they need and if you ask them and give them the space they will respond accordingly so if you have a long enough class and you can go into a second topic and do that whole kind of small group activity and then come back to the larger group that's terrific we have a luxury of a two hour long class and this works perfectly to break up the time keep the tensions focused and to go into depth and at least two topics in our class but if you have a shorter class you might only have enough time to do one breakout room now after you've gone through these topics gone into depth come back out before you in class do not fail to summarize and contextualize all that information that came out give a less than one minute summary okay we did this today that today and the other thing today we also heard some great examples from Alana in this case study that she offered us and from what Cynthia mentioned call out those particularly good contributions to the class then be sure you put all of that information from what we did today into the context of the larger class one way to do this is to have a connector what we always do at the end of our classes look at risk and protector factors that fits into our topic we say okay so now having gone through all of that information about whatever nutrition or the role of sleep or how memory systems work or whatever it was we ask the students to think of what are the risk and protective factors in your life that might enhance or inhibit your ability to take advantage of that for better learning so you can think of some kind of a protocol where you continuously do this at the very end so that you also give students the sense of closure to the information then finally point towards how this connects to next week's topic so you're building bridges between you know what we did in the past what we did today and what's going to happen in the future and why all of this really fits and makes a lot of sense together the very last thing we try to do in the last five minutes of our class is reflection and we ask students to say three things that they didn't know before class two things that they're curious about and want to know more and one thing that they might change in their personal or professional practice and we use the three two one as a way to gauge their participation grade but it also measures their development over the course of semester and at the very end of the semester we show them an accumulative list of three two ones throughout the 15 weeks for all of the people in the class which really stimulates a great appreciation for all that they actually learned in the class a secondary or additional benefit of the three two ones is that it serves as a feedback to the teacher if the students are calling out three things that you didn't necessarily think were the most important things or if the students are asking for more information they're curious about two things those are big heads up to help you know what types of information they now need to move forward it also gives you some feedback on your teaching have you included all of the right elements or were the things that stood out to them not necessarily the things that you brought up but things that the students were bringing up which you need to pay more attention to in the future reflection is super underrated in schools having downtime having the time to step back and say you know did I really learn anything or what's going on here is incredibly important and we don't give enough time to that so highly recommended it this has been actually one of the things that our students recall years later and tell us this is one of their favorite parts of the class is having just a few minutes to themselves quietly to think about what just happened in the past two hours what what did I really learn here what are the things that stand out to me and finally at the very end of our class when the time is up the time is up and we end on time and we tell them it's in honor and respect out of your time that we're ending right now if anybody has any loose questions please stick around but for those of you have to go go and we say this because we want them to know that we think their time is important their lives are important what they have to do is important what we find very interesting is almost more than half of the students end up sticking around just to try to catch up with those questions or clarify or hang around as they're writing their three two ones to listen to what other people are questioning which we find pretty fascinating but this time is vital because allowing them to stay after class for those few minutes actually saves us time in office hours for example if they're able to just get it out and it's also still recording if they ask a question that's important that we're able to respond immediately that's fantastic because it allows other people to hear I hadn't thought of that but I'm glad she asked it because now I don't have to ask it and that saves us time so step 10 step 10, 11 and 12 are things we do throughout the entire instructional design process it's not something that you have to do at the very end it's something you should think about throughout the big idea here is that evaluation or what some people call assessment or feedback or formative embedded assessment or diagnosis depending on the time that it occurs within the learning process all of this should be something that occurs throughout all of our evaluation it should not be something that happens at the end the biggest idea is that evaluation is a teaching tool it's not something separate from the learning process in fact people learn a whole lot from the evaluation process so the big idea or question to keep in mind here is how are different evaluation tools used in this instructional design what are we trying to measure and why so some of the class activities including evaluation that we do are divided into asynchronous and synchronous activities initially the students are on their own it's the flipped class right they take the quiz they watch the pre-class videos and they respond to the discussion board prompt and they explore the bundles or these many libraries that we have built right and based on some of the errors in their quizzes or some of the questions they might post on the content and logistics board or some of the patterns we see in responses to the discussion board these things help us do the synchronous structure of deciding what goes into the live class so based on the pre-class video we take that we strip it away and only keep one or two big ideas from that based on these other inputs we also have synchronous meetings when we go into sections in our course we usually have one to two sections every week in which we do a deep dive into a certain aspect of the topic of that week and we also have other workshops that help students with their semester projects after any of these live synchronous encounters we ask students to do a 3-2-1 three things they didn't know before two things they're curious about one thing that they want to change about their personal private lives based on the information presented now based on these workshops these are done to help students manage and make their way through their semester long project which is a very significant thinking as writing process we go back to the synchronous work which is basically two big things the first is that there's an ability to do over to rewrite information and there's also the ability to go deeper into the information so we find that even if they don't have to students will retake the quiz over and over and over until they get a perfect score and some of them will re-watch those pre-class videos and others about half of the people in our class watch the recorded two-hour session when we were live mainly because they say there's so much going on the chat that I didn't pay attention to what was being said or vice versa, right so they go back and re-watch the video they're also encouraged to go back and to extend those conversations they began with a discussion board but most of the time people are digging deep into the bundles they are digging into the materials they call it their rabbit holes they usually get a little bit sidetracked and keep reading and reading and reading even though they're told they only have to look at one of the things we have an average of about 100 resources in our bundles and students will look at 10, 20 of them just because they're interested in the topic this means that our evaluation scheme is ongoing it's embedded and it's continual and it's formative for the most part we'll talk a little bit more about that in a second about why we believe in a rewrite policy but more than anything all of these activities because we're online are incredibly well documented we have quite a lot of evidence for how many times people do these do-overs for example, how many times they take a quiz how their different iterations of their projects went we can see this development over time which is incredibly well documented and as we mentioned before because it's a balance between these asynchronous activities students have a great deal of flexibility in terms of time in when they do those things so just a clarification on our rewrite policy it's a lot of work however, having said that this is perhaps the comment that we get every year at the end during evaluations about what was the most important thing that allowed them to learn in this class so do-overs the ability to learn from your mistakes is something that students tell us has a significant impact on their learning and which is also supported by John Haddy's research and this is also part of this educational paradigm shift because typically as a profession we do not bake into our process enough time for a liberal rewrite policy which means to be able to apply this you really have to have a lot of great planning in which you've already incorporated the time for rewrites within your calendar structure our basic policy is that they have one week after they have been given feedback to resubmit to us so this means that we measure different levels of competencies and attitudes with different tools along the way meaning we're sure that we get to the core aspects of a course, attitudinal shifts with the most tools step 11 is another one that goes across the board throughout the entire class and that is communication how does that teacher-student exchange occur nor are there any formal structures there and so from the start of the course it's vital that students know where to find information and how they get help this means that a guiding question here would be how will you communicate with your students to ensure regular contact clear expectations and a seamless flow of information just allowing some kids to maybe text you on your phone and others to send an email and some others will come to office hours it's not good enough you have to create clear parameters and so in our course we use the emails that's within the platform you tell them don't email us to the platform we use announcements we invite their comments to our comments on the feedback that we give them for their discussion boards and their 3-2-1s during the live chat we give them feedback about their comments or where they've taken information and we have an additional discussion board that's only meant for content or logistic questions so they can go ahead and post something there if they say you know I watched the pre-class video and you said X and I've read Y can we talk about this so if they've posted it there that gets worked into our live class presentations we also have office hours that we invite students to come to and as I mentioned before coming a little bit early to class and sticking around a little bit later does a whole lot to facilitate the communication within a classroom the one thing we definitely discourage is using like instant messaging things in which you are on call 24-7 we tell students that we want to keep the communication chain within the Canvas platform because it helps us document and that's been crucial in the one or two cases we've had over the past seven years in which a student might have had some problems that really needed to be kept in the classroom and not in private emails so do try to formalize your communication patterns with your students independent of which of these tools you use all of them none of them just one of them regularity is really the key to good communication if students know when they can find you and how they can get information that's more important than which of these tools that we're actually going to apply so just make sure that you created a really clear structure of how you're going to communicate with the students so the 12th point is the biggest stickler here because it has to do with a total paradigm shift in the way we think about education and this has to do with time the hardest step to consider applying has to do with this paradigm shift in educational practices about what we give the most time to this structure requires you to give more time to evaluation, to feedback to this human touch, this contact with individuals it also requires more time to plan before I used to preach to people that you needed to have a one to one to one structure in your schools where every hour that you teach you should have an hour to plan and an hour to evaluate teaches what I would say oh if only I'm going to go even more extreme on this I think that the current structure we're recommending is actually a three to three to one for every hour of teaching you're actually doing three hours of preparation and three hours of evaluation that totally turns teaching on its head so we estimate if we want to do a reality check now to decide if this is really what you want to do for a class of let's say 25 students over one semester we think that you're going to prepare at least six hours per topic and you're going to evaluate at least six hours per week giving feedback or correcting reflections or mentoring on those projects or doing sections or workshops or answering those emails they send you and just two hours that you'd be face to face with those students plus that time that you would have before or after class so ideally what you do is you'd have the summer to do that prep work you know to do the flipping element to record those videos to get things structured and then teachers should never have more than five perhaps it's impossible time wise because that's working more than your eight hours a day as well as 12 months a year so this really means that the summer off that teachers theoretically get is not a real thing and that they really are doing full time work even though they might not be standing in front of their kids more than 10 hours a week they are putting in so much more time on the back understanding that evaluation process making sure that those kids feel motivated giving that one on one nudge that's needed often in classroom time so it is a change in the shift of the way that we think of teaching and teachers so in summary what are the pros and cons of all of this if you had to think am I going to do this or not it's a big decision think to yourself just as we mentioned right now it's a change in mentality to actually go about doing this and some of us are just sort of keeping our heads above water with these big changes of going online but if you are one of those people who are willing to invest and decide I am going to have the best online classroom possible shift in mentality won't be that hard but it is something that's going to take a little bit of effort right the benefits of all of that is that there's much more focus on this humane aspect of teaching and the contact with the students it's really much more student centered to structure things this way students report that it's far more personalized and that we can differentiate because of the offers within the bundle meet students at their entry points not just where we want to start the curriculum but you have to accept that this means a lot more front end planning this means taking the time to structure things having said that if your course is well structured you just tweak it slightly every year so the amount of investment of your time of structuring the course goes down after time it's far more better documentation we know there's a lot of other benefits here and at the end of the day it really means that teachers are doing a lot more listening and less talking and students have a lot more interaction so this high touch structure to online learning is something that's proven extremely beneficial over the years students report years down the road that they still retain the things that were learned in this format so I would say for those of you who are willing to do it it is really well worth it so in summary we looked at our starting point and we said you have to decide your global objectives of the class you have to lay this out in a very logical and organized way as far as the way that you present your topics so that they have a clear learning trajectory and once you've done that then you have to disaggregate those specific objectives for your class for each one of those weekly topics by going deeper into research and understanding the bundles that you would want to supply to your students so that they have additional resources you can then create that pre-class slide deck and do the pre-class video develop your discussion board questions and create that quiz and from the results of your discussion board and the results of that pre-class slide deck and the results of your quiz you can then develop the live class slide deck and think about those activities that you would do including that small group work where students are applying information or discussing or debating information and after that you create your 3-2-1 that would be a fantastic instructional design in summary since Rebecca somebody who's worked with me on this course for many years created this graphic in order to try to separate out all of these key elements the things you do before during after class the types of activities, asynchronous, synchronous things that are done in groups and alone the things that are digital things that are analog we invite you to have a look at this conceptualization of our Harvard course think about the elements that you would like to include in your own classes and be sure to let us know if there's anything we can do to support your work in creating the best instructional design for your course focus so what did we do with you today in this video we tried to go through 12 different steps towards a successful instructional design we looked at these one by one tried to give you an example of each of these a worked model so that you can decide which pieces of that you want to incorporate into your own work and additionally we really want to make this real this type of a class takes a lot of work but if education, if learning is your goal I really hope that you will embrace this as we mentioned before there's a lot more front end planning but the payoff is definitely worth it the time with the students the depth of the information you can cover and most especially putting that humanity back into education is really evident through this model so if you have any questions don't hesitate to get in contact please look at the other free resources we have on the learningsciences.com and best of luck with your instructional design


The gravel crunched under the tires of the black pickup as it turned into the small parking area near the trailhead. The headlights put the immediate tree line into sharp relief until the vehicle was turned off. The words Magafin County Conservation Officer were emblazoned in gold lettering across the side of the truck. The driver's door then swung open, allowing the chilly morning air to seep its way into the cabin. The engine clicked as it cooled, the only man-made sound amongst those of the forest slowly waking up. Jim Taggart stepped out into the early morning air from the truck's interior, dressed in a hunter-green jacket and black gloves. It was still dark out, with barely enough ambient light to see by. Jim's breath trailed behind him as a steam of smoke as he stretched. It was now that he focused his attention on the SUV parked on the opposite end of the parking area. Jim walked over to it and after a quick once-over, revealed a completely flattened rear tire, decided that it had been abandoned. Jim wrote out a citation and stuck it under the SUV's wiper. He then returned to his truck and leaned back in to retrieve his supplies. He hoisted a large pack onto his back first, made sure that his service pistol was still securely holstered on his right hip, then slid his rifle sling onto his left shoulder. The conservation office had been receiving calls about some weird deer sightings over the past month or so. Jim and the others had come to the consensus that there was most likely a buck that had developed a debilitating disease. Most of the descriptions of the animal fit this narrative. This was why Jim was out here. He'd drawn the short straw and now it was his responsibility to put the poor creature out of its misery. Today was probably going to be a long day. Jim started out down the trail at a brisk pace. He had both a GPS and a paper map packed away in his bag, but this was mostly a formality. He knew all the trails like the back of his hand and was able to navigate the surrounding areas almost as well. He hiked out to a meadow some miles out into the woods where the reports seemed to be loosely centered around. Following a spot among the thick vegetation, he set up a small deer blind and began to wait. As he waited, the stories of the campers and hikers that had made reports began to resurface in his mind. Most of them were fairly on track with the symptoms of chronic wasting disease, a deer unsteady on its legs, falling all over itself, looking emaciated. But a few here and there had crossed the realm into the bizarre. One man had reported seeing a deer with white eyes that had walked on its hind legs, like a human. His coworker Landon had been given the responsibility to take the report as he was the best at keeping a straight face and pretending to take things seriously. Personally, Jim had written off this report and the handful of others equally as bizarre and as the result of drugs or alcohol. Jim rolled around with no sightings of any deer at all, sick or healthy. Jim gave a wide yawn and set his binoculars down to the side of his chair. He began to rummage through his pack, producing a granola bar. Deer with this illness oftentimes no longer followed normal behavioral patterns, so there was still a chance he would see it in the midday. But it was just as likely that it was scared and hiding away, which would mean he'd be back out in a day or two in an attempt to actually track the thing. Jim jumped and cursed at the sudden sound of an animal crashing through the forest. It sounded as though it was coming from every angle and he snatched back up his binoculars and pressed them to his face. For an agonizingly long minute, Jim sat frozen in place. The air around him plummeted in temperature and he could suddenly see his breath again. The forest abruptly darkened around him, as though the sun had quite literally fallen out of the sky. As Jim started to lower his binoculars to examine his surroundings, the creature burst forth from the treeline. Jim had to switch his binoculars to night vision mode to really get a look at the thing. It was skeletal in appearance. He wasn't sure how the creature had the ability to hold its own head up, let alone ambulate. As he watched, the deer continued into the clearing on wobbly legs, nearly collapsing to the ground as it tripped over its own feet. Jim now moved to trade his binoculars for his rifle. He could understand now how some of the hikers had mistaken it for something less than natural. It was probably the worst case he'd ever seen. As Jim lowered his head to peer through the scope of his gun, the deer stood shocked still in the clearing. Jim carefully took aim for the head. He let out his breath slowly and finger went to the trigger. A bang rang out, bringing the contrasting silence of the woods to the forefront. The deer collapsed with a thud that seemed to reach even Jim's ears. His main task completed. Jim now went for his cell phone. He sent a quick text back to one of the others at the conservation office. Got the deer, I'm headed back now. All mark on the GPS were to find the body. He didn't want to stick around anymore. It was just a diseased deer after all, but things were wrong. He began to haphazardly pack back up his things, considering even leaving the deer behind. Someone else would have to collect the body. Maybe he would even ride along when they did, but right now, every fiber of his being was screaming at him to get out. When he stepped out, he couldn't help but look toward the direction of the dead deer, but it was gone. He hesitated for a moment. He wanted to leave, but his legs seemed glued to the forest floor. Cursing, he twisted to reach for his binoculars that he'd stashed in a side pocket of his backpack. When he turned back around, he dropped them in surprise. The emaciated deer now stood mere feet from him. There wasn't a scratch on it. It fixated him with a stare, eyes pure white and cloudy and far too forward for an animal of prey. Fur matted here and there, the creature swayed on its hooves as though buffeted by an unseen breeze. The iron musty scent of old blood hit his nostrils and caused Jim to grimace. Its sides did not seem to swell and fall with any breaths. Whatever this was, it wasn't a deer. Then its lips parted and began to raise upwards on its vile muzzle. The parted lips revealed a mouth lined with sharp, pointy teeth. The thing was smiling at him. Jim dropped his supplies in shock and in a split second decision, unsure of the direction of his truck under such terrifying circumstances, he made his best guess and bolted. Jim could hear the crashing of the underbrush just behind him as the creature gave chase. He allowed himself one glance over his shoulder. All traces of feebleness and incoordination were gone from the animal. Jim's ears were filled with the sound of his labored breathing and of the creature barreling through obstacles as though they weren't even there. The man shrieked as his foot caught on a route and he was sent skidding across the forest floor. He quickly rolled over and propped himself up to check on his leg. Sensing no damage severe enough that his adrenaline could not overcome, his eyes darted around the blackened forest. The creature stood several feet away from him as though Jim had neither gained nor lost traction from the moment he'd taken flight. Jim stared at the creature, mahogany eyes massive, his chest heaving as he struggled to catch his breath. He pushed himself backward in what felt like a futile attempt to distance himself from the creature. He pulled his service pistol from his holster and took aim, but he knew it would do no good. This was it. This was how he would die. The buck's mouth dropped open. Its bottom jaw hit its neck as if it barely hung on just by sinew. From the gaping maw of the creature came a noise that resembled the call of a hyena. Was it laughing? The not-deer bore down on him and Jim screamed. The sound of gunshots rang out across the woods. Jim's coworkers came looking for him. A couple of hours later, his truck was still in the gravel parking area at the head of the trail. The bright orange ticket fluttered in the breeze under the SUV's windshield wiper. The small party of three followed the trail and moved out to where the GPS marker had indicated the dead deer should have been. There was no sign of the deer, not even blood droplets. They found where Jim had set up and his abandoned supplies, but they couldn't find their beloved coworker. They brought in search parties and sent hounds the next day. They even swept the area with a helicopter, but the forest and mountains were dense and visibility was minimal. The dogs brought the officers to a clearing deep in the woods in the opposite direction of the trail and Jim's truck. His coworkers were at a loss, unable to figure out why Jim had gone this way. The dogs lost the scent in the clearing and they never picked it up again. Jim Taggart remains missing to this day.


Hello there. Good day. I can see you. I think I can hear you. Yeah, I hear cooking. That's not my cooking. No, that was mine. I had to swap the headphones back and forth. I try to tune out over the weekend as much as I can, but I did see you posted something on the weekend that I'll go back and look at. I lose track of all the stuff I heard sometimes. Good morning, but how do I pronounce your name, actually? João. Okay. Good day, João, because I know you. I will make any assumptions on the pronunciation of your name of where you live. So a minute or two. Yes, it's first up, but we'll wait a minute. If folks could sign in the message here. Yes, you'll be up to go through your PR. So whenever you're ready, we'll just start there. Just setting my windows. All right. Can everyone see my screen? Yes. Okay. So this one is revisiting some of the key management scenarios that we had discussed earlier. I know that there's a lot of new folks that have joined the conversation since we last had this chat. So I think it's worth reviewing some of the overview and definitions and personas that we had here. The link is in the meeting note, and I think we can take like 10 minutes to read through. We should stop reading at this point, which is right after requirements for discussion. Most of the requirements that are listed here in previous discussions, we were all aligned on. The one place that we didn't have alignment on a requirement was a having a mechanism for rotating a root key. There's both pros and cons here, and I've captured some of those in the document here. I'd like to kind of at least in the 20 minute conversation figure out, are there any additional considerations we need to take into place? And if any, what other data do we need to kind of make a decision here and move on? So I'll give folks 10 minutes to read through the link, and then we can start discussions. I wanted to do a quick time check. Do we need a few more minutes or is everyone done reading? Maybe a thumbs up for people when they're new. I'm done anyway. Okay. Does anyone need any more time? Okay, I think we can go ahead and get started. Share my screen again for the comments. All right, I see. Let me refresh. Okay, see a few comments from Steve. Should not to become, must not see the same comment throughout. Thank you. What's the difference there, Steve? Just there is two, I think it was two of them. This one was, let's see, signing an artificial require the perform additional actions. Yeah, must not. So basically, we want to be able to support offline signing. And if we say should, I'm worried that somebody would build a CLI that somehow enforces communication with the registry and kind of breaks that flow. So it's more a matter of to support offline signing. We need to be able to do it all locally. There was interesting conversation in the distribution working group related to that. And then the same thing for multiple signatures. I think that was, that was the other one. Okay. So this doesn't seem to the question of some, if you do it offline, of course, they could be revoked in the meantime. Is there some timeframe like you should, you need to be online every so often? Or is that just ignored or? Oh, sorry, the offline is, sorry, you're, you're, you're bringing the air gap environment. So that's a good conversation. So this is more of, I mean, the thermo build client. And we've been talking about the consumption of a thermal clients. And recently, we've, Brandon's brought up some good conversations around in the build environment. We need to be able to build in the thermal isolated environment. That's kind of like one of the twists on the protecting from the solar winds kind of scenario. So it's, it's signed, it's built and signed. And then, but it's not sitting there for weeks or months. It's built and signed and then pushed through a registry and a secure channel that there's no real time note into your point. Sure. I could be signing. That makes sense. And this one, does that need to be spelled out here? Or is it, would it be clear to everyone in that case? We probably can reread it and reword it in such a way that's basically saying the offline, I'll have to go back and reread it. And it's a great point. I think we should track that as a scenario. We do have a list of detailed scenarios that covered some of those use cases. I wanted to go back and visit requirements first before we do that. But I think that's the use cases so far have really looked more at deploying in the air gap environment, not necessarily building in an air gap environment. So those are some additional use cases, I think they can be added in. One other concern I have, which I think I've brought up before, is just the, the phrasing that the validation shouldn't perform any additional actions with the registry. Because obviously at the very least, you're going to have to pull the signature or the file that has that information, as well as any information about how you get that key and other things. I just think it's a little unclear exactly what that means. Yeah. It sounds like maybe the term, it sounds like maybe the term validating should be clarified by bringing it into more than one part. Because I think you're talking about validating as in, okay, we have, we've got the keys, we're comfortable with that they're up to date and then we want to validate versus I've got this new image and I want to validate it. I have to of course start by getting the keys. So. Yeah, exactly. I think I understand what you mean, but I feel like it could be very easily misunderstood there. Right. Yeah, I think that's a fair point. Do you mind putting in a comment so we can track that? Yeah, sure. I think to catch like we want to be able to say, I'm trying to deploy this thing. Can I get the signature before even getting like, you don't want to pull the container image scenario, you don't want to pull the image and bring the Trojan horse into the environment, just to find out whether it should be there. Right. The idea is let me figure out, find the signature, get the keys, maybe discover from the registry where to get the keys from that's something we haven't actually resolved yet. I think the pull, well, sorry, let me finish the thought. So you'd be able to get the signature, get the keys somehow, discover the keys somehow and get the keys somewhere and then be able to validate locally. Probably there's something in there about I shouldn't have to pull the artifact itself to validate it to move forward. The concern that I was trying to raise is probably more on the push side because there was this conversation that today the way the container clients work is the digest isn't actually computed in typical workflows until it's pushed to the registry and that kind of breaks the workflow that we're trying to articulate. So I think the build, the pull and the error gaps got commingled in here in a way that we should try to do in a clean way. So Steve, for your next comment, the additional signatures. I hadn't tracked this as a key management scenario requirement. I think this fits more into the signature format itself. Do we have a separate dock that's tracking that? Can you help me with what the difference is? Because I don't know how many signatures a publisher would put on the same artifact versus multiple entities would put additional signatures on a similar artifact? So being able to add in additional signatures, that's really dependent more on how you are formatting and storing the signature, right? And I think that's something that we want to capture in a requirement that says what are the requirements for the signature format itself? I think we have some requirements there in terms of where a signature is stored, how a signature is retrieved that don't necessarily tie in with the key management scenario. So we can create a separate signature format dock and I think that addresses some questions around what exactly are we assigning in terms of what bits and what does that mean in terms of like how containers get packaged. So I think that is an area we need to dive into. But I think that's probably a better place to capture this requirement than in the key management scenario. Fair enough, that's fair. Which makes the question, do we have a signature requirement stock? We don't have a signatures requirements stock as we have the requirements for the notary and then workflows, which that is captured in it. The signature is there to support the requirements, so we didn't really make a requirements, a signature requirements. Can we track this in the overall dock then for now? Yeah, I'll pull this out. I'll reject this one. I am curious, why do we need multiple signatures though for a publisher? Like is the requirement there that in the same signature file I'd have multiple signatures or can an author, like what I'm so little... I can throw an example. Yeah, I can throw an example. I can think of some client environments that might use signing as kind of an auditing method to say this artifact that they've signed has gone through the CI build. Okay, now went to the next step of it, went through a security team that went through a security scanner on that and a passive security scanner that might have its own signing on it. So I could see something like that within a single entity where they use signing for that. Okay, so that's more of just the multiple signatures as opposed to specifically multiple signatures from the publisher. It'd be a single publisher but multiple signatures there, different roles within the publisher. What I'm trying to figure out is there's something unique that it's the publisher that's signing multiples. Can we also be looking at signing multiple tags? I'm sorry, Brenda, go ahead. Can we also be looking at one publisher signing different tag names for the same artifact? What I think I'm here just to say can move forward is I'm thinking of hearing things more around multiple signatures scenarios as opposed to something to do with the key. So I guess we can move on. I'm just a little confused by this one, like is there something specific around the publisher versus everything that Brandon's talked about is completely valid, but I don't get it that it's tied to the publisher. So I don't think there's a difference. I'm so curious, who's CFC of distribution? Because I thought I was. I'm curious who else is. Can you elaborate a little bit more on the what this comment is about the ephemeral loop line? Want me to chime in on that one, Steve? Yeah, please. And so in another one of the discussions, I forget where we were talking about supporting ephemeral environments and being able to inject the keys into that ephemeral environment. And this is kind of hitting on the point that I put back to Steve over the weekend, which was that you not only have ephemeral environments verifying signatures, but you may also have ephemeral build environments that are creating signatures. And so we probably want to support both of those scenarios. Is there anything that needs to get added to this requirement to make that clear? Because the way I was reading it, I thought it would cover that. I think it's covered, Steve. Yeah, I think we might just want to tie it back with those words. Okay, create a loop. I think maybe this is something we need to add to scenarios to kind of go into more detail. And then Marina, on your comments, I think the clarification here is once you pull a signature and once you pull the artifact, those two should have, you shouldn't need to pull anything else from the registry is what this was getting at beyond just sort of like being able to pull artifacts from the registry itself. Okay, I mean, I don't understand this requirement. Like, is there any technical reason why that would be a problem to pull additional things from the registry? So we don't want registries to be a dependency for validating a signature. Registries can serve the signature, but they should not be in the validation path. Okay, but so there'd be additional, so you'd have to have an additional dependency outside of the registry to then pull that information? Potentially, yes. Okay, but is there any reason like there couldn't be the option to have that also on the registry just to centralize things and make it easier? So I think this is one we've had previous discussions are registry operators can act in multiple roles in the sense that they could, depending on how we align on the key management designs, there may be roles that they take on as part of it, but these should not be part of like what a core registry needs to provide. The goal here really is to make sure that you can move artifacts from one registry to another and have the signature still be intact and having a registry dependent workflow there means that you'd also need to replicate additional things beyond just the signature and the artifact. So this is really standing a requirement to make sure that step is not needed. Yeah, but I think that the requirement there is that images are able to move and that registries don't have to do any additional work. I don't know if the requirement is that you can't use a registry to do anything else. If you're using the registry to do something else, wouldn't that imply that there's additional steps the registry needs to perform to validate a signature? Well, the registry could be one option for additional steps. Because I think I could see some situations in which you wouldn't want to interact with the registry and you want to do a third-party service, but I could also see situations in which you wouldn't want to set up all these dependencies on third-party services and it'd be easier to just have that all in one place. I just feel like forbidding it doesn't make as much sense to me, even if it's not like the go-to solution. Yes, what was that we were trying to protect with this sentence? So if you have, let's say, an additional API or additional information that you need to go to a registry for, then when you move one artifact from one registry to another, you need the second registry to support similar functionality. And so this is essentially where we're building in additional workflows for signature validation. I think probably one way to address this is calling this out as optional, in which case we can say that some registry operators may decide to provide this additional functionality, but that would sit outside of what a registry operator is required to provide. So a registry operator, there's nothing preventing them from potentially operating as a CA if that's the right design forward, but that doesn't mean every registry would need to be a CA at that point, and we need to kind of make sure you're going back to the original CA that issued the search to validate them. Yeah, exactly. I think that makes sense. Basically, it's not whether they can't do it or that they shouldn't have to do it, and I feel like that's the distinction. Okay, so I'll clarify the language to make sure this is optional. Isn't the should not already assigned that this is optional? If we convert this to a must not, then that is a problem. But if we keep the should not, it shouldn't, it can, it's up to the implementation, right? But if you're going to be more specific, we can add an optional on in front of it. So basically, we'll just try not to have that very coupled thing we did with no Rewind, is that the goal here? Yeah, essentially, you wouldn't want every registry to have to set up a notary implementation and then plug signature data from one artifact from one registry to another for that. I'll explore making this clarifying this language a little bit more optional. I think we're aligned on what this requirement needs to say. So I'll track this in the comment and think we can move on. Okay, were there any other comments on the requirements? I don't see any other in posted in the pull request. I'm in there, but that might have shown up in a different place because I think I probably clicked review instead of comment. See, I don't see any more. Could you maybe go through them and I can. Yeah, they're over in the requirements file itself, but it's just not showing up in the comments. So I don't know if I just clicked the wrong button. Yeah, I see a bunch of comments from Marco. Trishon. Can you send me the link to your comments? I see comments from the previous iteration. Let me pull what Brandon put in. Yeah, I've got everyone in there where I was putting the comments directly in the scenarios file. I'm sure I clicked the wrong button. Get up, does that to me now and then. So on 28, the only comment I had there, line 28, just that the push is going to limit the number of APIs we can call. And I know we have APIs for things like querying and other things like that that we might want to be able to run for some of these things. So just want to catch that little piece in there instead of saying just a push and unsound artifact, but there might be other things to query or pull or fetch and unsound artifact to elaborate pulling or pushing. We definitely want to solve the problem with no review when we're, if you, if you were turned on signing or not, you shouldn't get different content, but the digest should always be the same content regardless. Yeah, the actual manifest you pull back to be the same, but the scenario I'm thinking of here is that signing an artifact may require that we query the registry to say show me all of the tough targets metadata for this thing. And so you have to run a query against it to pull that back. So you can send update for the newest tough art, you know, metadata. I would worry about that when we do the implementation, if we did it that way, but I'm not, that's what I'm trying to figure out. Like if we were to do anything with time stepping and other things, I'd assume it's the registry that have to maintain it. Yep. Yeah, I think maybe that's also something to cover in the signature specifications in terms of what we actually end up signing. An ideal situation is where you're able to sign the bids before they've even gone to the registry, right? So having to query registry for data before you sign it might break, might reduce some of the things we want to cover there. So I think that's definitely something we want to push up in the signature format discussion. And this, that's a good point where we might be separating the pushing of the signature from the doing the sign itself. And so you can sign it locally to all that. And when you push the signature, that might involve a couple back and forths of saying, okay, not only do I want to push the signature, but also want to push an update to the targets metadata on all the signatures up there, right? I would say it's implementation detail. I still think that would be a, there shouldn't be any client interaction to do that. Like what do we want to capture from requirements? Again, we're not trying to spec an implementation. We're trying to make sure the requirements are capturing the constraints we want to work with them. But I think part of what NIAs had written around the additional integrations with the registry, we're trying to protect from us building this, I don't know, I don't know how to describe the sidecar dual implementation thing. But then we just stick to the requirements. I think we'll be okay. Yeah. And I think I found the button that has missed. So my changes might show up now. All my comments. Oh, yes, they show up. Nope. When you say starter review, suddenly there's everything goes into a holding state. Another one of my comments, line 37 for signature validation must be enforceable in air gap environment. Kind of goes back to another comment I had earlier of does this handle checking for the key revocation? Yeah, it needs to. What we had done when we jumped into the scenarios in the previous iteration was call out what air gap environments would need to do additionally to enable that signature revocation, whether there was copying revocation data over and a push versus pull model, I think their design considerations there. But the overall design needs to make sure that it can be enforceable. I think there's a trade-off decision here in terms of like, you know, how quickly we can refresh that data, but a mechanism to refresh that data needs to exist. Okay. So it's not immediate as soon as you revoke a key that it's immediately discovered within air gap environment, you're saying there's going to be an extra process that could be done in that scenario to make it possible. Right. Okay. And then as we're moving between different repositories, make sure I'm on the same page you are. As we're moving artifacts around the concern I have there is more when you're signing tags. And it's, I think it's possible it's just something that's on the client side. So I just want to make sure that we have the same thought process that they'll probably be a client side process to handle verifying tag signing. Just trying to do that on the server feels like it's almost impossible or in the API itself. Yeah. I think we had questions around tag verification before around sort of like, you know, whether, what does that offer? Do we want to enforce digest verification only? And that might be more a conversation to have in the overall requirements because that does I think leads to some challenges. Yeah. And we've got an issue for that one opened up already. So that's probably best handled in that issue. I just want to make sure that we solve that when you're looking at this one because it could have an impact for this. Okay. And then I'm getting into the root key rotation stuff. I will save this comment because I think that's going to be the larger conversation we want to have. Yep. And then back to the push comment. We already discussed that one. We've already discussed. Okay. Did anyone have any comments on the up to sort of like the requirements here that we haven't addressed? It's before the requirements for discussion. Okay. Let's move to this requirement then. I think there's both sort of like pros and cons here. So Brandon, since I had a comment here, do you want to elaborate on what your thought process here was? Yeah. The thought process that once you lose access on that root key, you're pretty much housed in terms of security. Someone can make any number of delegate keys they want to do signing on. They can sign any artifacts they want to. And so your repo is pretty much done at that point. You're going to be forced into some kind of out of band update of the root key and getting that pushed out to all the clients and doing the revocation and everything else on that. So you've got that headache no matter what. And so I'm trying to understand the value add of not allowing an in band rotation of a root key. If the concern is someone could also potentially, hey, they can potentially use that hijacked root key to make new root key signatures. I think you're already in trouble with the point when your root key gets compromised. Yeah. I think that's a fair call out. The way we were thinking about this earlier is that your root also comes with certain information associated with it. So if you think about standard X509 roots, there's potentially a CRL there that you can go to and that CRL can give you an invalid response where you now know that the root has been compromised. So there's a mechanism that exists in tying a specific root as long as that root also has some kind of like a mechanism for sharing revocation data where you can notify everyone that's using that root that something's broken, you need to go update this. The challenge with having an automated update mechanism there is that now the attacker could potentially have a new root with the new CRL, which now they're vending, which means that you lose that capability of breaking your end users whenever you have that compromise. So that's the I think the tradeoff to kind of consider there is that would you prefer to have that automated mechanism of letting your users know something is wrong and then have them decide what next steps they need to take versus you want to take that step on their behalf and then if something does go wrong, you then have to kind of do a mass communication to kind of get that messaging out. So that's really where that tradeoff boils down to. So there be a way when you're thinking of doing these in-band rotations that you could say, hey, if we revoked one root key that that also revokes all the other ones that got pulled into an in-band rotation process. I think we need to think through that because what you're essentially doing in that rotation is technically not a rotation as much as you're issuing an intermediate that's also acting as a root. And so then yes, you could revoke the original root and that could potentially trickle down, but that rotation doesn't really do much in the sense that you still have to go back and check what the original root is signing off on. So what does that rotation really then give you? Well, I think it gives you that the implicit revocation of key is so you can have your root key time out eventually. And after that time out, you can create a new root key seamlessly to users because you don't want keys that last forever because anything can be broken in forever. Right. And so I think the way we've typically seen root rotations handled is you start using a new root before the current root expires. And you give enough time where the two roots overlap rather than signing the new root with the existing root. Authenticating the new root with the existing root means that you're saying that this root, you should trust this root because it was signed with the original root, but then to know if there's been any revocation or anything's happened, you actually need to go back and check, still go back and validate whether the original root was valid or not. So you're creating a chain there, but deciding to stop partially up the chain in terms of validation, which didn't really make, I don't think quite addresses the thread vectors there. Well, the scenario is I'm thinking of where you have a user that went through the process to trust your original root key. And then you go through the process internally saying that we need to update it because the old one's timing out or we need to sign it because we want a higher key length on our root key. You know, they're a long list of reasons you might want to rotate a root key. Right. And so for those clients that have the old root key that they are currently trusting, do you want to provide an automatic method for them to be updated? And so I'm thinking of all those ephemeral builders that might have a key injected in there, should that automatically say, okay, I trust the old root key, so therefore I should go ahead and trust the new root key as long as I verify that everything hadn't been revoked in the process there. Yeah, and I think that this doesn't exclude the possibility of an out of band rotation if and when that's needed. But I think in most cases this question can be solved with an automatic rotation. And then only when things go truly, truly wrong, do you have to bother with all this out of band communication? But having support for this mechanism allows attackers to take advantage and set a new root information is the trade off here, right? And so if you have things like revocation information or other things built into the whatever the root key is signing off on, that is no longer verifiable and that no longer can be used in a mechanism to stop as a stopping function, right? I think that's the trade off here that we need to address. Yeah, but if the only reason the client trusts the new root key is because they're trusting the old root key and you revoke the old one, I feel like that could potentially propagate through as long as we design this right. And so the question then becomes what are you solving when you say, okay, the old root key got compromised and that's potentially exposing everything well until you send out that revocation out there and that gets approved, everything's vulnerable at that point. Doesn't matter whether or not you sign a new root key or not with it, you're already vulnerable. Yeah, I agree and I would say like what does issuing the new root achieve there, right? Because if you're going to still go back and have to chain back to the original root key, you have a technique, yeah. Yeah, and that scenario once you're compromised, I see you're going to have to go out of band. I don't think there is an in band way to resolve that situation. You've been compromised at that point, you have to send a new key out out of band process. But I'm looking for all the cases where you haven't been compromised and trying to make scenarios more useful for the users where they don't have to go through all this out of band process, just do regular maintenance on the stuff. There could also be a question here. I think that we may want to look at in terms of who owns root keys and what do root keys signify, right? Like if you think about the traditional model of like CAs issuing certificates, like a CA essentially ends up holding a root, in which case individual developer search rotations, those things can happen more frequently, right? I think that's probably a question that we haven't addressed here. And I think there may be more of a threat model that we need to go into with those roles in place to kind of determine what is actually the right mechanism here. Is it a more secure update if you say, for example, like a set of trusted roots within a Docker client that you could push out that has certain trusted entities that may be a model that that's more easier to update roots with. So I think there is something more to be had here. Do we want to take a pause in this and come back and have like a more detailed doc on this? I think there are more things to consider here than what we've discussed. Let me throw one quick comment out and then I would say, yeah, if we want to go more detail, that makes sense. I'm just thinking from the scenario of just updating a Debian laptop here. I don't have to, when Debian says their old root key is coming close to expiration time and they send a new one out there, I as a user don't have to do anything. I just do my regular package updates and it pulls down the package that has the current root keys. And because I trusted the old key, that got pulled in and trusted and approved and signed everything. And so it makes a much nicer user experience when I don't have to go through a whole lot of external out of band processes to get my clients to be able to work with a new environment when they already worked with the old environment. That's a great example of the key discovery and acquisition workflows. But not only, we've kind of never really answered it yet, but if we say we're not going to worry about it for now, that's fine. I don't have to worry about how I get the Docker or the Web network signature for that matter. But what you're talking about there is that problem as well. So to be able to, we don't want to do trust on first use, but I think once you have a key, to be able to say I can get updated versions of the key seems pretty important. Yeah. Is it fair to say we need a mechanism for key distribution, right? Where we can get these updated keys that that's really the requirement to work backwards from? I think so. I think once you have it, to be able to maintain it is important. I don't know, like, how do we do the update scenario without getting into the tofu scenario? I think that's kind of it. Yeah. We can, we can look into that as we can start going to scenarios, but I'll update this to kind of capture what the overarching requirement is without making any specific calls here. So just to keep to our timeframe. This is great. We've been trying to get to a good key management conversation for a while. So this is really, really helpful to make all this great progress. So we'll keep it going. Let's try to get for what we can do to get it merged. So the actual document reflects our current state, then we'll have to read through all the notes. So I'll do my part to make sure my comments are in a place that are obvious to be mergeable. And if we can do that, you know, it'll be great as well. Just on the point of order to get the status update out, was there any other feedback? I incorporated as much of the feedback as I could. On the status report, it's in the HackMD doc. Let me just pull that up real quick. And so everything from the white backgrounds for the dark background effect to there was some good conversation around balancing to the example commands provide too much detail that, you know, should I pull that out? But I thought actually there was a really good conversation that came out of it as a result of the ordering of pushing digest before their tags so that we can get two independent entities pushed to the registry. And then later do the tag update without trying to make some master transactional boundaries. So to me, that kind of felt like a good balance. But if there's some more in that way that we need to do, that'd be great. I think there's a some of the other there's two other pieces of feedback that was one, it looked complex. And two, are we really making progress just putting a status report doesn't really show progress. So the complexity ones an interesting one that's part of I can use more feedback as far as making progress. That is another part that I wanted to talk about some timelines and things. But before I talk about the timelines going forward beyond what I put into the status doc was ready for feedback we wanted to talk about before we merge and publish. Okay, so on the timelines, let's transition to that. Because that is the piece I think a lot of us are getting in a difficult spot going, you know, for the ones that are working on it, we know we're making progress, we always want it to be faster. Outside of it, there's a lack of confidence, because they're not really it's not obvious. And people are starting to spin up other efforts that it'd be great if the other efforts were actually as holistic, but they're much more sandboxed and don't really promise the cross registry solutions we've been working on. So after this doc, I have one thing related to teleport that I have to go finish up. But once this is my next thing after I want to be able to outline the three, I think it's three places that we need some investment that could use some help. One of which was the validation with open gatekeeper. We talked about doing that. The teams in Azure that are happened to be part of open gatekeeper, they would like to but they're not going to get to it right away because of other critical work they're on. So if there's other people that are at least close, if not in the gatekeeper open gatekeeper community, we'd love to prototype that. The reason I really wanted to be close to the open gatekeeper community is I want to avoid us. And it goes for all these things. Building something that works around what exists today, as opposed to this is what we would like to see changed, whether it be open gatekeeper or notary. So we really want people that are close enough to say, yes, I know enough around open gatekeeper to know that this would really work if we made this one change to open gatekeeper, as opposed to let me manipulate this thing to make it work only because I don't think I can change it. These are all projects that are just code. We'd love to make changes to it. So yeah, I am referring to open policy agent gatekeeper implementation. So basically the idea is we want to validate this with something very generic, like Kubernetes, because it's a very open framework, and that we can actually validate the end-to-end review. And if we can validate the end-to-end with a true policy manager like that, then we assume that we can do with other projects, whether it be cloud-specific projects or other things. That's the premise that we're doing there. So that's prototype, that's effort one, is to get something going there with the validating the next end-to-end scenarios. The other one was the CNCF distribution changes. We'd like to see some changes there related to the artifacts manifest spec that we've been working on and a list API that we can get the link list for the signatures out of the registry. I've been assuming that we were going to do that from our side because we were kind of already prototyping on that and we would make their next round of iterations. It doesn't have to be us. And once we have the artifact spec, artifact manifests more detailed, that should be easy for others to build in. And then of course, the updates to the MV2 client. So those are the three things. There's something here for the person who won't identify themselves. So challenging to even acknowledge them. Anyway, we are in time, so we can talk more about it. But I did get at least one plus one for the just publish it, let's move on. Because I think a lot of people keep on asking, I have not been able to say, here's the written status of where we're at. So I'm hoping this captures it. With that, I'm sorry, looking at the comments. Anyway, with that, we'll pause the recording, we'll keep going and thanks for everybody's progress. Next slide, folks.