I am joined by three outstanding instructors, also all from Canada. We have Sean and Jana and Fiona. And why don't I start off quick by just having each of you introduce yourselves and maybe discuss just briefly some of the classes that you teach. So why don't we start with Fiona? Okay, thanks for having me today. So I am a lecturer in the Faculty of Math at the University of Waterloo. And I've been teaching calculus one and calculus two for science students for the past year. And before that, all kinds of linear algebra for engineers and math majors and mostly first year undergraduate courses for math students. Excellent, welcome Fiona and Sean, do you wanna go next? Sure, yeah. So I'm an instructor at the University of Lethbridge. And I guess my teaching is mostly in calculus and in linear algebra. And I do a lot of work on the side with open textbooks. Yeah, very cool. Your pretext stuff is really exciting. Jana, tell us a little bit about you. So I'm also an instructor at the University of Lethbridge and I teach mostly first year classes Friday, a lot of statistics and a lot of linear algebra recently. Okay, great. We also have on the call, just monitoring the call, my colleague Casey Hudson. So if members of the audience have questions or comments, please feel free to use the Q and A or chat below. And Casey will help me find those and propagate those forward for discussion by the panel if there's time and so forth. But I welcome all of you to our webinar. This webinar will be recorded and shared subsequently. So if anyone in the audience wants to share it with their friends, that will be possible in the future. So I'm really excited to talk about education after the pandemic. And I have a few themes that I'm gonna try to go through in some questions. And just to kind of set the panelists up for these themes, I wanna talk about students. I wanna talk about learning outcomes. I wanna eventually talk about assessment and kind of what your future state after going through the pandemic might look like as an instructor. And we'll work through those in the coming 55 minutes or so. Sound good? Okay, so I'll start it off with my first observation. After maybe six or seven years of being an instructor, I had an epiphany. It was too late to have this, but it happened. So I used to think that if I was teaching a fancy class like general relativity or if I was teaching partial differential equations or calculus, I kept thinking about like the chair of my department or the person that might be reviewing me for tenure, wanting to make sure that they see me as the expert on that subject. So I was entirely focused on teaching subjects. And then the epiphany was, that's not the point. The point is to teach the students and you should be focused on the students, not the material. It took me a while, but eventually that really changed the way that I taught. And so with that lesson that I learned late and that I love, I wanna ask a few questions of the panel about students. So through the pandemic, students have been interacting together and with us in a new way, not in a sort of standard classroom environment where maybe the playing field is level or maybe the playing field is not level at all, depending on how you look upon a classroom. So my question is, did you get to know your students differently? Did you develop new empathy for your students or understand their lives in new ways because you had to peek into their lives in a new way that's not the same as in a classroom? And I'll start that off with Janna. Do you have any views on insights into your students as a result of the pandemic? I think that we definitely got a better view or maybe just a different view of the students. I know that I met with different students, especially having the experience last March when we switched from in-person to online. I met different students. Different people would be willing to come to office hours. It was easier for some students to come and pop into the Zoom office hours. We were seeing a bit more of them as a person with an external life. There were more kids joining in, doing work next to them or being stuck down in a dark basement because the rest of the family was upstairs. And I think it led to a good empathy between the instructors and the students and it went both ways because they were seeing us as humans who have lives who aren't just living in our office somewhere on campus. They were seeing us as people who have other demands and children sometimes running in to say hello. And we were seeing that as well because it's easy for us sometimes to look at things and think, oh, well, it's only an assignment. They have time to do this tonight. When, well, if they have three kids that are pulling on them in office hours, they don't have an hour to do it tonight. They'll get it done sometime this week but they have demands on their time and we do too. That's interesting. It's like you characterize us all as humans rather than titles because we have these multi-dimensional lives outside of the role that brings us together in this professional environment. What about you, Sean? Do you have any new insights into your students? Somewhat, yeah, I mean, it's interesting seeing, because a lot of my students, I never made my students turn their cameras on and so most of them didn't, but there were always a few that kept them on and seeing you're like seeing into their homes sometime. I have one student who, you know, he kept his camera on all the time and even though like his mom was doing dishes behind him, you know, like he set up at the kitchen table and you could see them like doing, you know, things going on at the background all the time, which was kind of interesting. But yeah, I mean, and we also saw, you know, the students like the times when the schools were closed, you saw like they were there and they were attending with their kids and we had kids at home too, right? And so, yeah, you're realizing that they're having some of those issues. And I like that you opened with this, you know, characterization between teaching the subject and teaching the students. There's a quote from Francis Sue on that that I put in a talk I gave this summer at CMS, right? So in his Math for Human Flourishings book, he says, too often those of us who teach professionally say my job is to teach math as if teaching math were only about the facts and the procedures, we forget that my job is to teach people whose experiences often interact with mathematics in completely different ways than our own experiences do. And I think that's very true, right? Like the way the students are experiencing mathematics is different from, you know, us as professionals and even when we were students, right? A lot of us make the mistake of wanting to teach the way we were taught because the way we were taught, hey, it worked for us, we're here, right? But the way we were taught was probably not very good for a lot of the people that were in class with us and we just, yeah, as students, you don't notice that. Thank you, thanks for that quote from Francis Sue. So Fiona, I wanna make a variation on the question just slightly for you that picks up on one of the themes that Jana said. So the traditional classroom might be an environment where, like Sean said, the four of us thrived, we did well and now we're instructors, but it may not have been an environment for other students to thrive. So the pandemic gave us an opportunity, maybe, for some students that wouldn't have otherwise flourished in the classroom, maybe they flourished in the online, maybe some introverts got an opportunity that used to be filled up by extroverts taking up all of the classroom participation space and so forth. Does that resonate with you? And do you see ways that we can maybe reach more fully some students through a combination of ideas and methods that we used during the pandemic? Okay, I'm going off on two longer questions. So I'll stop and maybe the gist was to convey. Yeah, I mean, this is very top of mind as I'm sort of preparing for the fall term and thinking about what this kind of combination or blended method of learning is gonna look like, but I do think back now to that time in the classroom where it set a specific time of day, you're there for 50 minutes, the students are almost locked in in a way, right? They would never get up and leave, like they're in there for those 50 minutes and no matter what's going on in their lives or in their brains, like they're just there and they have to absorb the learning in that time or that's the expectation. And it was the same with me. I needed to be on for those 50 minutes to my best self and I've got to throw everything out and just be present. And I actually often did find that I was, like I would almost forget what day it was, what time it was, I would get very absorbed in the material for those 50 minutes. And so it was quite different to then go and prepare a really short video and post that and know that students could be watching this, I don't know where, I don't know when, what time of day and also that I'm sort of competing for their attention, right? I mean, maybe I was before in the classroom when I just didn't realize it, but for the most part I could see if they were looking at me or they're engaged, you'd get this sort of interaction with them or nods or hands in the air or something. But now knowing you're putting this out there and you're competing with their, I don't know, they might be texting friends, they might be then switching from my contact to, content to something on TikTok or checking their Instagram. And it does make you think about sort of the quality of the content that you're putting out there, finding ways to engage students now that we are kind of competing with all of this other stuff where previously you've got them locked in and loaded and you're just gonna get the stuff out there. That was your first concern, right? I gotta stay on track, I've gotta get all the material in in this 50 minutes. And now I feel like there's so much else that we kind of need to think about. Well, there's a lot to unpackage there, it kind of moves into the second theme that I forecast at the top. So for a given course, there are target learning outcomes that we hope for our students to achieve. And along the way, there may also be targets that we hope to achieve as instructors helping students to achieve those goals. So have your understandings of the learning outcomes in the courses you teach to transformed because of the pandemic or the learning outcome static. And it's just really a way to get there. What about you, Sean? Do you have any insights on impact on learning outcomes? I guess it depends on what you tend to lump in with your learning outcomes. I know that in a lot of cases, the learning outcomes is kind of that list of content or competencies or the thing, the bits of math that the students should be able to do by the end of the course. And that maybe doesn't necessarily change how you might get them to the point of being able to do those bits of math. That's the part that I think is going to change more than the outcomes necessarily. But I think, yeah, especially, I think most of us here, we teach a lot of these kind of large service courses where they're prerequisites for other things. And so you don't have the freedom to kind of just throw the course outline out the window and do your own thing because they need to know certain things for the, when they get to the end of the course because they're going on to some other course where they're supposed to have that knowledge. That said, I mean, how much do they retain? Like if your outcomes are these knowledge points and they learned it long enough to kind of answer a question on the test, I don't think that necessarily means that they're going to know it when they get to the next course. I was really affected by some of the things that you shared on Twitter maybe about a month ago where you shared some of the feedback that your students have given you. And one thing that really struck me about you as an instructor, as, you know, vivid by your somewhat humble brag on Twitter was that the students, many of the students who wrote about you said things that weren't about learning outcomes but rather about a change in attitude that they had about mathematics. So your instruction seems to have allowed your students to see themselves as mathematically capable even though before the class they perceived lots of fear and anxiety. And that's not necessarily a target for a calculus class, you know, no longer be afraid of mathematics but you appear to have achieved some of those unanticipated but maybe even more fundamental outcomes. So bravo to you. What about you, Janet? Do you have any perspectives on changes of the target learning in your courses in light of the pandemic? I think I mostly agree with Sean for a lot of the classes that I'm teaching kind of depending on how you list those learning outcomes some things are staying the same. I still need my students to know what a P value is at the end of their stats class and kind of how to run through a hypothesis test. One thing that's changed a little bit is it used to be that you think about, okay I need to teach them this list of things and to teach them those are what I need to put on the board in lecture, that's what counts as being taught. But then as I'm moving things online and putting things on Moodle then all of a sudden it's the, well that's not actually true, right? If you do an assignment and you have to work through the concept and you have to kind of use that that's interacting with the material more than me standing on a board or recording a video where you can passively watch it. And then it became a question of if I'm thinking about the learning outcomes it's not these are the things that I taught and I put them on a test and I asked a question about it it's what things did they really get to interact with and really work with in some hopefully sometimes in depth way some things will be just facts but how did you really work with it? It's interesting, it almost sounds like you kind of characterize yourself as more of a coach and less of a sage on the stage so to speak where you don't necessarily have to do everything that the student is supposed to do once in front of them but rather you expect them to also go beyond the material that you specifically lecture on to allow them to fill things out, I quite like that. So Fiona in your last comment you made some remarks about kind of the attention war if you will, it's really hard to get students it's hard to know if students are paying attention in the online environment especially when you're doing things in an asynchronous way where they might be watching or not watching a video in contrast there's this proxy for actual engagement when they're in front of you in class and it looks like they're not looking at their phone looks like they're paying attention and so this kind of attention is one measure of engagement maybe as a precursor to a learning outcome but do you have other insights on maybe the way that you are helping your students achieve target goals did the meaning of the learning outcome in your classes transform at all? So I guess in a couple of ways I remember when I was first starting to teach online last summer the pandemic sort of inspired me to try to come up with ways to make calculus and math relevant I mean, we know of course it is but this sort of gave an opportunity to show some first years these were science students so to put it right into their world I made up a little assignment on the SIR model just giving a little intro into differential equations and what are all these reproduction numbers what do all these things mean that we're hearing about in the news and even simple things like the logarithmic scale that would be talked about in the news and does anybody really understand this? These are some really basic kind of calc one topics that we can talk about with our students so that was one thing that I guess it wasn't technically a learning outcome of my course but the pandemic provided an opportunity to kind of take some of these learning outcomes and apply them to the real world so that was one way another thing that I realized just back to the whole grabbing their attention I almost found that so I did spend quite a lot of time making these videos and then in the fall term I actually taught the same course again which was great because I had all these videos but I got some feedback from students in the fall term that it was almost there was too much content because in my videos I went through like how I would do a lecture let's introduce the topic and I broke them up into smaller pieces but there were I don't know over 200 videos for them to watch and what I realized is like some of that content they can get that anywhere and maybe it's more exciting than the way I'm presenting it but like what can I bring to them that isn't just definition, theorem, you know and so what I got from some students was they want examples, right? They want me solving an example and maybe an example that takes a couple of topics and combines them into one question so that's sort of something that I've been kind of working on quite a bit this year is saying, okay, forget it it doesn't need to be this complete set of video lectures it can just get up there, do an example and throw that together and that's the kind of stuff that I've been kind of working on this year to improve these lectures but that was one of the things to keep in mind like there's so much out there don't do too much, you know kind of dial it back a little bit so as not to overwhelm the students. It's interesting, while you were speaking I was also remembering that you're this fabulous pianist and I remembered that I remember this analogy that occurred to me I don't know, 10 years ago so let's imagine that the Cleveland Orchestra made the definitive recording of say the Beethoven 5th so are we done? Do we need to have another orchestra ever play the Beethoven 5th again because we've got this incredible performance and by analogy, you know, Lopital wrote a really good book and then many, many other people have rewritten that book and monetized it and now other people like Sean are rewriting that book and taking the money out of it and making it more freely accessible so do we need to teach this stuff all the time? Like calculus is old, why do it? It's kind of like Beethoven's 5th but I think Fiona's example shows that the relevance of calculus today because of SIR as a model for infectious disease propagation it brings this classical material to life in a relevant way and the instructor kind of shows how this classical stuff this infrastructure for logical thinking is foundational to the way that current society evolves so I quite like this kind of like recycling lectures from even the semester past convenient as it is for an instructor might not be ideal given the possibility to tell things that are connected with today. Does that kind of linkage with society show up in your list of learning outcomes or is it kind of a corollary of the way that the class is planned? Fiona? So in the calculus for science students there are learning outcomes there about having specific applications and there are your typical ones about population growth and volume and pressure and some chemistry type applications they're sort of the standard ones that you always sort of hit but this one seemed it is obviously related to biology but this one just seemed so it's almost, yeah, just a spread of disease and have it be so something that we were all kind of experiencing right now it always seemed just like this theoretical example right, it didn't seem sort of real so it was, I mean awful that there was a pandemic but good that it provided a way for them to see the relevance of some of these topics. That's actually Jenna or Sean do you have any comments on this kind of linkage with the now using concepts from the past as a part of your role as an instructor? Jenna do you want to go first? Go if you want. Well, I might just say that I'll comment that just an admission that this is something that I probably need to improve at I'm still, I tend to teach more or less well, let's not say it's Lopital's course but I'm not deviating too far from the standard cannon when I'm doing a lot of calc and part of the problem is you can kind of, as you kind of progress as an instructor you can change one thing at a time maybe two maybe sometimes I get carried away and I try to do like three things at one time but you can rethink how you want to do your assessments or you can try to rethink some of the ways that you're relating to the students or you can rethink the content but it's hard to do those all at the same time. Yeah, and it's especially hard and bigger maybe at like larger universities where you have like a cohort of like teaching focused instructors that they can maybe work together and pool ideas and share resources if you're somewhere smaller and most of your colleagues are still research focused and they don't really have the time to sit down and think about, oh, how are we gonna teach our core service courses? How are we gonna rethink them? They don't want to rethink, they just want to, well, when they want to avoid those courses if they can that's why we're here I guess as instructors but yeah, so it's tricky. I think it's hard to do them all. I think there's some kind of larger community movements building up like the first year math and stats in Canada is one group that's kind of done well at this and also I think that the first year math people are we're slowly infiltrating like the CMS education committees. So maybe we'll start seeing some changes in resources being shared and those sorts of things but yeah, it's, you can't do it all at once I guess is the trouble, especially with time and resources being as limited as they are these days. I read a blog post recently about this concept called an innovation token and so the idea is if you wanna like install you know, a new communication system or you wanna change something in a group that you work with it might seem really easy, just turn the damn thing on and it's gonna be fine, everybody's gonna use it but there's a cost to learning anything new and that cost means that you can't really do too many innovative things too quickly because it just becomes overwhelming. So I wanna move into that in a little bit and I wanna talk about obstacles to change that you might be experiencing within the university structure but I do wanna give Jana an opportunity to chime in on that last comment, you know connecting old topics like linear algebra and calculus to modern things. Is that something that you try to do in your classes or is that not so relevant for you? It's something that I do to some extent and I don't always necessarily go with the I want to have the most up-to-date modern example but something that at least is interesting and relevant in some way. For one of my courses because I teach in the liberal education department sometimes one of the favorite topics for kind of getting into the math mindset is looking at my numbers and playing around with kind of mathematics and in that point of view and I've spent, I don't know how many hours searching for current data sets for teaching stats but still the Titanic data set is one of my favorites because at least it's a we all have this cultural understanding of what went on, what was going on and there's a bunch of statistics we can do with this actual real data. So it's definitely a push to have real world examples with actual I'm not just making it up this is the actual numbers whether or not it's the okay this came out last year but an interesting useful story from 20 years ago sometimes it's better than the okay this is data from a study that came out two weeks ago but it's kind of messy and isn't quite what I want rather than the here's a nice simple one from 1930 and looks up that it's out of date but the stats and the story still works. I'm gonna jump back in just briefly because when you mentioned linear algebra that reminded me that I guess one of the other changes that I've made now next year I don't have any linear algebra on the timetable but I've taught our sort of upper division linear algebra course a few times and one of the things that I try to do there was to say well you know it's really silly that we're teaching these advanced concepts in linear algebra but we're restricting ourselves to like two by two matrices because we want the students to be able to do these calculations by hand on a pencil and paper test and I'm like this is stupid I wanna teach things like singular value decomposition and I wanna do it for like a 10 by 10 matrix and well so why don't we do that why don't we write a little bit of Python code or something or you don't even have to write code because there are packages that do it for you and I thought I mentioned that because I mean I know that's one of your hobbies as well as some of the Python and the Jupyter stuff and we've leaned quite heavily on our access to our Jupyter Hub here at UofL to deliver some of that stuff. That's cool you know you've invited me to stand on a soapbox for a minute so I'm going to go. So I worry about our subject becoming increasingly baroque and especially among the researchers that work on narrower and narrower and more technical things and you know don't remember that Gauss solved differential geometry problems to figure out how to tax agricultural lands that happen to have a hill and you know a lot of the really important mathematics that we work on was really built by natural philosophers who were trying to understand the world you know like geometry and we don't play with satellite data like it is earth measurement like never before and geometry classes aren't looking at satellite data very often but climate change and all these other things are problems that are going to require solutions from people with mathematical thinking so I have a very broad view of the mathematical sciences and I'll now step off of my soapbox. So going back to the things that we've discussed so far we have a new kind of multi-dimensional perspective on students we understand them more as human beings who have lives and sinks full of dishes and sometimes kids pulling on their ear and we've also considered the target goals that we have for our classes these learning outcomes and maybe some adjustments on that but a lot of that is kind of static because we know what we want to teach in the context of a program. So my next kind of series of questions is how do you know if you're achieving the goal? How do you know if your students are meeting the mark? What do you do? One proxy for that that Fiona referenced was when she teaches in class she can see that most of the eyeballs are on her and on her lecture you know on the stuff on the chalkboard but that's only one signal. So how do you know if your students are achieving the target goals of the class? Do you want to answer that first Fiona? Sure, only because you mentioned the eyes on me or the hands up first I will mention I actually used this platform loom to record my videos and it creates a link and I can actually see how many students have watched it and they can even leave little reactions at certain points in the video whether it be a thumbs up or a thumbs down or a yeah or a laugh or whatever. So that was one sort of way that kind of replicated the, at least I could see okay I'm putting these things out there because I really had no idea how many students were gonna look at them. So it was a bit of reinforcement. Okay, somebody's looking at these things but then I guess to get into, you know are they achieving goals? I mean, I think we must be heading towards assessment because I feel like that's gotta be the natural place. And again, we kind of moved very quickly from in a lot of my classes which were many paper-based quizzes, tests, final exams to all remote assessments. And it was difficult. I mean, I experimented with some variety of types of assessments. I tried some group assessments and some students liked them and some said, no, I don't have time to work in a group. I'm working full time. I'm not on the same schedule as some of these other people and there was some pushback to the group assessment from some and there is this concern in universities that if we're not doing a proctored paper assessment then students are cheating and we can't trust anything. I really couldn't focus on that when we were completely online because I really didn't see what the alternative was. So I did do lots of remote assessments. I did lower weight assessments. I didn't do a final exam which was kind of new for one of these first year courses. I did term tests and frequent weekly sort of mini assignments and had students sign a academic integrity statement and hope that that was enough. I know I'm getting into a whole other ballpark here but yeah, lots of low weight assessments was what I was looking at to try to see how students were doing. Yeah, there were many themes there in that response that I wanna dig into. So there's these concepts of mastery versus getting a good grade. And as an instructor and in this role as an entrepreneur I would love for technology to assist instructors in bringing students forward with mastery. And I'm less interested in just ticking the box so that somebody can pass the class without necessarily really achieving knowledge the way that we are supposed to be imparting. So I'm really interested in that kind of theme of, it's one thing to just hand out grades and have the registrar be happy that there was a proctor involved with the exam. But that's not the same thing as really helping my students that I'm newly empathetic with to achieve certain goals. Janna, what do you think about this thread? I mean, I definitely had the usual issues of how do we suddenly transfer things to online? It definitely has led to a common theme of reducing kind of the big high stakes assessments. My course I'm teaching right now is actually the first one where I've done it without tests because in an online environment I'm not sure what the difference is between an assignment and a test apart from for one, I'm trying to mimic what I would do in class. It has the benefit of, there'd be oftentimes in my stats classes where I would be teaching something like normal tables. And I'd have to teach them a way to do it on a piece of paper because that's how I would have them write their tests. And then the realization, well, why am I teaching them how to find a number by looking it up on a piece of paper that I photocopied for everyone? I wouldn't do that for anything in math class. And finally, well, if we're doing it all online I can just let them use R and write things in the notebook and they have the code available to them and they could fix it live while they're writing their tests. But it took a surprising on my part a surprising amount of time to accept that we're still talking about the normal distribution you're still learning the same amount of stuff you just haven't learned the esoteric skill of looking up numbers in a weird way along a table. But it seemed like that was, oh, we're cutting that out but really it's the typing it into a program that makes a lot more sense. I'm not gonna make them learn arc tan by pulling out a book off of the shelf and looking up numbers there but it took a big lot of time for me to accept that I'm still teaching this just in a slightly different way. And having that the assessment where they do get some of the feedback of, okay you need to do this, you can at least see if the computer is doing what you think it is before you send it off to me to get some feedback from me is so much nicer than, okay, you wrote it on a piece of paper you made a silly mistake but you couldn't possibly have known it until I've graded it and told you this was wrong. He gives them the chance to fix their mistake because that's how you do math, right? The fixing your mistake, the understanding something went wrong and let me figure it out rather than, oh, I made a mistake, I lost three marks. So that last bit is really interesting I wanna follow up with you immediately. So dialogue, the peer review process for the researcher is analogous in some sense to providing constructive feedback to students so that they can process what they did wrong and then improve. Did you engage in more or less dialogue Janet during the pandemic? I tried to do more there is always the time constraints that we're all working with. I taught a proof class in the past term and I was lucky we had access to hypothesis which allows for social annotation. So that was really nice we were doing some synchronous classes but it meant that there was a process to check in with the students who were asking questions about things in the textbook. And so I found definitely with online teaching one thing I need to work into was, okay I need to be checking in with them every day in some way so that there's some kind of there is more interaction, it's no longer, okay I teach 50 minutes three times a week and then answer some emails I need to have more of that feedback. And I just sent my current students they handed in their worksheet, their first playing around with an R notebook and sent it back to them this morning with the comments of some of them were just, yeah that looks good and some comments about the data set we were working with and others that though here's what you needed to type in to make it work. That's a shame, so you're using hypothesis to provide feedback back around our notebooks for example. I was using that for the textbook and I'm using crowd mark for the R notebooks. I get them to submit it and play around with it. Very cool. What about you, Sean? I think in the past I had the sense that you have some strong opinions about grades versus mastery so I'd like you to throw some spice into this conversation. Yeah, I could keep going on this until our time is up, I think. I'm moving in the direction now where I'm no longer convinced that the timed proctored close book test is this sort of gold standard for assessment. I'm moving away from it. I think this is one place where Jana and I argue a little bit, I think she's less ready than I am to kind of ditch it. So yeah, I mean partly through following people on Twitter and other places where I've been to workshops, met people through working on pretext and things like that who are kind of much further down that road than I am and I tried this summer to go to a conference, a mastery grading conference although they've changed their name. They don't like the, there are some I guess connotations around mastery that they're getting away from but so they're now grading for growth is what they're calling themselves but that conference didn't work out because that conference was at the same time as the CMS summer meeting and I was also teaching that week and we had also booked some camping and I couldn't do like teaching and two conferences and camping at the same time as it turns out so I didn't really do their workshop. I kind of wish I had but yeah, I, you know in the spring semester I did, you know so I was teaching what for me are small classes of about 50 students and I think, I mean there's this perennial issue which has gotten worse over the, you know in the last couple of years in Alberta with the current government that we've had some pretty severe funding cuts and they've taken, you know so we have no money left to hire student markers for example, which really limits what you can do if you're teaching a class of 200 or 300 students but with 50 I said, hey, these classes are small enough I can maybe I'm not gonna go full on like mastery grading but I can at least let them do revisions on like any piece of work they submit they can have at least one chance for revisions and the revisions are nice because, you know now some of them will just email them to you but others will book office hours and they'll come in and then you get to have a conversation about what they know and what they don't know and they can kind of work out and get to the point where they're like, okay now I understand what I need to do on this problem I can go away and I can fix it and I'm gonna send you the improved version later in the week and then you know when you start getting the feedback from students you know, I got a lot of comments from students saying like, oh wow, like, you know I appreciate this because now I can look at tests as actually an opportunity to learn something and not just as a measurement tool, right and that really changes the conversation it really changes the perspective on what they're doing, you know in that test and you've removed some of the pressure because they know they'll have another crack at it if they get it wrong you've removed some of the incentive to cheat I mean, and there will be, there will still be some and you know, like I, one of the guys who he's running next week I'm doing a week long training thing on team based inquiry learning which is this NSF project out of University of Southern Alabama and they recruited me to take part and like the guy that's running that, Drew Lewis he's done this stuff for like he's a certified instructor in mastery grading and still he saw his students, you know posting work on to check sometimes and it's even though they have like unlimited attempts to get it right and it's just that sometimes the students get to the point where like, I need to have this done and I don't have time to do it because I have a thousand other pressures in my life right now so I'm gonna let somebody else do the work for me, you know and that doesn't make it right but at least, you know those of us who are seeing, you know the cheating going on in our bigger classes I felt a little bit, you know I don't know, vindicated that, you know even like these expert teachers teaching 20 students at a small liberal arts college are seeing the same problems that I'm seeing so I know it's not entirely my fault but yeah you mentioned some constraints that you're running into like, you know, provincial support leading to budget cuts and so forth so I wanna take a different tack on that and maybe I'll pose this question to Fiona and we can come back around so let's suppose that resources were not a constraint imagine you had, I don't know I'll just make up a number like $100,000 budget to equip yourself with new resources, tools, teammates to teach in a new way, what would you do?  What would you do differently? Wow, I've never thought about that nobody's ever put that to me before I mean, obviously, you know support in terms of grading help is a big deal and I am quite fortunate at Waterloo that we do get quite a bit of graduate TA support for our courses and something that I was really pushing when I was teaching online was that the TAs needed to provide really detailed feedback because sometimes even when using Crowdmark because it can be so fast it's just score, enter, score, enter, score, enter and I discussed with my TAs up front like I don't want you doing that because these students are not getting any face to face I feel like this is the place where we really need to give them feedback so I would set up the comment library in advance just to try to have it there in their face so they could see all you gotta do is just drag these in and then create some new ones as it comes up so but yeah, some support for doing things like that obviously more graders make it go faster and allow you to provide even more detailed feedback there's, I don't even know what would I do get some really fancy feedback? So the big thing that you would do differently is to try to give more feedback on students work to students.  I don't know. That's a question I'm gonna think about more. I don't know. That's very interesting. There's a famous study in education that's been reproduced many times that was I think first done by Black and William which was called Inside the Black Box. And so the study goes like this you've got say three random groups of students Jana will probably correct me on my stats here but I'll say something anyway. So three random groups of students one gets scores alone one gets feedback comments alone and the third group gets both. And then you track these students over a series of assessments and then at the end you give them a summative assessment to see which cohort learned the most. And it is uniformly found that the students that receive feedback alone and no scores outperform the other two cohorts. Cause you can imagine what the student does. 79 UGG or 79 and all this red ink I'm not gonna read that. Or wait, what's my score? I'm gonna read through everything. And so the student engages differently. So I quite liked the way that you revolved around feedback. So what about you, Jana? How would you spend resources to improve your instruction and what do you think you would achieve with those resources that you can't achieve now because of the constraints? Well, I think side the advantage of thinking while you were answering Fiona I was thinking of if I had kind of the army of graduate students available to me what I would love is if you're often trying to get students to work together and talk together if they had a graduate student mentor that maybe the group was meeting together and talking with. So I've often thought kind of oral exams would be really useful and nice in teaching a math class because then you could give a hint for the person who gets stuck on a little bit here and really see where are they? What's the problem? What do they know? What do they need a little help with? Downside of that is, well, it takes too long if you have 150 students. And it's incredibly intimidating if nobody's ever done an oral exam before math is not the place that you wanna be the first one saying, oh yeah, we're doing oral exams in your calculus class. But if you could have that kind of check in where they're having that conversation and have somebody that they know then that would be really nice. It would be great to teach a class where you get it a little bit when you happen to have a nice smaller class where you're actually getting more of that immediate feedback but in your big classes if we could break it down and have the, okay, you have your group work you need to get this much done and then meet with your mentor on Tuesday that would be wonderful. So you'd like an army of assistant coaches so to speak that are helping out. That's really cool. What about you, Sean? How would you change things if you weren't so constrained by resources? Well, I mean, I like Jan's idea of having the grad student mentors working alongside us. I'm not sure that $100,000 would solve that problem at Lethbridge because our pool of grad students is very small. But I mean, one of the things I was trying to get done this summer, I hope we get it done this summer is trying to recruit a pool of undergrad students who could serve in a similar role and do it as part of like an applied study project so they get like experiential learning credit and I get help in the classroom. And so it's like a win for everybody. But the other thing I do, so if I had just $100,000 I might use it to buy off one of the crowd mark devs and I'll explain why. You've readily pointed out that there's this study and it's been run a number of times where students kind of, learning outcomes are better when the students get feedback rather than a score, right? And then like the numerical, like the percentage scores are, I feel like they're unreliable, right? Like I hate at the end of the semester when you've got like, you've got 300 students in your spreadsheet and now you've got to assign letter grades and you're like, oh geez, I got to put the cutoff here between like the 74.8 and the 74.6 students, like, you know, like, because they're just so like compressed together, right? And people like, there's a phrase that's been coined for this, right? This objectivity theater as if you can somehow, if your grading is so objective that you can tell the difference between a 75 student and a 76 student, right? You can't, right? Whatever grading system you're using, it's not so fine-grained that you can tell them apart. And, you know, but if I look at say a system like CrowdMark, I'm kind of forced to give a score, right? I can't really return the work to the students without scoring it. And if I give the students their work back with feedback, even if I figure out a way to say, hey, I'm just going to give you feedback without a score, there's no mechanism for them to resubmit, at least not within the same assessment, right? I could create a new assessment, which is like revisions for the other assessment and have them submit a whole thing all over again. But it would be nice to have them resubmit and have, you know, the different versions in one place, and they can see their progress through that particular competency that they're working on, you know? And so if they need four attempts to get it right, you know, those four attempts are there and they can see their kind of progression in their growth of learning that particular topic, I'd love to be able to do that. I'd love to be able to get to the point where at the end I can sign off and say, yes, okay, you now understand this topic, you're ready to move on to the next thing. And I guess tied into that. And I think on Twitter, I think we were chatting with you and Steven Kwanze, who's developed a tool for kind of generating, problems that are associated with learning outcomes and having the ability to say, okay, like I'm doing, you know, students are gonna submit to CrowdMark but there's like a question bank, you know? So they have to do a question for this particular learning outcome, but each student is getting a different one or maybe you just wanna be able to generate multiple versions of the question because the student got it wrong on the first try and you wanna let them try again, but you wanna give them something different the second go around. You wanna make them start over from the beginning. And like, so I would love to have some sort of all in one place where I can generate my questions, distributed them to the students, get the submissions, give them the feedback, collect the re-submissions and just kind of iterate that process. I was fascinating. Thank you. So our development team is working on some things aligned with what you said, but there were some ideas that you shared there that I think are as new. And so with Casey's help, we will process what you just said and see if we can use it to inform our product roadmap. So thank you very much for sharing that feedback. So we just have a few more minutes left. And so I want you to think about your future self. So the Sean, Jana and Fiona in November of 2021. So between now and then, you're gonna do various things to prepare yourself for your class and the classes that you're teaching in the fall. But when you look back in November, what is it that you hope to have improved in your own instruction between now and then? How will things be better for you as an instructor? What's your goal for improving your pedagogy, your role as an instructor over the coming semester? And I'll start with Jana. I heard question. I think I'm one thing that I am as I'm starting to sometimes think about my September classes, while I'm currently teaching with summer ones, is how to build in that routine interaction with students outside of the classroom. Cause in the classroom, it's easy. We're used to that. We know how to ask questions and get some kind of immediate feedback. But how can I make sure that I'm checking in and responding to them throughout the term? That's interesting. It's another dialogue theme, but it's not necessarily a dialogue on the basis of assessment feedback or in the context of the content delivery through the course. But it said just having a kind of a relationship that goes beyond the course itself so that you know each other's people. Is that a fair summary? Yeah. That's very cool. What about you, Fiona? What does November Fiona look like that's different from today's? So what I'm doing in the fall is I'm actually, I'm coordinating a big calc course. And I'm teaching the online part, but I'm coordinating the on-campus part. So it's actually just this crazy overwhelming task. I'm trying to sort of bring a little bit of both into this world. But I think something that I wanna do for students as well as instructors is to try to create some flexibility. Like I think this is something that universities were missing big time pre-pandemic. If things were so rigid for both students and for faculty. So I mean, I thought about, okay, for the on-campus, the students wanna be there, let's have paper assessments every lecture. And then I started pulling back on that thing. That's putting a lot of pressure on what if someone's kid is sick? What if somebody is sick? All of these issues that we've now come to learn about students and the teachers, there are things going on in their lives. So I'm trying to make sure that I have a remote component. What that looks like right now, I'm not completely sure, but that's really top of mind for me is be flexible and be empathetic to both students and to faculty. That's very interesting. At UBC, there's a little bit of a chatter where the university is very eager to go back to the good old days of 2018. And so they're imposing certain restrictions on the way that courses could be taught. Like it must be fully in-person. And some instructors who have learned along the themes that we've been discussing today, wish to take what they've developed and improved during the pandemic and embed that into the new version of their 2021 or 2022 instruction. And there's this possible collision around academic freedom and academic integrity. And at the core of this, a change in the relationship between students and instructors that I'm concerned about. Because I think thoughtful instructors need to drive the pedagogical changes more than administrators that are trying to do whatever protecting they're trying to do. But maybe I don't go down that path at the moment. What about you, Sean? What does November 2021, Sean, look like that's different from? I, compared to last year, I'm hoping that I do a little bit less of the talking in my classes. It's easy for me, especially with, say, calculus. I can stand with zero preparation. I can stand there and talk about calculus for 75 minutes. And the students will be like, oh, that was great. We learned, I'm like, we didn't really learn. And you listened to me talk. I don't know how much you actually absorb. But I've been trying to slowly shift the focus away from me and onto the students and the work that they're doing. And I always say that I'm gonna do it better next time around. And I always kind of slide back a little bit on it. But yeah, I mean, I'm teaching, Jenna and I, both of us are teaching multi-section classes where one of the sections is online and another is in person. And so we have to think about how do we, how do I design a class that works in both formats? And I'm kind of leaning toward, despite maybe what UBC might say, I like the design should focus on the online first. Design something that works for online. There might be some things that are a little bit different in person. I mean, aside from the meeting space, but make sure that things work for the online students first. And like, even going forward, I mean, I'm in a working group where we've been talking about the fact that like, no, let's not go back to the way things were because we have a lot of evidence that there is for pretty much any scenario, there is some form of blended learning that is way better than just fully in person, like lecture-based, whatever you're doing, like that's not the best way to get the students to learn. And how do we convince our colleagues that there are ways to make this better for everyone? Figuring out what parts of your course work best online, which parts work in person, and how do you design it? So, hey, if you suddenly have to shift online again, you can make that transition and it's not quite so painful. How do you accommodate students like, it used to be like, if a student was sick for a week, we almost didn't even give it down. We were like, oh, too bad, buddy. Hope somebody gives you the notes, right? No, we should maybe be doing something to help those students and accommodate them, right? And yeah, and like these points of like flexibility and empathy are big ones, right?  You know, they're like, and not just they're like learning outcomes, but even just their like sense of well-being was better if they thought that their profs cared about them. Unfortunately, the same study found that like over a four-year degree, it was something like only a quarter of students felt they had one prof that cared about them as a person, which is really sad. Well, as I think about the question, thank you. As I think about the question that I posed, I often recall how enthusiastic I am in August when I'm writing the syllabus and then the incredible energy that I bring for September and the first half of October. And then by November, I'm just like tired Jim and not sure I can make it to December. So because I'm on leave next semester, so I'm not gonna be teaching, but if I weren't, I think one of my goals would be to find a way to maintain the early semester momentum all the way through the end of November. So on behalf of your students, I wish to say thank you because it's very clear that the three of you are very empathetic and you're focused on their learning and their accomplishments. And I think that's a fabulous reason to showcase the three of you. Not all instructors have that same goal, like Sean referenced, hey buddy, sorry, you were sick. But thank you all for contributing to this excellent conversation. And I wish you all great success in your professional development and in the teaching of your students. Any final words from the three of you? Good luck. Let's connect again and see what worked and what didn't. Sounds great. Well, thank you all. I hope you have a nice summer and get a chance to spend some time outside. And I hope you all stay safe from the pandemic and that your family and everybody stays healthy. So thank you. Thanks. Bye.