I've been working at Walmart labs for the past two years and a few months ago I joined the display advertisements team of your marketing team. What we do is we show ads on the internet, hope that people click on them and then when they come to Walmart.com, they end up buying stuff from here. So one of the challenges we're facing in this, one of the problems that we're solving is this, real-time user scoring, forbidding and display retargeting. I know it sounds quite a mouthful, so let's just break this down into small pieces and understand what exactly we are doing. First of all, what are display advertisements? A display advertisement, as is evident, is simply an ad which conveys its message in the form of images. It can be text or anything. You click on any of the items. So take into Walmart.com the item page over there and then you can shop ahead over there. There's a huge amount of inventory of this kind on the internet. And this is the kind of ads we're trying to show to the users. Okay, we mentioned retargeting. So what is retargeting? In retargeting, you're trying to reach out to customers who have already come to your website. This is in contrast to a prospecting campaign where you try to get new users to the website. We're not doing that. The advantage of a retargeting campaign is essentially that you have information about the user. You know what items he's browsed on your website. You know items what he's bought from Walmart.com, right? So this helps us model his interest. These are the things he's interested in. And we can also model whether he's more likely to buy a game from Walmart or not. This helps us do a highly personalized advertisement. We can decide whether we want to show this user an ad or not. Is it worth it for us? And if we do want to show him an ad, what exactly items do we want to show to him? Okay, what makes all of this possible is this technology called real-time bidding. Okay, here's what happens. Earlier, what would have happened is if you wanted to show ads, you would go to an ad network and say, here's some money, get me, it shows so many impressions or so many, get me so many clicks. What happens now is this. Suppose you have a blog post or something and you want to show an ad. You'll allocate a space on it, give it to an ad network. And now every time your page is loaded by anybody across the world, a call is made to the ad network. Hey, somebody's trying to open my page. Now the real-time bidding part kicks in. The ad network recognizes which user is trying to open your blog post. It communicates this information to every single advertiser it has deals with. So every single advertiser gets to know this user is trying to open this page right now. They're allowed to have roughly 10, 20 milliseconds of time to decide how much money they're willing to spend to show that one impression to the user. So instead of buying bulk of impressions, you're now going to be bid for every single impression. All the advertisers submit the bids. Whoever submits the highest bid gets chosen as a winner and he's allowed to show an ad to the user.  There's an auction going on on the internet right now. Okay, so where do we sit? We as a user is browsing on walmart.com. We continuously keep scoring the user. We compute the bid values, how much we're ready to bid. But we do not exactly sit on the bidding on the auction house ourselves. We apply this to a partner and the partner act does the bidding for us. The reason being that it requires a huge amount of engineering effort to actually set up these, you know, you have this quick time response constraints that you have to reply within 10 milliseconds and you have billions of auctions going on, right? So we don't do that yet. We just score the users. We compute the bid values and give this to a partner. Of course, if we do win an ad, then it is up to us to serve, to show the items in the ad. Okay, so it all comes down to this. Computing for every user in real time, a specific dollar value. We do this for every single user who is visiting walmart.com. And real time is important because what we found out in the market was that you have to target the users within minutes. You're gonna do this in one day process or two day process that you submit the bunch of users to the partner every day or something. This has to be in real time. Okay, what's the data that's available for us to do our modeling? We all have the, we have the first party walmart data, which is what is all the site activity that is going on and some form of metadata. Let's look through. Site activity would basically, he's viewed this item. He's added to cart this thing. He's ordered this, all that kind of stuff. We keep track of everything a user does for 30 days. All of this is marked with a timestamp session ID. All of this is stored with us. We also have some form of metadata. For example, we keep track of the user ever logged into walmart.com or not. How frequently he comes to, how frequently does he visit walmart.com? That helps us mark a user's active or passive. The walmart and variety of marketing channels has a user ever click on any of the ads anyway, so that we can mark him as a clicker. And then he tends to click on ads. Yes, okay. One thing we would like to point out is we do not use any personally identifiable information. What this means is that we have a user ID say XYZ. We know what all this person did. But we cannot mark this person as who he is in real life. We're not doing that. We just use a user ID and for user ID, we build our models. Okay. So how do we go ahead to build our user bits? This is done in four stages. Let's go through one by one. First, we compute engagement score, which is essentially a measure of how engaged he is on the website, how much he uses walmart.com, and which converts into how likely he is to convert for walmart.com. Okay. So if you have various types of side activity, like I just said, item pages, add to cards and all that stuff. So for each type of side activity, we convert this. We take the historical data and we try to measure. Hey, if a person does so much of item pages, he's viewed, how likely he is to convert later on. So based on that, we compute a weight for every type of side activity. Then we just multiply this with the number of that type of side activity is done. Weight of item page into the count of item pages of user has viewed. And just sum it over all types of side activity. This gives us a single value for every user. It's essentially how we rank our users. We use this ranking to compute the conversion probability. Okay. Here's how it goes again. A user, you have the user, you have his side activity, and then you have computed the weights. You can use both of them to compute the user to engagement score mapping. Now what we do is we group users for very similar engagement scores into buckets. And then you have this for each group of engagements. I mean for each engagement score group, here are the users in the bucket. What we also take is, okay, now out of these users, how many of them actually ended up converting? Okay. Which gives us basically for each engagement score group, you have a conversion probability. So we can say that if a user, once you compute his engagement score, he falls into this bucket. The probability that he will convert for us within the next few days is so much. This is a graph between the conversion probability and the engagement scores. So essentially, as you can see that as the user browsers more and more on walmart.com, he becomes more and more likely to convert for us. Okay, now up to the next part, which is recommendations. Now, you're not only just bidding for the user, you also have to show him some ad, right? So we need a recommendations engine. We have two models, which are essentially Markovian models, which say if a user has browsed this item, he is most likely to end up buying one of these set of items. Okay. There are two models that we use. One is browse recommendations, which is to say, okay, if he's viewed these items, he ends up buying these. For example, if you're viewing some tablets, then we might end up showing you the most top selling tablet, which is similar to what you were viewing. But also we have another model, which is the next best product. Now, suppose you have bought the tablet, right? Next, you're not going to buy any other similar tablet. You're going to buy some accessories, maybe an external charger or a case or cover for it. So that is the second model, which we use, which is the next best recommendations. Okay. Now we combine both of these and in real time, as a user is browsing on walmart.com, we keep updating his recommendations, using both the models that we just discussed. We also group these by categories. What we do this is because you don't want to show an ad, which would have a pair of jeans, a TV, and a camping tent or something in it, right? You'd want to group them into similar categories so that the ad looks much more coherent. And then we use, and then, so again, these recommendations, which is computed in real time, is used by the ad server to show the ads. Now, the final part of it is the expected revenue. Here, we assume that the items that we've shown in an ad, he's most likely going to buy one of these items. Then we take the average revenue of these items as the expected revenue. So if he ends up buying, this is approximately the amount of dollars walmart.com is going to make. You might first raise eyebrow on this assumption that it seems too aggressive, it seems too confident that he'll buy one of the same items. But our logic here is that if you're showing him ads of TVs, he might not buy one of those three TVs that we just showed him in the ad. He might buy something very similar, in the same cost range. So it gives us a ballpark figure of what we expect, of how many dollars we expect to make out of him. Another thing we noticed, we used is, the metadata we had just mentioned, like whether he's a logged in user, an active user, or whether he's a clicker. We could not use them in combination with our side activity because those were linear counts, like he's in 10 item views, or 20 item views, or 30 item views.  Has he logged in in walmart or has he not logged in in walmart? So what we did was, we took the entire space of walmart.com, all the users, and then divided them into eight groups. One group would basically be all the users who log in, who are active, and who click on ads.  And similarly for all the eight groups, it is all done mutually exclusively. Now that we have all of this data, computing the bids is simple. Okay, you have a conversion probability, which is, if he clicks, this is a probability that he will convert. You multiply that with the revenue you expect him to generate, and then you simply get the expected revenue per click. Okay, this is, if the user clicks, this is the amount of dollars walmart.com expects to make. ROAS goal. So essentially, if you're a marketing channel, mostly your business goal will be something like, achieve me, get me an ROAS of 10. ROAS stands for revenue over ad spend, which means if I give you $1, you should be able to generate $10 of revenue. So then you can clearly see that ad spend per click, how much money you have to spend per click, is the revenue you're going to generate, divided by the ROAS goal. And now that you have this, you can simply compute the ad spend per impression, just by multiplying by CTR. CTR is click through rate. Essentially, if it takes, if CTR is a 0.1% which means it takes you thousand impressions to get one click, you multiply that with ad spend per click, and you'll be able to get an ad spend per impression. In this whole process, we're able, for every user, we're able to mark that this is the amount of dollars, Walmart.com is ready to spend in a bid. Right? Some of the next features, next steps that we have in plan is essentially to identify more features which would help in identifying, hey, if a user is in this bucket, he tends to click more, or if a user is in this bucket, he tends to convert much more. Right now, we only have a basic statistical model, but we're also looking forward to trying out many machine learning algorithms which might give us, help us improve our performance. Here's just an overview of the systems that we use. So the bottom left you have is the offline weights and conversion probability computations. These are all basically hive jobs which run daily to keep updating the parameters of the model. On the bottom right, we have the recommendations engine, wherein we generate the recommendations and keep them in a MongoDB. At the top left, you have the event stream server which pumps in every event which is happening on Walmart.com. So as events happen on Walmart.com, our real-time system receives them which is the MAPDA at real-time system. Okay, we receive the site activity. We read all of these data or the model parameters and the recommendations for the user, the information about the items. We all read through this into a real-time system. It computes the user bid it computes the recommendations for the user and then puts them into a Cassandra's tool. Since, of course, on the scale at which we are working, we cannot obviously have one machine, so this has to be a distributed machine. We use Flume to keep logging all the bid values into HDFS and then of every few minutes an uploader kicks in, wakes up. It reads from HDFS the bid values that has just been put into HDFS and uploads it to our partner. And then the partner takes care of bidding for us. This is, I guess, the current performance we have. We upload nearly 180k users. This is only a fraction of the total Walmart volume. Since we're in a learning stage right now, we have capped ourselves to spending $500 per day. But even with this, you're able to win up to 250 to 300k impressions every day, generate of almost 270 to 300 clicks at a CTR of 0.107. So yeah, that's our current performance. And yeah, that's about it. Yeah, hi, it's Shakti here. So the basis for all your processing is, you're doing a reactive kind of approach. It's the approach is reactive, right? You're based on the historical data and you're going to give some recommendations. Do you try something that's more proactive? For example, try to incentivize the person who's seeing the ad to click on it. I mean, make it more probable for that person to click through and go. For example, you know this guy cannot say no to a bargain. So there's a one-time code that comes up and you say, okay fine, you use this code and you have $10 off. No, I get your point. As of now, we cannot on the fly create such promotional offers for people. That's something you'll have to work with the business with, right? Because we don't have access to that. So when it comes to clicking, getting the user to click, the best you can do is you... So okay, there are a couple of things. What from our side what we can do is show the best ads to the person. That's about it, what you can do from our side. Yes, if possible, you could add some promotional features to it. What happens also is that our partners to whom we upload these bid values, they take care of on which venues the users most likely to click. Okay, whether on some of the sites that users do not at all click on Walmart ads. Whereas on some of the sites, users do tend to click on the ads. So CTR optimization is something which the partner mostly takes care of for us on that side. As in they optimize on, these are the sites where the ads are doing well. So they'll show more ads over there. But proactively, as in, this is the best, the best you can do is show better and better ads to him. That answers any more questions? Yeah, are you using... Oh yeah, sorry. Are you doing user similarity in your algorithms in any way? If so, how? The only similar, no, we're not doing any similarity. As I said, it's a very simple statistical model. The only similarity we're doing is once you've computed the rank which is basically the engagement score, we just club users together based on the ranking. There's no clustering as of yet we're doing. So as I said, we have to try out many of these machine learning or data mining approaches. It's a very simple statistical model we have as of now. It's only been a few months we started this project. We've been live only for a month or two till now. So that metadata about the user, what was that? Metadata is basically... What we've noticed is these features... You might do a lot of side activity on Walmart.com, but you're not at all a clicker. For example, I'm not a clicker. I do, I buy stuff online. I don't click on ads ever. Clicker is a metadata which you can use to identify such people. You show lots of ads to people and then you see whether that person whether he's clicking or not. If he has shown a tendency to click in the past, he's more likely to click for you again. If he doesn't show any tendency to click for you, it's very less likely he'll show you, he'll click for you again, right? So that is a metadata like clicker we use. Then we've seen that logged in users tend to convert more. If you go to his website and have done the effort of logging in with the email ID, then you tend to convert much more. I don't have the data to show, but yes, I think the clicker was almost 2x better performance. You'd click two times better, two times more frequently. Thank you. How long will you keep me... Keep targeting me with television ads. So how do you know when to stop? Sorry? Oh, okay. So his question was basically if he visits volma.com and he's browsing through televisions, how long will we be showing him ads with regards to televisions, right? If I don't browse anything else in the future. Yes. So what we've noticed is this, I mean, these are real-time systems. You try to target the user as soon as possible. Within the... I think we try for two, three days as of today. And if he isn't... If you do not do anything else on volma.com, then we'll stop showing you ads. You can try and make it a little more, six, seven days, but I don't think you'd go much beyond that. At least we don't as of now, yeah. Hi, this is Vikram here. So how do you make your ads context-sensitive to the publisher's content? For example, the publisher is a real estate site and your ad is something else that you're selling. No. So as I said, we do not do that as of now. You're basically... I think since volma.com is a very generic... It's not like we are into sports business that we'll show only on sports websites or sports... I mean, something like that, right? We're very generic. We show all sorts of... We serve all sorts of items. So it's essentially based on whatever you just viewed. That is a bigger context for us. Whether you're on any site, it's not as important as that you actually added this item to your cart a few hours ago. That is a much stronger signal for us. And wherever you are, we'd go and show you the ad about that. Just the question I had was, when you compared all your click-through rates and the ad penetration, clearly there'll be a lag vis-a-vis, let's say, target and even ad words and people who are, let's say, earlier in the game of the online channel. So when you're looking at your technology, architecture and scalability, what has been your experience in terms of the various components you had that architecture chart in terms of what you have done right and where you probably need to change the system? Right. So we use a MAPDH system. Getting the system, as you said, okay, we target only very few users. When I said the number of users 180k, it's only a fraction of the users on walmart.com. But our systems, even as of today, run for the entire walmart.com traffic. We receive every single site activity that is happening. We use what's a MAPDH system, a real-time processing system. It was built by people who, by a startup which got acquired as walmart labs earlier. So we had tech support for that. Some of the challenges were essentially making sure the system is up all the time. We had to pick which databases to use as in our computation, half of it is offline, which is computing engagement score group. This is a conversion probability that's based on historical data. So all of that is computed offline. So we had a high, we, all of them are high queries. We use HMA to schedule our jobs. It keeps running them. We keep dumping all of this into MySQL. And then again, for recommendations engine, for example, since again, this runs on the scale of, you know, the huge catalog which walmart.com serves. We are initially using HBS. It was being stored in HBS. Then we found that, that we couldn't query it in real time. So we ended up taking a dump of daily dump from HBS into MongoDB, just to be able to serve with, whether it's fast enough for us. About the other parts, I guess, the Cassandra store. So we didn't have much of a choice when it came to where we store all of the mappings. It is the user to site activity, because MAPD8, as of now, is tightly coupled with Cassandra. So we do not have much choice, but then we run into a few problems and then we had to talk to the people maintaining MAPD8 to, I mean, fix those problems for us. I guess, time's up. Thank you very much.


So I wanted to give just a little background before we get into the discussion with the instructors about how this all came about So it's based our discussion here is based on as Mike mentioned participation in a TLF project That RTSIT helped to help to lead and it grew out of a needs assessment in the Faculty of Arts that was done I think around 2018 where RTSIT worked with a group of instructors to To collect needs from instructors across the faculty about which learning technologies were most important to them To enhance the support of those and one of the themes that emerged was the need for collaborative writing tools and support for collaborative pedagogies And so RTSIT along with a group of co-applicants instructors submitted a TLF project that was funded And then between January 2021 and January 2023 Worked with nine different instructors and even more courses across the two years to pilot And evaluate different types of collaborative learning activities different types of technologies across a range of different disciplines In year one of the project all of the courses utilized Microsoft teams office 365 is the primary tool for collaboration And then in year two all of the courses gave students choice about which tools they would use And then for the evaluation student feedback was collected through surveys. So each of the classes there was a survey to collect student experience in year one There was also instructor interviews done to understand What the instructor's motivations were for adopting collaborative pedagogies along with some of their experience with the pilots In the course of some of the challenges some of the learning benefits they saw In addition in year two We did focus groups with students to dig deeper into some of the issues and questions that emerged in the surveys in year one And then we collected all of this and do a final report which we will share At the end of this we didn't want to spend too much time going over the details of the evaluation results I think the discussions with the instructors will be more interesting But we will share all that information at the end and share a link to our report So To get started I will pass this over where we hear from the instructors now Who will introduce their courses in a little bit of context before we get into the discussion So i'll pass it over to katharine who can talk about what she did in sociology Great, thanks jason So global pandemics is a sociology course that explores the subfield of disaster studies And it's a medium-sized course with no prerequisite So it draws students from across campus and it's also approved as part of ubc's health and society minor So there's a whole variety of students in the class In january 2023 when I taught it we focused on As well as some historic pandemics and other types of crises that intersect with pandemics like the climate crisis like The opioid epidemic and gender-based violence And throughout the term the goal is for students to explore this big idea of social vulnerability Asking how does inequality shape The capacity that groups have to anticipate and prepare for Pandemics and other disasters to cope with them as they're happening and then to recover from From the pandemic or disaster both in the short term and the long term And the course weaves collaboration into everyday classes through group case studies every week And as well as a major assessment called a community partner interview synthesis that i'm going to focus on today For this synthesis student teams interviewed a nonprofit organization that supported vulnerable communities during COVID-19 And each team at the end of the term produces a public-facing report or media artifact Explaining the societal challenges presented by the pandemic and ways that their specific organization work to address these challenges And final assignments were delivered to the community organizations Some of whom drew upon The student work for their media promotion like on instagram or for grant writing And this assignment was designed and facilitated with tamra baldman and galeen davies from ubc's office of regional An international community engagement. I couldn't have done any of it without them in terms of technology I gave a presentation to the students about microsoft teams and we also had a teams workshop facilitated By artsisit but students were actually allowed to decide on their own what technological platforms They ended up using and I can talk more about that process in in the q&a and The most important thing that I learned from this I think is Time so I I would encourage everyone to budget Extra like double the amount of prep time that you anticipate and then also the time you think It'll take to sort of administer and facilitate the assignment throughout the term and also not just for Instructors time, but also budgeting time for students to complete it So not adding it as an additional course component, but but building it into class time whenever possible to give them time to work on it Okay, that's all for me. I will pass it on. Thanks Great. Yeah, let's head over to uh, uh, dr. Uh, shavon wood technique for you with geography 121 Good morning, everyone Sounds like kathryn and I may have the same cold Stoned around So, yeah, so 121 I've actually been teaching this course for well Almost all of the 10 years that I've been at uBC and it's had various recreations over that time Um, but this particular well always though I have a focus on case studies. So, um Just to give a context as well. It's a first year course. It's a requirement for our, um environment and sustainability minor Uh, and it's a major and minor and it's also Um a requirement for our urban major and minor and it also attracts. It's an art selective Um, it'll be part of the new place and power Um, and we get a lot of science students actually who take it as their art selective So we get students from all over campus um So the course as I said, I always have a um a focus on case studies to really Try and ground The abstract geographical concepts that we're looking at and which are fundamental for students going forward So things like place space time space compression political ecology Settler colonialism these are all terms. They may have heard they may be aware of but really grounding them in uh in geography And in in the place that that uh, we're in as well So the case studies really help with that. Um, and so you I try and make them as relevant to students like as possible So during these were always done. Um in person even when the course switched to blended Um, sorry to to online during the pandemic Uh, there was most the the case studies had always been done in person Obviously with pandemic this changed and so I was exploring ways that this could um continue to be done The collaboration could continue Hence my involvement with jason and the the large tlep So, um, I was on leave for the second year of the tlep. So, um I obviously didn't do the second part. But for the first part, uh, the students used the microsoft teams In order to do their group work around the case studies. So they were allocated An hour of time a week And it was in their schedule, but they could meet at any time But suited them because it was it was online. It was not in person Um, this was when we were back in person. So this was in 2021 um The autumn term, but this course was a blended course. So part of it was um Insynchronous and part of it was asynchronous In terms of how uh, we facilitated students using the technology um, we I did give an overview in the uh zoom lecture And I actually had mike as my one of my ta's for the course and he was specifically focusing on the use of microsoft teams And the collaborative because there were 360 students in this class. So it was a larger class Um, so mike created a short video to show them how to use it And then in terms of how we assessed it, uh, what we asked students to do is to use the file option in microsoft teams Which enables you to have multiple uh draft drafts of the document So we could go in and see the collaboration and obviously then there's the chat and they could use the video option if they Didn't weren't able to meet in person um So, uh, yeah, as I said specifically It allowed for student case studies to happen anywhere anytime Um, which really helped with the notion of a blended course Uh that there were moments in the course where the students all had to be together with me The instructor uh are in their discussion sections and then there were other moments where it was really up to them to um And that for me is part of the collaboration. It's the tool, but also it's the negotiation of when they meet In what like whether they meet in person Because the you know with using these tools it isn't only to facilitate them all being Uh flung across different parts of Vancouver the lower mainland And though some of them did actually meet on campus and then just use the teams For uh drafting the very the different iterations of their assignment um So yeah about being an active learner and collaborating and taking responsibility for their own learning So I'll stop there. Uh, yeah. Uh, thank you. Siobhan. Uh, could we move on to uh, Dr. Uh, Brienne, uh, or Elvarez? I was Spanish 280 revolution Thank you so much. Um, so my the course that I did this project with was Spanish 280 and I did it in two different contexts one was completely online and one was um in person with sort of a multi-access group activity on fridays So Spanish 280 is actually a course taught in english. Don't be fooled by the course code About revolution and the course was created for Students that are interested in satisfying the arts literature requirement. So it is for non majors and minors And also it's the idea of the courses to build interest in content related to hispanic studies in english So that perhaps students would be enticed to take a course in Spanish in the Spanish language I'm going to explore some of the content that we have in our courses there so the course um Was large for me and our courses are typically 40 student enrolled and this one has about 80 typically With ta support. So I was benefiting from a large class and then also ta support and um in my experience the In the in-person version the classroom space is really important and I'll I'll explain why in a second but For the online, of course breakout groups in zoom was was quite helpful So this course similar to the The courses explained already Does revolve around a big question or a core question and it was how does revolution evolve From throughout the 20th century and into the 21st centuries in latin america. And so we explored texts typically testimonial nature first-person narratives of participants in different revolutionary movements From mexico, cuba, nicaragua and venezuela as a sort of Emblance of a humanitarian crisis that a lot of our students were actually interested in and wanted to learn more about so the The core question focus made it very easy to involve a new technology and also to involve students as much as in the learning process In terms of collaborative learning this course has a really interesting Set up in that I was typically doing a lecture On mondays and then wednesdays was a typical seminar where we would do some engagement in small groups And then fridays were almost completely student facilitated. So I would have Anywhere from eight to ten student facilitators And they were in charge of a 30 minute session with groups of eight to ten students each So The process of collaborating was a multi step one They would need to turn in a facilitation guide to me by wednesday And then I would give immediate feedback to all of the presenters for or the facilitators for the friday session And the idea was to give them a sort of inquiry based Feedback on their guide from anywhere from questions about, you know, what how do you think your group members would respond to this type of question to You know, you've brought in an outside source Maybe we could contextualize it a little bit further so that it jives better with what we're talking about in the in the texts of the week Things like that. So just giving them feedback That I might expect if I were giving a lesson on a topic to a group of peers and then The students facilitated this session and they used one drive as a collaborative tool So similar to google drive, there was a doc that that the facilitators were taking notes on both in the online version and the in person one Of this course and the idea was that the facilitator would take notes on some of the key takeaways of the session So that the other students that were in different groups with different facilitators would have Sort of a bird's-eye view of what had happened in each of the discussions And this was particularly important because another collaborative tool that I used where it was canvas discussions It was a large course. It was a writing centered course. And so I wanted to make sure that students were able to articulate Some of the key skills that we look for in a writing course summary framing of opinions Evidence-based research because the final project was a research-based project So they were gaining all of these skills through the canvas discussions and the the collaboration through one drive so It was a complex process, but The student facilitated nature made it very easy to do multiple things at once with the same core topic each week So the The activity that students were that I was mentioning that students were facilitating each friday was called the assembly and the assembly was Essentially a student-led tutorial that happened within the same classroom space digital or physical And so I found that for the online base course the zoom rooms We all know what a breakout room experience is like. It can be intimidating. It can be um having Just a couple of people dominating the discussion So I found that the use of one drive was really helpful for for getting people who didn't want to be on screen able to participate in the discussion and add their feedback and comments in ways that were inclusive and Inviting and then we would see their voice. It was also helpful for me because I was one instructor and there was one ta and we had anywhere from eight to ten breakout groups in the online section um Each friday So it was impossible for me to jump from group to group and spend meaningful time there And so the ta and I would sort of target specific groups Where we would spend time each friday so that each group was getting some attention But from the back end we could see exactly what was happening through one drive and each and every of the group So if we saw a discussion happening that we were really enticed by we would jump in and and see what had happened there So it was it was sort of helping me Explore the group dynamics without having to actually physically be there and then the same for the classroom experience. It was Very easy for me to follow up with different groups on clarity I needed or questions that I had related to their discussion because of the one drive I I can answer more questions. I know it's a complex activity the assembly But like Catherine One thing that I learned is yes time management. I couldn't believe how long it took Me too. I I felt like I would understand what I was using as a tool Took me a long time to realize what one drive was and the students the same and I think a big part of it was that the The terminology that they used to describe the technologies that they use like google drive instagram tick tock This cord like they just couldn't wrap their minds around the teams based terminology On the one hand and then there was a lot of lag in the setup time because at the time When we were doing these projects not all ubc students were coming in and getting a student dot ubc dot ca email address Which would allow them to use pipa compliant tools So it also took a while for them to understand why we were using Tools that were privacy protected at ubc and getting the buy-in When they're just so used to using google and other things that are at their fingertips. So those were a couple of things that I used and i'm happy to Reveal a few more in the question answer session. Thank you Yeah, thank you brand and um just a reminder if anybody does have any questions We have fixed the chat so you can just put your questions right in there To start off though Maybe one that we've kind of crafted and that is what is the role of collaboration in your course designs and What do you see as the value of collaborative learning in your specific disciplines? So geography sociology You know spanish and italian studies that sort of thing Would any of the panelists like to kind of start off answering this question? So, uh, I think that the the field of disaster studies that my course global pandemics focused on Is uh itself a really evolving field where leaders in the fields and both theoretical and um applied senses are Demonstrating collaboration out of necessity for solving some of the really complex challenges that can't be addressed just for one disciplinary lens and so I wanted to Guide students and also adopting that practice of you know thinking beyond the terminology and jargon of your own of your own journals and your own training And really trying to understand where other people are coming from and where you have common ground where you can work together and also, um, I think my hope is that art students develop a set of transferable skills that are cultivated through collaboration Empathizing with others perspectives and life experience and also the capacity to Sort of discuss ideas and weigh The pros and cons to come out with balanced arguments though that those are things that can't be done You know as an individual person um, and so because of because of that I built collaboration into I have a learning outcome specifically tied to collaboration in my core syllabus and then And then the case study assessments and the the interview synthesis assessments also have Learning outcomes tied to collaboration So I tried to make this transparent for students and to justify the significance that I thought I think this holds in the learning process and then I think more generally for myself Um scholarship and teaching has never been an individual process I'm always consulting with people and drawing on other people's work and learning from mentors So I don't really see any of the work that we do as um, you know, fun and isolation Yeah, and you know, you you mentioned that they're working with community groups, right? So it's not just students collaborating with one another but collaborating with, you know Non-profits and NGOs and such. So how does that fit into sociology? Like, oh, yeah, thank you So I think in the context of the course and in sociology more broadly I tried to set it up so that the Community members were the experts on their lived experience so that the students were learning from The community members and so a part of that was for student teams to have time in class to research their organization to brainstorm The interview questions they wanted to ask them and then they had to agree as a group on the interview questions and then they had to Nominate a communications coordinator for their group who would actually reach out to the community partner And that was really intimidating for some students. Um, just to write the email and try to set the meeting time So that was a learning process as well And then all of the team members were were on the interview And then they had to synthesize what the community partner said And bring in broader research and present it back to the community partner. So it was um, Yeah, all of it was a pretty collaborative process and we were really grateful for The community partners taking the time to to speak with the students and to share their expertise Fantastic. I'm shivan brian any comments for this one? um I mean collaborative learning You know geographers have been described as good systems thinkers I think we're good system thinkers because of the nature of the discipline which is Constantly in a dilemma of identity um, and why I feel it's so appealing so you know Having students work together Is the very nature of Well, social science if i'm being honest to kathryn's point as well, but certainly within how How geography is done? And you know, it's it's always the dilemma of do we do these kind of collaborative learning pieces in first year courses where you usually are dealing with large Classes you're dealing with students from a whole variety of faculties and departments um, and also, you know, they've many of them have just transitioned from high school and so In collaborative work, you're you're really asking them To be critical and to engage and so and and yet I I think it's so important because uh as you know to to kathryn and brian's courses Which are 200 level i've done more a lot more collaborative In third and fourth here where yes, so sometimes it's working with community partners often its students working on term-long projects and those are really really important um, but for me if it it really needs to be introduced from first year And so these case studies allow me to do it in a in a very kind of gentle way That is part of the assessment of the course that is linked to the objectives And and then it's just navigating how to best organize that and hence why we're here today to think about the tools that we use Because I taught the same course this term and I actually decided to do the complete reverse Which is all the lecture material and all the content result was all asynchronous And the only in-person part was the group work And i'm currently grading the assignments where I ask the students to reflect on that And I will admit for someone who advocates for the use of technology and that and I know I'm hoping we come to this a bit later that The feedback from my first year students this term is is is what they're saying is they loved the group in person Because it allowed them to bring together a lot of the asynchronous content Sitting around a table with three other students Um, can we recreate that in microsoft teams? I mean, I'll continue to try but uh, Yeah, but anyway back to that's why collaborative for me and and also just to echo what Catherine said I think as as faculty members we often Silo ourselves But I have learned the most when I work with others especially outside my discipline because it challenges me Yeah, no and as your ta for that course that was certainly my experience was um trying to kind of I know trying to recreate that that vibe that that collaborative environment that they get in person in a team's environment was really difficult, so yeah Um, Brianna your course on revolution. I mean revolution by its very nature is collaborative amongst the community members Um, how how did it fit into your discipline and what you were trying to teach your your students? Well the I think you know Collaboration was a very key component. It's a key component in my teaching the It's part of my student-centered approach to teaching But the particular exercise or student-led activity that students were working toward each week was called the assembly because in revolution the assembly is the group that Comes together to sort of forge ahead as a society And so it was conceptual and practical in the course context It was intentional to choose the assembly as the student-facilitated activity um And I think you know just to build off of what shaban was saying too I mean, I think as academics we often do work in isolation And then we are not trained to work as a team member in all settings. Um, at least in in literary research It's very solitary, right? We're just alone with our books and then we go to conferences and talk about things, but I wanted students to experience From a very early level What it's like to work with team members from different disciplines with different opinions particularly because when we talk about revolution Anything could come up and I've been surprised to hear some of the things that do come up and how we Work together to maintain a positive classroom environment and so the concept led to collaborative discussions on What a safe and friendly classroom community both in person and online through the discussions and through the one drive activities You know how we work together to to hold each other accountable um, and I think another key component for this particular discipline was just modeling different ways of Creating because I think students often shudder when they think group project or something like that because it is so time consuming It's hard to find Time to work together when they're all taking four or five courses living on or off campus And so I think just giving them a classroom space within which they can consistently come together to discuss the same topic and then seeing Seeing how different groups approached it because the way the assembly worked was I would choose a question And they knew the questions from the beginning in fact when they signed up to facilitate They were signing up for the question and texts of interest for them. So we read Gio Contabelli who is a revolutionary from Nicaragua and she is also a feminist poet And so one of the weeks we read about her the topic was revolution and feminism How does this work? And so the students that signed up were you know, there were some big opinions and things But it was really interesting to see just the diversity Of ways in which the facilitators approached that topic and how they brought their disciplines and personal interests in and even the extra resources and stuff so they all left each class period with a sort of um Take away of eight different ways to talk about one topic. So And then the canvas discussions offered sort of a back channel way of continuing to reflect on the topic So it was it was necessary and it was fun and it was not always easy, but I think it worked Yeah, did you have any like specific approaches you used to? Facilitate kind of an inclusive and respectful atmosphere in your classroom. Did you come up with like a classroom agreement or anything like that? Yeah, so that's a really great question. One of the first Questions that I ask in the course is what is revolution? And so you can imagine the amount of responses that you get to that seemingly simple question And so during that exercise when we're trying to define together what it is How we all interpret the concept that we're going to explore together for 13 or 14 weeks Um, I sort of used that to pivot into Um active listening like we've heard This classmates say this and this classmates say this and sort of modeling facilitation from the get-go and then um You know when we share opinions as a class, you know, what is the expectation and reaction that we expect? So in a very subtle way, we're building sort of a manifesto to classroom space functionality So yeah, it's I don't know if the students realize that that's what's happening But definitely that's what's happening and then the canvas discussion space is more explicit We kind of come up with guidelines together of like, you know, how do we hold each other accountable? If there's something that discomforts me, um, what do I do? And we I have had to do that before unfortunately Um an act on it, but not too often Yeah, I I think that's so important. I know like Siobhan in geography. We've had students whose Parents are in the midst of war and they're over here at university, right? Like These sorts of topics can be really difficult for some students and I think it's so great that you're creating that kind of atmosphere for them to have discussions in brand so Yeah Um, Jason, would you like to take over? Sure. So, um, the next question we wanted to ask was more related to the learning technology Um, so one of the interesting things that came out of the surveys and focus groups when we asked students about the types of technologies They were using is that even when instructors try to encourage the use of ubc tools and microsoft teams um Students often reported the the tools that they use the most for like instagram um discord google docs And they use those things so we're curious about How do you engage students around issues of digital literacy and some of that? And we've heard talks about privacy and some of those questions How do you gauge students around those topics and about the choices they're making with the tools that they're using? Who wants to start I'm happy to start this one off briefly. Um, I I mentioned this as one of my um learning moments was the Just even the way we talk about technology is different and I'll I'll be very brief because I'm sure there's a lot to say about this but um, one of the things that that I noticed was that um When these tools were implemented particularly in 2020 when I did the first round the there was not institutional support So there wasn't like this. It wasn't widespread that we can use teams. We have um emails for all students that aren't the alumni email that is more for their personal Um use and things like that. So I did feel like there was a lot of setup and Work to do to get their buy-in um But if I were to do it again, and I think I did this the second time around just knowing what technologies they use In advance of starting the work with the new technologies. We were going to introduce Help to sort of create parallels, right? So If I was going to use one drive knowing that they were familiar with google drive Was really helpful to explain the functionality of the tool how we might be using it for our um collaborative work in class and then how they might use it to communicate with one another as group members um there was also um A student I remember clearly who Just would never use one drive. There was just a complete rejection of The ubc base tool. So I didn't find a solution to that they preferred discord and then their group needed to use discord to be um in communication with those collaborative ideas in that particular group. So um in a very early phase If there was a rejection to something I didn't really I wasn't able to find a solution to it that was Privacy protected or FIPAA compliant. So um Yeah, I think Building it in having the institutional support having the parallel terminology to be able to work with them and help them understand why we're doing these sorts of things and um And then also how this complements the learning management system canvas and how how these things are all intertwined is really important to get student buy-in Given or Catherine. Did you want to yeah? um I mean, it's a tough question and I think it's a tough question because Has brian just said it's like I think You know the approach of building on what they already know like google drive and um, but also maybe just thinking through um The use of of of tools or technology as you know, something More than just social media um and that Unfortunately, well, I don't know if unfortunately it's the right word, but in my view is Has to be instant faculty institutional wide There's no point in Catherine brian and I harping on day after day as we do with our students encouraging them Um unless you know, and then they go off to another course and they learn a completely different tool So then it becomes more about the tools rather than you know, so it's nice like in a session like this where we're talking about the collaborative aspect So it's less about the tool and more about the collaboration because that's ultimately why we're all using the tool, right? um, so You know, it's tough because you know students will say um, you know, it's great that we're doing this and this and this But it's just overwhelming And and how many new tools and I know that was a huge issue in the pandemic um, so I think we need you know as As a faculty we're the three of us are in the faculty of arts But then of course as as an institution think about um How we Approach the use of these tools and if we're interested in our disciplines in collaboration How we foster that? If we're, you know, thinking about other ways of of approaching asynchronous courses Uh, that isn't just watching hours and hours of videos Then that's another conundrum. So it's like approaching it more as like Big questions that we need to think about so here we're thinking about collaborative learning and and I personally happened to think that microsoft teams is a great tool outside of our classes in fact the arts ISIT if any of you were at the arts ISIT conference welcome back conference in early august um The staff uh the arts it arts ISIT staff Did a presentation on how they use microsoft teams? In a very collaborative way I'm just speaking from what I saw at the presentation. So maybe jason you can add more to that But I thought that was really interesting and I know in other spaces. I've been involved with on campus We've been asked to use it and there's been a lot of resistance from faculty members. So it's um It's you know, and I know even within my own department We've gradually over time being coerced into using the You know the the airspace and that so it's it's more like a sense of like We need to all get on the same page. Doesn't matter how you mean this is not talking about how you do things in your own classroom I know that's our Our private space so to speak But I think it is more about like what are we trying to do here? Like what are our objectives when we use the technology? So that's how I think about digital literacy and I think it's on us and like our students Yes, they may be born with iPhones in their hands, but Um, that doesn't mean they know how to use them uh in the way that we're thinking about critically just because you have you know, uh Well, just because you can afford a fancy car doesn't know, you know, doesn't mean you know how to drive it. So I think that yeah Cars were racing by me as I cycled up to ubc. So that's why it's in my head Yeah, Shivam one thing that I thought was interesting that you brought up in one of our planning meetings too was how students were using The collaborative documents even when you were having them discuss in person I was wondering if you wanted I thought that was really interesting. Yeah, sure I mean, and I think this is more and I put my geography hat on here. I think this is more a A moment of the context of the pandemic and My first year students a lot of room did the last two years of high school fully online and maybe even the first year of university Um, so in as I said this year my students do the the group work the case studies in person And I noticed in the first class or so that many of them are just pulling up a google doc And the three or four of them are sitting around a table looking at each other but working on a google doc So not speaking And I sat I I sat down with them and asked them and and they just said we were all they I could see they were all communicating But they were doing it through text And for them that was the and I didn't push it because honestly, I'm like, well Why should I be telling them? No, you have to speak. They are collaborating. Isn't that the point of the exercise? um So again, I'm reading their reflective papers now and it is interesting how for some students That was great and for others there because I got them to switch groups halfway through the term Just to you know mix it up And some of them said oh in my first group we did a lot of discussion But in my second group everyone just wanted to work on a google doc so again In in both of those situations, it's it's both subjective in like how the student wants to approach things Um, so they have to work that out with their group members, but ultimately you know Those groups who are using google docs and those groups who are having conversations If I were to do it do a statistical analysis just from grading them each week Yes, the groups who did do more conversation Over the course of the term did get better and did score higher But the groups who were using google docs or whatever, you know collab also did well It was just a different form of communication. So back to my point of like, what's the objective? Is it how to use the tool or is it collaboration? Catherine did you have anything to add on this question? Yeah, for sure. So I couldn't agree more with uh, brianne and chevon. I think that um for me until the Until there are standardized technical platforms across uvc that are supported, you know from from first year on I've been focusing in global pandemics in giving on giving students a sense of agency. Um And about what technology they want to use and making sure that their decisions are really intentional and transparent And I think that's where the digital literacy comes in for me in this course Um, so I presented the pros and cons of microsoft teams to students and made sure they knew there was lots of support for it but then I ultimately allocated class time for groups to um discuss the pros and cons of the other platforms that they use as well, um, and then The group actually each group decided for themselves what what technology they wanted to use and also how they wanted to use it Like what would they share where and you know, who was available in the morning who was available in the evening Just getting a sense of their schedules And I think in the end, um about 20 of of teams used microsoft teams in my course. So that's kind of low. Um And then the other teams sort of used a combination of platforms They were already using and that they decided together were accessible to all their members. And you know, I understand like Recently I was on a research team and everyone decided to use slack and I was like Whoa, and I had to learn about it really quickly and that kind of I took a kind of a couple hours to figure that out and I was falling behind on discussions And I was like, well, there wasn't an email about this and everyone's like it wasn't slack. So, um, yeah I I have empathy for them It's interesting that you bring that up because I mean, we were talking about the students using a lot of new technologies Like discord and instagram and everything Have you three learned Much about these new digital tools through your students at all while teaching these courses if they're using all of these new technologies Learning isn't one way but that's simply why i'm asking so yeah I think um The i'll speak specifically to discord because um, I've been I'm an advisor for majors and minors in spanish so I've been meeting with a group of students that are majors and minors And trying to learn more about what what they would like from us as faculty members And the sense that I got was not the sense. It was a real thing. They said they want a discord channel But they don't want faculty there And so I was like, okay great So you want a student channel without faculty involvement and then they want like a faculty student Channel of communication. So it's really interesting They do seem to separate out like the discord and wanting to have that because they know how it works And it seems to work well for student clubs and things here Um, but they recognize that that is their space. And so I thought that was really interesting um, and then we We have an instagram, um, and youtube we have everything in fh is Um, and they were like, we didn't know that where are you guys? And I was like, wow, okay, so you guys are on you're using instagram actively But they don't seem to merge these together with academics So I don't know how to how to bridge that gap if if we have instagram We have youtube we have everything as a department, but then students aren't seeking out academic content In these sites that they like for socials, right? So it was interesting In terms of how I learned a lot Um, I think I mean I I love New ways of approaching the world generally so and Yeah, I So I think I learned more from my students I'm amazed at how they use the tools and also How and and I don't mean this in a derogatory way because I think this is a general problem How little they know about the tools And I mean things like, you know Privacy um And but also the full breadth of how To the tools can be used um, so it's it's um And that back to the point of digital literacy. I think it's it's that, you know, and I think so I'm amazed And it's interesting, you know talking more to students now who are in uh Not so much the first years, but more third and fourth years critiquing instagram for example as a social media platform, which really I know a lot of our students use um And starting to reflect on the fact like I had one student who Was uh saying that they questioned their Parents as to why they Put so much of them on there when they were kid, you know um, and how so I was like great. Yeah that start asking those questions, right? It's not saying don't use it as instagram It's more about thinking about How you're using it so I think I I'm just fascinated with How my students approach Uh using the various tools um, and then again how they use um Collaborative tools and social networking tools in their own life And in their university life too Yeah, as they go on and they start communicating things as experts in academics, right? They're going to be using all these tools and learning to critically evaluate how they're doing that is just so important so Yeah, Catherine. I don't know if you have anything to add to that or if we should move on to the next question, but yeah Um, no, yeah, you can move on. Thanks. Can I say one more thing? Sorry? I just thought of something fun that you might all be interested Yeah, of course This is with my second year's which also has a collaborative piece but not that they've told me very adamantly Now this is a small sample size That online dating is on the way out for them that they're like done with the whole Yeah, so we'll see as I said, this is a small social geography class. So um, I don't know maybe something to come in terms of how These tools are being used Well Yeah, so I guess moving on The the last thing that came up a lot. Um while we're speaking to students during the study was that uh grading fairness was a A major concern for them. Uh, so they were looking at um, you know How can grading fairness and group submissions? How can the grading be equitable and like how can Profs ensure that um the distribution of work Was equitable What strategies do all of you use in your courses to kind of address these concerns because uh Yeah, they came up a lot. So I'll go for it Catherine. Oh, okay. Thanks. Sorry. Um, I allocated Class time early in the term for teams to actually Um, you know read through the assignment instructions and draft a work plan step by step Which I think is a really useful skill for students in second year because they might not even be doing that for themselves On their own assignments yet. Um, so yeah developing a work plan And then identifying who's gonna be responsible for each each step and then later on in the term When they submit their assignment, they include an appendix that includes the work plan And a report of who actually did finish the task. So there's some built-in accountability But it's written by the group themselves. Um, I also encourage students to Bring up if they do feel an issue is developing to to try to address it and think about it immediately So that it doesn't get larger and then by the end of the term it's sort of too late So we try to make a um a formative shift in how the group is evolving So I did have one student who came to me Midway through the term and we were able to chat about strategies and then they they felt that based on Um, based on their sort of new plan, they were able to shift the dynamic and they didn't have to Get a regrade at the end of the term, which was really nice. Um, also I build, um, I think it's just like 1% Into an eye peer reflection on your own work and your peers work Which has a qualitative component and a quantitative component And so the qualitative component is to encourage students to think about their own contributions to the team And what they learn and what they could do differently And then the quantitative component helps me assess whether I need to reach out to the groups to talk about a possible Uh, regrade, which I didn't actually have this term Um, yeah How was how is eye peer like was it easy for the students to use? Did you have any difficulties getting them to use that tool? I haven't had any difficulties getting them to use it. I think it's just a link that I built into canvas. Um, I think it's pretty I think it's pretty simple for them I haven't had anyone complain about it. It took me Maybe like an hour to figure it out and then I have my standard rubrics that I can use again and again Um, so that's that's nice after you put in the the hour the first time you're pretty much good to go Oh, so you can kind of front load the work then Yeah, because oh, okay, that that's great because I mean one thing that you all brought up was how much time collaborative learning takes for you. So um I also really liked how you mentioned kind of that scripting for students at the beginning because That is something that is backed up in the literature on this again and again Is that students need scripts and examples of roles and how to interact with each other like so Yeah, yes, and I think that actually next time I teach this course. I'm going to provide sample work plans Because students even if they try to work to identify all the steps, they're still going to miss some So I'm going to provide them with actually more structure for that piece. Oh cool. Okay. Um, brann. Were you going to say something at the beginning here? Yeah, I'm happy to jump in now too. Um, just to build off of what you just said kathryn I think one of the keys to collaborative work in the course that I've taught which I've taught I mean, I think since 2011. I've probably taught it five times. So quite a few times Is actually providing them a sample facilitation guide and a template to get them going for the assembly activity because it does It takes me a couple of weeks for them to understand what the assembly is. So I have to model it Two times before we have a student facilitator come in and then these supporting documents have been really key to having them them jump in And so in terms of the grading I students are very grade centered. I've noticed and Um, so I build I have sort of a scaffolded approach to grading and giving feedback In different ways to students for this particular assignment It's not something that I do for all of the course assignments, but for the assembly Facilitation guide and facilitation activity in general Um students I have TA support. So the students um are required to turn their guides in on wednesday if they're facilitating on friday And so the TA's hold office hours Tuesday and wednesday so that the students have a chance to kind of get some feedback From them before they turn it in to me And then the feedback that I don't give them a grade before the facilitation because the whole process itself is is graded from the In-class experience that they create the guide they've created to do that And then their co-facilitation and engagement in the canvas discussion after Throughout the the weekend. So the discussion is open from friday after class through wednesday of the next week. So it's an entire cycle of of an experience But I give inquiry based guidance. So I ask questions and Based on those questions, they go back and make adjustments or they respond to the question and saying Oh, yeah, I thought about that and this is the solution that I've come up with For that. So it's sort of a dialogue before the friday session So it is a lot of work for the instructor and the TA, but it's so rewarding because in a class of 80 This is also my opportunity to get to know them and how they think and and to really um get them excited about co-facilitating and I think that um, I've had at least three students over the years go on to do student directed seminars Which are opportunities for students to actually lead a course on their own come up with a syllabus and stuff like that So I do think that that's a very small percentage But it's something and the goal is to get them doing things um, and then the day of the facilitation Peers give feedback and they use the same rubric to give feedback as I use to give feedback and I have a I have a daughter who is in the Vancouver school system and so they have a four point scale It's like extending proficient developing beginning And so I have created wording for each of those categories and then they get that plus uh a number Grade and a letter grade of course And so I I think it helps giving that general feedback and then I give personalized feedback of course before and after the session Their peers always score them extremely high. Everyone is extending in their mind, but their comments Say something different So it's interesting just um Seeing how they want to rate their peers as highly, but then they do want to give um, you know, kind of constructive critiques And so I I give an I anonymize the feedback. It's always anonymous anyway, but I give the peers who have facilitated all of the comments that have come um, because most students even if it's a criticism, it's very uh properly formed and it's constructive So I give them all the the feedback and then they do their own self-reflection so that Metacognition as part of learning is is built in so it's very complex um, but I do find that Because I use an inquiry based approach to giving them initial feedback. They also use initial or inquiry based Approaches to giving themselves feedback like I could have done this differently. Could I have done this differently? Yes, and if I would have this is what would have happened. So that kind of aftermath reflection is really important too. So That's kind of what I do I I love that because I know as a graduate student for instance. I didn't really learn about the intricacies of giving kind of constructive critical feedback about other people's work until Really my master's program because it wasn't really a soft skill that was taught Um in a lot of my undergraduate courses. So yeah, I I love that you're doing that And shavon, um for your course um, how did you like assess? The collaboration of your students I suppose because they were they were they were posting these things online, right? So yeah, so I mean it's always it's really tough and I've used more peer feedback in my third and fourth year Uh courses, but that's usually also where um The the the project is a is a term-long project and it's worth Like in one of my fourth year seminars. It's worth like 55 percent of their grade. So it's a it's a decent amount so the collaboration and the peer feedback is really really important in in um In that piece whereas here with the first year it's um Uh, this is 15 percent of their grade over the whole term, right? So it's a It's not that it's not important, but it's it's much more about um The soft skill if you want to call it that but as Brienne said students really do want grades as well Um, and so that's the tough piece. Uh, so I don't use any peer Feedback in the first year courses. I I've just found that that if there is a real issue At least my experience and having taught this course numerous times If there's a really big problem the students have tended to come and speak to me. Um But otherwise it's usually just the usual like free loaders or someone who's very opinionated And honestly, that's a skill that I feel Important that the students work it out like between them But in terms of grading it so the the the submissions that the students make are graded on a weekly basis Um, it's easy to monitor collaboration when it's in person. Obviously whoever is not there just doesn't receive the grade um online how we did it using microsoft teams was um looking at the The chat or if the students use the video option It posted that or if they did it in person. We looked at the the drops of the um the various Submissions that they made their file there And and i'll be honest that wasn't done on a weekly basis. It was the tas and i grading it was kind of more you get a sense Uh when we did it in the online, we let the students work the whole term with the same group because you know It does take longer to build rapport online um, so as more checking in I would check in on the various groups Uh over the course of the term and if I got a sense that there was one and it did happen There were a couple of groups where it was always one person posting in the chat frustrated Um, where is everyone? You know, we arranged to meet at this time. Why aren't you here and so on? um, I would then reach out to the group and and try and figure it out and obviously always honor the person that was really putting in the effort Uh, it does take a lot of work. I won't lie, especially in first-year classes. Um, you really um Yeah, you have it's that you spend the time and you have to you know, it's I mean, obviously first-year courses I have I had a Six ta's for that because it was such a large course aside from might um But of course you all have to be on the same page and asking them, you know, that it's not about necessarily Uh looking for problems but it's more looking to see that there is like the students are being able to meet and see the the Why they're doing group work, you know, as I always joke at the start of the term and say I don't make you do group work because I'm trying to punish you, you know, it's uh Yeah, it's an important part of learning So it looks like Breanne. You've done the same there. Yeah. Yeah That uh that last thing that you mentioned though, we're not trying to punish you. We had a discussion um Kind of in the planning session surrounding teaching evaluations and collaborative learning um, and that Some of you have found that your teaching evaluations go down When you use collaborative learning in the classroom or do group work like Can can you speak about this a little bit and maybe how we could better support that as an institution? So um, sure. So, uh, I've done collaborative learning in a number of courses. There was one course I had taught for five years. Um with With a collaborative piece and then I taught it one year I was just so tired and I took the collaborative piece out And I got the best evaluations I ever got And it was so frustrating. Um, because for me it was like way less work and I I felt, you know, I felt bad that I hadn't done it And and then yeah, so it just made me feel Like what are we being evaluated on and I think that the benefits of collaboration might not be immediately Prominent in the students minds, but like six months later a year later their first job or something Or many of them are in work now, but you know down the road. I think that there are these intrinsic benefits that don't come out on evaluations Yeah, and When we surveyed students 77 of them said that they thought the collaborative learning activities that were being run Were positively impacting their learning And then when asked if they would then like to do these collaborative activities again There's something like 50 of them said That they'd be amenable to that so they said that they were effective but they didn't necessarily want to participate in them and Perhaps that has to do with how much work Is associated with it right how much extra work both on your end and on their end. So yeah, and like shavon brand like have Do you have any ideas on maybe how we could Get better buy-in from students about collaborative learning? No, I mean no, it's sorry. No, no, but it is it's a case of I mean because 100 percent I And I'm in the privileged position of now Being post tenure. I don't have full professor yet, but so I tend to Be more open to doing those like your collaborative things that I know kill my evaluations because You know, it's like well What can I do? Um, and I think the only way is to keep doing it and and it which made me very sad This term coming back after parental leave to see so many Of my colleagues going back to in-person exams You know moving away from take-home exams to stage exams where there was a collaborative piece or projects over the term because of the fear of generative AI And so, you know, it's like we just got to keep pushing on and so it's like we're not going to I mean nobody like I like sitting in my office and being left alone and when someone comes and knocks at my door and says We got to go for a coffee and figure this out My first reaction is no leave me alone And then we go and then it's great, you know and our students are the same like it's like Of course we have to we have to push and so the finding of the survey Is absolutely spot-on like it's like Um, it's like going to the gym exactly or getting on my bike to cycle up the hill. It's great once I'm here, but you know um, yeah, so I the only answer I have is just to I get to make that's why I make a joke out of it. I'm like, I'm not punishing you I'm I you know, I was like I I use the example of as a faculty member I've chosen to be in a faculty member because I get to spend most of my time by myself Um, you know, but but but then when I do have to collaborate, it's very rewarding But it does mean that I have to go outside of my comfort zone um, yeah, so I think it's just Keep on Pushing forward. Yeah, so sorry. No easy answer Oh, I was expecting you to solve this problem right now here and now Now Brianne any ideas Yeah, I mean I I actually I was trying to think about it I can't recall off the top of my head what my student evaluations were in these courses, but um I think my students had a really positive experience. I during the pandemic I think the overload of technology would have been mentioned for sure In that term that I that I would have implemented this plus canvas plus zoom Right. It was everything all at once, but I've actually um I've had students right in the evaluations that they haven't only gained knowledge and classmates, but they've gained eight new friends And that's always my goal. I always express that this is one of the reasons why like shabon I do the same groups for the entire Semester because it's like a family they come in on fridays and they're just they just unload on each other They have the best time. They know how each other thinks they know what to say um And so I really do feel that they've built interpersonal skills and they've made friends and I think one of the keys to the success of the assembly is that the work is done As an individual they do the facilitation guide alone And then they have me there and they have the ta support also And the collaborative the collaborative part Comes in the classroom space. And so by the time they get to that point They've already built their confidence up through getting some feedback talking through their questions and sort of posing Some insecurities that they might have going into a 30 minute session And so I think that helps a lot, but I just ran into a student at the grocery store The other day and she's like I still meet up with my 280 group and I think she was in the class four years ago So that's that's seeing something Um, so I think it is it is fun But I'll have to go back and look at my student evals and then I will Send another note to you guys on how that went. I can't maybe I'm too optimistic But I do remember the I was a disaster in the technology in that 2021 myself With all the stuff going on so I can only imagine they were like, yeah, okay No more We are at 15 minutes too. I would like to post some questions. Um That have been put forward out laura meek had a really great question. Um, she asks, um, I'm also really interested. Um In this uh, she says I'm wondering how you've all Incorporated collaboration into your own work and did you receive or provide training in how to collaborate? That's a really good question. Um, I think, um I have not received any training in collaborative Work up to this point, but I collaborate every single day. Um, I think probably like arts I said, I wasn't at the presentation, but I can imagine Our department uses teams probably in a similar way like for all committees the the chair has A channel with the committee members. All the documents are there to organize and stuff like that um, so there's that side of teams that helps facilitate the The team dynamic of an academic committee, for example um, and then on grants and things I think collaboration has saved me because You know, sometimes I'm up at like 5 30 in the morning and I've got time and I want to work on my part of something and I want to share it and then You know, it just gets things going in ways that are functional for me and also for the other team members And so I'm I'm very used to collaborating like that, but I cannot remember if I've I don't recall ever receiving training on working with others Unless it was a I was a part of a TA team as a graduate student I remember that that we had a coordinator and we worked as a team to create exams and things like that but beyond that no um I also haven't received any formal training But just on the job like I would say maybe one third of my courses have been teen taught over the years And so that's a really really close collaboration with the other instructors on the course And then I actually do a lot of research across disciplines And that's just been really fun and interesting. I think chevon mentioned that as well Learning, uh, you know how people approach things in engineering or whatever um, or whatever discipline and um I do find though that the university structure where you are sort of submitting your merit report as an instructor each year and literally competing for bonuses with other faculty in your department kind of fosters like a culture where you're supposed to Identify all the things that you've done and not really emphasize the collaboration And so I have found the most successful collaborations are actually with people at ubc But outside of my department where we're all sort of mutually benefiting and we're not positioned institutionally in any kind of um competition Yeah, just say echo what what what katharine said there that I think for me also being um in Well, I mean being a part of the educational leadership as well at ubc means we've had to create uh And because I've been in it almost since it was created And so what does it mean and and the only way when you know at the beginning being one of the only faculty members in my department Have just by nature having to reach out and work with others um And so that that is a benefit in itself like and then of course, yeah working across disciplines Any formal training uh what I will say and anyone who's interested what I find most helpful and I hope it's there in your discipline um in geography we have And then I've also done it in my software work, but we have like collaborative writing groups where as part of conferences and that you you actually write a paper together on a topic um It comes from in in in the kind of more science the geosciences they tend to write multiple authors on a paper So in the human social sciences, we've kind of brought that in too in that uh, yeah, it's around Working together usually in a like a fixed period of time um to To to write a paper on on a topic And that that for me was I did one as a Grad student, I think yeah, and then I've done three since then As a faculty member and and those are all that international collaborations So that then helps but and I would encourage you to seek that out if that is something in your discipline If not, definitely if you're interested in subtle the scholarship of teaching and learning. It's a huge Uh piece of subtle research. It has to be collaborative, you know But it might be there in in in your respective disciplines as well Yeah, just learning by doing and you know them in groups making mistakes and dealing with maybe uncomfortable social situations and such, right? These are all part of the learning experience and I know I I know all of you are great models for this in your classroom. So That's in the literature. Anyways, that's what students have Said that they they need most is a good role models on how to collaborate in the classroom And that can be done at the lecture level and then that can feed down into the the discussion groups and uh in group work, right? So Jason, do you have any, um Last minute questions or have you seen any questions from the chat? In the chat, I think it might have been answered in the chat but about um, what I guess there was the question of what portion of class time dedicated to collaborative work And then also the grade waiting.  Yeah So I don't know if there's anything any of you want to add to either of those questions that weren't already answered in the chat That's about a third. I'm roughly just under a third overall over the course of the term But worth 15 percent. So it's it's not quite for the reason that I said that that And this is I'm talking about the first year context As I go higher up in the levels that I teach I would increase the waiting but not necessarily the amount of time in class Um, because I feel that for first years, you know, it's still a it's a process Um, and so, you know, don't allocate as much of the grade but still the same amount of class time Yeah, I think Catherine and Brianna both uh Answered in chat, right? Yeah Yeah, any other questions Jason that that you have any follow-up No, I don't think so No Well, I mean, we're basically at the end of our time But I would just I'd really like to thank all three of you for taking part in this project and Uh, we do have a resource available for everybody. Jason. Do you maybe want to oh perfect? I just I just posted it. So yeah, we just um, shared the the evaluation report Up on the arts ICT website along. There's also some Catherine shared one earlier from her where we have all of the instructors participated here have a short teaching story that explains their Activity in a little bit more detail and I'm shavon You have your art the article that we've linked to the conversation article that we've linked to from there that Describes the course and some of the evaluation results from from each of the courses So if you want to learn more about what they've done, you can find them there And and if anybody has any questions about the evaluation report, um, we're happy to answer those later. You can reach out to us Yeah, what I found most interesting about this report is that it draws a lot from the student experiences as well as the faculty experiences. So um, there there's a lot of like illustrative quotes in there from students about The benefits and the barriers that they faced while Um participating in collaborative learning as well as the technologies that they were using. So yeah Oh, thank you, Laura Okay, and I'm very excited to see what all three of you do next in these sorts of activities and how they evolve So, yeah, I hope that we get to do a follow-up study or something. That would be really great. Yeah And uh, with that, I think we will say goodbye to everybody. Thank you very much for joining us today Uh, it was a pleasure having you all so Bye. Bye. Thank you. Bye