I'm Jason Harlow. I am a senior lecturer in the physics department at University of Toronto. I teach a course called Physics of Everyday Life. It's a science course meant for non-science students. It has about 350 students in it, some from humanities, some from commerce. And I have two midterms in the course, and I crad marked both of them. And I have eight graduate student TAs who have to share the marking across all these students. So being able to scan the tests and have the TAs mark from their offices and each TA mark a particular question to keep things fair seem to streamline the process quite a lot. So the ability to add comments was very useful. So I like students to get feedback about what they've written and particularly where they've lost marks. And often the same mistakes come up again and again. And normally when you're marking by hand, you're writing out the same comment again and again and again. But what my TAs found themselves doing is they kept a word file open where they had pre-typed out the standard five comments that keep coming up again and again, and they just cut and paste them right into crowd mark. And that way giving students good feedback and also speeding up that process. So that was good. Also, I guess crowd mark has these features where you can use keyboard shortcuts to go through. Because a lot of my questions are either 100% or 0%, not a lot, but maybe one in five or something. And so just being able to say, okay, 10 out of 10, just type 10J, it goes right to the next exam very quickly. And that helps speed things up. Which by the way, when you compare it with grabbing from top from the stack, finding page three, and then writing 10 and then writing on the front, then putting it in that stack over there, it's a lot faster. In the remarking phase, I just told students to send me an email. Any time they had an issue with a question that they thought may have been marked incorrectly. And what I do is I can find that student immediately on my web page, look at the remark, remark it according to the rubric that I gave my TAs, and then either change the student's mark and then respond to their email and say, problem fixed, sorry about the mistake, or I can explain to them in a little more detail why they got the mark that they did. From a student point of view, you want your mark back fast. That's very important. And when I have a big stack of tests, or I did last year, had a big stack of tests that had to get passed around physically between eight TAs. I had to finish with that stack and then pass it to the next TA. It would take sometimes two weeks to get a mark back to the students. With crowd mark, all my TAs were marking simultaneously. You can see these marks filling up on that marking grid, I guess. I could see it from my desk what was happening. The TAs were getting it done. I said, let's have a deadline. Let's try to get this done by Sunday. This was the test on Thursday. So Friday was spent scanning it in, April scanned it in. Then Saturday and Sunday, my TAs, wherever they were, one of them was in none of it doing some research, but he logged on, got his marking done. Everything was done by Sunday at around 6 p.m. And by Monday morning, the students had not only had gotten it marked, but they were able to log on and actually look at where they lost marks. And they could still kind of remember. Because if it's Monday, you could still kind of remember doing the test on Thursday. But if you did a test two weeks ago, it may as well have been infinity ago. You can't really remember what you were doing. You no longer care is the main thing. And so this has actually turned into a teaching moment for me because they can remember what they did, why they lost marks, and that's something that they're never going to forget. Oh, okay, I got this wrong. I had my physics mixed up and lost some marks, but they're not going to forget that in the future. Whereas if it's two weeks turnaround, there's just not that opportunity. My TAs were estimating it was taking them about half the time to go through and mark. So we scanned 350 students, four pages. That's 1,400 pages. It took her 50 minutes on the clock to scan it. That just got the images on her computer, and then it was about another hour for her to upload all of those to CrowdMark. That scanning time was worth it for sure in that now we have an electronic record of what the students did the time they did it, which can't be modified. It's somewhere up on the cloud, and the students can see it. I can see it, the marker can see it. We can all see it simultaneously if we want to have a phone conversation about it. It's all there on the web, as opposed to being in a physical piece of paper that has to be moved around. So it's a way of archiving their work as well as marking, I think. Yeah, so one question I sometimes get from students for long-answer questions is, are we allowed to write in pencil? And when I first got asked that question, I said, well, why wouldn't you be? Of course you can write in pencil. What if you make a mistake? It's going to get very messy if you keep crossing things out. And students told me that with some teachers they were told if you write in pencil you can't ask for a remark because there's a suspicion there that you've made some changes after you've got it back. And then, oh, I want it remarked. Look, I did it right. And then you get an extra mark. Well, obviously with crowd mark, that's not possible. I've scanned what they've done in pencil and the pencil scans seem to come out fine. And that's what's marked. And in fact, they never even, I don't even give them back their original version. It just stays in a box in April's office. And if they want to view it, they can. What they get is just the scanned version. So it takes away any temptation for any of that kind of shenanigans. Probably about half of the requests that I get for remarks are legitimate concerns. And maybe half of those actually get their mark raised. I mean, mistakes do happen. Especially when my 1TA is marking 350 tests. Sometimes a mistake happens actually either way but you very rarely hear complaints when students get a mistake in their favor. And then the other half of requests for remarks are kind of phishing for more marks that just usually result in no change. So as a professor, I've trusted that my TAs are off there doing a great job marking. And usually the first time I actually look at the marking they've done has been when I'm remarking students' tests after I've distributed them back and I've tried my best to review things, but it's a little hard, especially when you're under a time pressure. And I've found that with crowd mark, part of what I'm doing when I'm waiting for the TAs to finish marking is I'm just scrolling through and looking at the marking work they've done. And that has helped me... I guess helped me advise the TAs on how to be more consistent, especially if I have more than one TAs marking the same question. If I see that one TAs is marking not the way I intended, I can more immediately give some feedback and get them to correct what they're doing.  I really think that should have been a four. You're not really following the marking scheme that I gave and then when the TA had an exact, actual example in front of them, they were able to see what I meant and change their marking scheme. So it just has made, I think, things a little more consistent and fair for the students. I could be looking for something similar, I think, if I couldn't use crowd mark because it's going to be hard to go back to the old way, I think. Well, these are some marked work. So this student got two out of six because they didn't put anything there and there's some comments about what they did, why they got plus two, but why they didn't get these marks. I can go down to the next student on the list who got six out of six just to mark. No real comments when they get 100%. Lost a point here for something. Wrong units, calories per gram, so a little x there. Half a mark there. So I can pretty quickly see what my TAs are doing. That's question two. If I want to move to question three and see how people did a different question marked by a different TA, I can immediately see what each student did, why they lost the marks. So you can either, a TA can actually use the mouse to write on the screen or they can trash that. Or they can double click and actually type comment, Harlow comment. If I was to remark this one, I might put Harlow comment and then type in my thoughts about that particular question, but I can also just cancel that. And then if you want to look at the whole marking grid and see all the green means that it's been marked by a TA. Scrolling down there. And so we can see this one's pretty much marked.  It's a way of sharing copies of all of these tests so that we are carrying around physical pieces of paper that we have to keep leafing through.


Thank you very much. Hi, everyone. My name is Florence Akhara, about me. I am a Kenyan citizen, but I spent 20 total years in both Kenya and Tanzania, so I mean in Tanzania and Botswana, so I like to feel like it's part of a flair of who I am. I'm a corporate lawyer. I worked for five years doing corporate law before branching out into social enterprise management where we were doing menstrual health education.  So when it comes to data, honestly, I feel very scared of data. I'm very shy about it. When I was working in the non-profit sector, I worked with the data team. My job was mainly coming up with stories, understanding the patterns, why people were behaving or responding to the data in the way that they did. When we went into the field, the kind of engagement we got from the participants, so observation of the participants and understanding what patterns they had engaging with the people who were collecting data and also the kind of models in which we were using to collect the data. The program model which we were operating with in Kenya and Tanzania was predominantly using local facilitators. Local facilitators would create, would collect the data, would have a better understanding of the cultural patterns and spend enough time and have a context and understanding of what the situation in each community is. So it made them more sensitive, which I think is very important when you're collecting data because sometimes, particularly if you're dealing with sensitive issues like gender, you can find that people become apprehensive to when you're collecting surveys and trying to ask them all sorts of questions. Why decolonization? So I picked this up not only from the experience of the participants in the field, but also from my own experience with my fear of data. Colonization did something to Africans when it comes to the way we see ourselves, the way we position ourselves. And for me, that has been largely what has been the pattern when it came to Africans engaging with data. The fact that we just don't see the capacity, we don't feel like we're adequate, we feel like we are steps behind and there's a lot of catching up to do. A lot of the colonial structures and systems are still very highly intact. You can see this particularly in the way we treat women. The way we engage with non-Africans in the field. An example of this I'll give is when we were doing workshops in the field in the past, we used to have a lot of volunteers that would come and help us distribute the survey questionnaires and ask questions to the participants. Usually the teams we worked with, the participants we worked with are between the ages of 10 and 18. And when we were doing this, we did two, we started with having international volunteers coming to the field and doing this with us. But we noticed from the surveys that we put out, some of the questions were not being answered, some of the questions were being answered too perfectly. Like questions like if, how many toilets do you have in your house? It seems like such a simple question, but if it's a white person asking a black child this question, you find that there's an intimidation. Nobody wants to look bad in front of anyone. So sometimes they would lie. And we realized that this wasn't a model that worked. Another thing was during the sessions, we found that the facilitators would be at the front. So we had, we have predominantly African facilitators and when we had volunteers in the back of the room, just observing or taking pictures during the sessions, the students were mostly distracted. They would stare, they would, you know, maybe they've never seen blue eyes before, long hair, all those things. So we learned that we needed to shift the way we were working by making sure those more local people so that the effects of colonization, the fact that Africans look down upon themselves, the lack of local context and culture understanding created a huge barrier towards the reports of the data that we're receiving. The genderization, why this? Speaking not only from personal experience, but also the positioning of women in Africa. There's a lot that we need to do in terms of understanding how to work with the patriarchy because it's just like colonization and the effects and consequences of it. It's just not going to go away today. You're not going to have feminist women all of a sudden and to make matters worse, the way we communicate feminism in Africa portrays a measure of trying to replace men because, you know, the people who are in the forefront of speaking about gender and feminism are usually white women and some African women do not want to be associated with feminism because they feel like it's denying their culture. So we need to look at how we are communicating what feminism is. To reshape the narrative, I think it's really important to look at two things, the program design and the program model has to keep adapting.  You need to go into the field to understand what is the cultural bias that the people have. And another thing is to understand that Africa is a continent so each country will have a different understanding or bias and some communities will be more accepting and others will not and even in terms of exposure. So in terms of the program model, once you've designed it, it's important to learn from what you're implementing in the field. One of the things I will say, oh snap, I think I skipped, yeah, one of the things we learned in this picture, it's not very clear, but I'm here with some boys. Previously we used to do menstrual health programming just for girls and we learned from that model in that, you know, girls are very happy to learn and understand and they picked up our model very easily. But when they went back into, you know, the community and something happened, boys were the first ones to laugh, particularly if they stained themselves during their period. Boys were the first ones to laugh at them. Boys were the first ones to ridicule them, to tell them that now they're ready to have children and, you know, just shame them and make them feel like they're less worthy because they're having their period. And when we decided to shift the model to educate boys, we learned that boys are also now very interested in contributing to make sure that, in contributing to make sure that boys, that girls feel safe because now they understand that periods are just normal, it's not an affliction, it's not a disease, it's not an ailment of any sort. So based on my understanding and my experience in the field for the past five years, I think that this is the best program model for accurate data collection, particularly if you're talking about gender, make sure there's male engagement and women empowerment so that you can break down those patriarchial models. You can also make sure that the language, local culture and local authorities are also involved because this is the only way you can make sure that there's ownership, true ownership and true willingness to engage and also for the community to feel like they also want to learn and grow from the challenges that they're facing. They will also help you come up with the solutions because in most of the cases, even including boys, it was the boys who said like, we laugh at them because we don't know what's happening. Yeah. In a nutshell, that's what I've been doing for the past five years and if you have any questions, I'm here. Thanks Florence. If there's anything, any question in the room, just raise your hand and I'll come to you with the microphone. Yes. Well, first of all, I think this was a very interesting talk and I loved it. I wanted to ask two things. I don't know if I missed it, but I wanted to know more about the menstrual program itself. Like, what do you teach and what kind of resources do you give kids? I don't know if it's only for kids or also for teenagers. And I also wanted to know regarding this thing of also teaching boys and not only girls. If you took some measures, like for examples, having boys or men talking to boys and women talking to girls or if it was more like, or if it was the same. Okay. I guess I can answer that by telling you what the current model looks like because the current model has been refined. In the past, we didn't have any boys, we didn't have any men, but now even in the community session. So the program model at FEM, we work particularly, we were working with 10 years old to 18 year old girls initially. And it's a one year long program. I mean, one year long program and data collection cycle. It starts with, we start with an introduction to the community. So in that meeting, we will invite the parents of the students, the community leader, so the chief or any county leaders, depending on where you are. And then we will educate them and inform them. Like, we have a comprehensive menstrual health package and we educate them, we even show them the products. We worked with two products, which was reusable pads and menstrual cups. After educating them, we would then go do a survey once they approved it. We go do a survey with the students in a day and then just find out, like, what are the challenges because each community in Africa is different. Some areas, the biggest challenge is water. Some areas, it's just the lack of information or the lack of resources. So once we did that survey, we came back two weeks later. Once we understood what the specific challenge was, maybe lack of understanding of what virginity is, we'd educate using that comprehensive menstrual health curriculum and then focus on the specific gaps that we found during the survey. So the workshop takes five days, two-hour daily sessions, and then we come back six months, six weeks, six months and then one year. So six weeks, survey, six months, survey, one year survey. During the workshop, the student would receive a product of their choice, whether they wanted the menstrual cup or the menstrual reusable pads. And then the surveys that follow up would be based on the product and just the feel, like, whether their self-esteem has changed, their relationship with their period, because for us more than anything, we wanted them to have the information and also just feel confident, build confidence here. So in the current program model, we make sure men are engaged as well in the introduction and in the training sessions, we now have male facilitators. What normally happens is we start the sessions with all the boys and girls mixed, and then when you're getting into really technical things, like the bodyworks and everything, some communities, the boys want to be split from the girls or vice versa, and we honor that. So we have standby male facilitators who would then come to focus with the boys, and then the girls would be with girls facilitators. But we leave it up to the community to decide that. Thank you. Other questions? Thanks. Okay, let's hope I can express as well. So I've noticed that within Africa, a lot of the problems or issues that we work with, so basically, menstrual health, et cetera, a lot of it comes from cultural issues. So even though you are educating kids, there's a big cultural issue or this cultural mindset that maybe the mental cup is bad for you, and certain things are right, some things are wrong, boys shouldn't learn about it, and it's being taught generation to generation. And then you go into schools and try to educate kids, but they still have, at the back of their mind, they remember what their parents have taught them, what their grandmothers have taught them. So how do you actually deal with that? And is it an issue that you see, or are kids more open to now unlearning? Yeah, well, kids are always, I feel like in our experience, kids are always very quick to pick up the new things, particularly the menstrual cup, they're always very excited. But we noticed that in the previous model that we would have the kids taking the cups after doing the workshop, they're like, yes, I want to try this, then they'll go home and their mother will be like, no, you can absolutely not try this. And that's why starting with the introduction phase with the parents and the community leaders and showing them everything, showing them how the product works, eliminating the cultural perceptions, particularly virginity and breaking the hymen, those are the things we would focus the most on, so that the women would also pitch to them to try and appeal to them. So for instance, in some communities, in a lot of the communities, the men would care about the cost of the product, because mostly they would be the ones providing the funds for it, and they would also care about preserving agricultural land. And in most rural communities, you find waste management systems inadequate, so you'd find pads just thrown around because dogs dig them up and things like that. Men also spend so much money when you do the cost value from using single-use disposable pads versus the reusable products, we found that they were saving up to 83%. So we would pitch to the parents, but also just give them as much information and allow them to question us during that introduction session as much as they could, so that if they felt like there was anything they were not comfortable with, we would then sit down and address it, like what's the issue. We've had a challenge in a Maasai community in 2019, we were actually chased from the field, because the Maasai tribe is a very, it's one of the very, how can I say, very conservative tribes in East Africa, and for them to have young facilitators, women coming to talk to men about periods was very offensive, so they chased our facilitators from the field. So I think that's one of the points we realized, we cannot go to any community without speaking to the leaders. So having authorities was also another one that we shifted. The introduction to the parents meeting is something we implemented maybe like three years ago, it wasn't very, like it's not one of the things we started with from the onset. We have time for one last quick question, if anyone has one. Yes. And what do you do with the data collected? Who uses it and do you go back and inform the people you collected it from, even when policies change? So my main role was to come up with stories for fundraising, because that was my specific duty with the organization. But we've been collecting that data for, you know, I joined FEM five years ago and it's been in operation for 10 years, so we have like 10 years worth of data. We're at the moment trying to figure out who we can sell it to as a way of fundraising. We have collected data specific to the products that are preferred the most by the students, but also we want to help costs, like do an economic costing of the program so that governments in East Africa can then use it to prepare when they're doing the annual budgets. If they want to do gender programming and they want to include menstrual health in the package, it can be very clear to them how much they'll spend based on, you know, like our economic costing covers everything right from the interviews with the introductions with the parents, like what are the number of letters you need to write, what are the number of meetings you need to have. We're doing this with, at the moment, there's a research in a town called Muanzah together with the National Medical Research Council in Tanzania as well as London School of Tropical Health and Medicine. So we are not very, how can I say, I mean I'm not a data person and we're not very comfortable with the data to just freely use it yet, so we're still trying to figure out ways. So yeah, hence the process of figuring out, we're figuring out how we're shopping to see who we can sell it to, but also through this research we hope that the report would advise governments in East Africa how they can continue managing their programs without like funding other non-profits or like other non-profits just, you know, continuing to, what's the word, sorry, but like we feel that there's a lot of money that goes into investing in gender programming for menstrual health, but there's no impact at all. So we feel that this model and this data should be able to give that insight on how to make a more impactful program, like, yeah. Thanks very much, Florence Sakara. Big round of applause.