My name is Don Brower and I'm at the University of Notre Dame and then the very Family Center for Digital Scholarship, and I'm going to present how we organize and ran some student annotation projects last year. I'd to first acknowledge my colleagues who are collaborators on this. It was a team effort, in particular, Peter Cornwell of Data Futures, Nali Myers and Julia Vecchio, both from Notre Dame and Eric Decker at the University of Basel, all of them contributed to making this successful. So, over the last three semesters, we ran seven annotation projects overall some of these projects. We were planning some of these projects for a while but they were all prioritized in response to our institutions COVID related stay at home orders. We all suddenly like all our library student workers and also student workers from elsewhere on campus, either way to work remotely so that you could get paid for their work study positions, and these projects help to the bill and they also provided some useful scholarship. So we focused on what we call structural annotations. These are the kind of things that help others or help you find your way through a pile of images or to locate things that are used for further analysis. It's like some examples would be like finding chapter headings or locating diagrams or identifying math legends and so on. Sometimes even marking page numbers can be useful. And our projects ranged over a variety of corpora we have the epistemological letters which were kind of 1970s physics philosophy zine, in a way. We have some Vietnam government newsletters from the 1950s, a medium sized map collection and then we have finally the East Asian treaties, which had like 10s of thousands of pages of scan text. So, each of these projects we collected different types of metadata, and we and two of them have had the results pushed into custom and venue repositories for access so. All right, so to kind of set the stage is kind of what the just so we have something in concrete and think of. This is kind of what the annotation viewer looks like we have. We're using mirrored or two. And we have like the image in the middle with annotation tools in the upper corner and then we have these workflow links in the bottom left and that's actually I'll talk more about the workflow later so that's actually a key thing. So this is what annotation would look like. In this case we're identifying a title of a map. So in this in this annotation we had some metadata which is a transcription of the title and then we also had, we're classifying the page, the page type so in this collection, like there are road maps so we're trying to pull those out for example. And then here's another example of annotation this is from the physics zines, if you will we were identifying equations so. Okay, so let me let me start just go over the technology really fast. We, the technology is standard and as simple as possible. So basically items being annotated restored as imagery sources on the triple if server. We kept the annotations in a separate database kind of a standoff annotations. We used orchids to identify the annotators and to sign in, and then we used your door to instances, custom one for each task because we had to adjust the metadata collection and then we used the data futures and a store service to actually be like the workflow server attract each unit of work and and who is working on it and whether it's finished and then in some other workflow things. So, people overall we had many students undergraduate and graduate express interest we also had some people, some library faculty and staff and some other people like some, I think a high school student signed up in the. We had about 19 annotators from at least two different institutions and they made more than 10,000 annotations over the three semesters. And this subset probably wrote the super annotators and they seem to do like majority probably around two thirds of the annotations. So the, with that many people, we needed the one of the one of the two big things we had to do was keep keep everyone organized so to do that we made a live guide page and use as the home base all links and documents were put there organized by project we also put links to like the sign page and the other websites on there. We use a mailing list for communication, we try to keep regular communication on it so like weekly weekly ish status updates. And then we wanted annotators to know that there are people here who are actually helping support them because they can get lonely so we encourage them to contact us with questions and we also held virtual office hours and because for some people they preferred spoken or hands on examples. So, so probably the training was the one of the harder things our workers were at many different levels of ability and they could come and go as needed, or as a desired to address this we had a few strategies first, each distinct tasks had a document describing how to do the task, complete with screenshots and examples of what to do and what not to do. Sometimes it got we went, we went into extreme detail just to make sure people would do things as in a consistent way. We chose annotation tasks did not require expert knowledge and we try to give clear guidelines of what we expected, and then, and try to eliminate as many judgment calls as or as there could be. And then we also encourage annotators to ask us, reach out to us with any questions they might have. Like, they were going through, you know, especially in the treaties thousands of pages there are situations that we didn't expect, like fonts would change or, or the typography or the way things were organized would change so we would be often would have to address special situations. And then in those cases we would go back and update the tasks documents with and, and maybe even have QA, some FAQs and adjust those. So, for the most part we didn't have too many problems with, with QA, some projects didn't have much of a QA process maybe the project curator would spot annotations for others. There was much more extensive the project curator and maybe even with an assistant would do much more review possibly looking at everything. So this systematic problems, we would update, again we go back update the task documents, and, and often this will be like, like a case where we had a subtle ambiguity or there was a easy way to misinterpret something in some other way so we would, we would fix that. And then we would send notes in the mailing list to clarify what we noticed and address ambiguities often students are very conscientious about quality and some of you went back and fix their annotations after after we pointed out some of these so let's see moving on. So these are kind of example of kind of what the raw annotations would look like at the end like we have here like we're looking at from the treaties we have like a canvas identifier that annotation on the page. And then we also have like the metadata in this case a the transcription of the text a date and then the annotator who made that so another example of some annotations this case we pulled out these titles along with their, their sequence numbers this is like a two dimensional sequence number. Okay, so the other problem we have besides organization was just kind of keeping the work moving. We needed the annotators could really chew through a lot of the work, especially, especially when we had, when we when we had a lot working at the same time. So this was a constant immediate concern to find more work. This is probably the biggest challenge some. Some of the smaller projects, people, the anterior school work through in like in a few days to a week. I mean the larger projects were months so that was nice. And we want to work that was useful and not just work for work's sake so we actually we actually try to find things that were well tuned the annotations, especially the structural annotation. And things were where we already had source images ready or almost ready for annotation and we did not want to have to go through a digitization project first. And then once we did that we assigned it like a curator or someone who are internally who would, who would be I could be the person responsible for the research side of the thing they would find the goal and they would figure out what metadata would be useful and we would work with them to figure out what we need to collect. Sometimes this would be actually a research PI sometimes it would just be like a domain expert who understood the material. Let's see here. And then when we chose the task breakdowns. So we, you know, again we chose things that that that we do not acquire expert knowledge or or a judgment. The most important thing here was that we actually didn't broke things down into work units, we tried to identify. And here's to be able to finish a work unit in like, like five to 10 minutes was kind of the rough idea. So usually this would be a single page or a packet of pages so we would make like a sub manifest for just just a few things. And then our workflow, the Anastore service within give the annotator that work unit they would work through it and then they would click next. So that would kind of cement that and it would mark that work unit as finished. So it's a good way to get a lot of people to working in parallel without duplicating effort on these corpus. So, alright, so now I like to. Like, so that was kind of the house of what we did. And, and I'm going to discuss some thoughts about how well the whole thing worked out. So, okay. So it turned out using orchid was actually a very good idea. We can handle people from different institutions and we did have a few different institutions. We also had like I said a high school student so we have people who didn't have any institution and no problem. And additionally we had we had now we have a scholarly identifier persistent identifier for the person so we can we we associate that with the annotation so that way when they get reused we have a, we can cite everyone who contributed to it to them. So that was probably a teaching moment for students they could learn what orchids are. And then we also, it's been so it was actually pretty useful and we've used orchids as a sign method method for other digital projects we've been working on. We're very lucky to have we already had a prior relationship with Data Futures and this Anna store platform that was the key piece that let us have, have all this work, have everyone work in parallel and without duplicating effort as I said that was very, that was very important. We, we met our goal of providing useful remote work for a student workers. That was, that was probably one of the most important driving things for getting this done. This isn't me like this is like a year ago, very, and then on top of that, we also this also was a good moment for in the library we could show other people in the library what annotations were and how they were useful and and let and then and then we tend to actually offer this as like an internal service to other people and in fact we have a few projects already lined up just from showing people what we've been doing. So. Alright, so now going through the some problems that we were it was the first problem was, it's just it was challenging to find that work to keep people busy, because one of our tasks was to provide useful useful remote work that was, that was an important criteria that made that just just challenging because it takes a long time to develop projects. Now that the immediate need gone because we've been kind of going back to onsite work this that that made this is not such a big deal but that was definitely a problem at the time. Let's see the technology infrastructure works great once this setup but it was getting a set effort felt like it felt like we're dealing with just slightly more complexity than we could grasp, and then dealing with all the servers and identifiers and it was just just a lot. In addition, one of the things that that we would, I would like to see personally is more is more things involving a discovery or access to annotations like we can we can view the annotations today, I mean, and we have, and we can access to them of course we can also just pull them out of the database because it supports a WADM API, but we also but I want to be nice to see a better discovery or exhibit platforms for these kind of annotations. And then for next time. What I like to see is, or for our next projects which we, which we do intend to keep these going is definitely focus on community. Ideally, I think that's continue this people let people know how, like, get into the subject of what they're doing and also, because they're going to be going through all the images anyway they should like at least have an idea of how it's going to be useful and how and how they're contributing to research. Bring answers to research questions. I also work on the better way of tracking progress this might even be like a slight gamification type process maybe like a leaderboard I'm not sure but I think, I think have given me more feedback as as far as like what has been done versus and how much is left, it could be good for and then also another one is we definitely working on getting better like checklist in place in new projects and then in developing a vocabulary for describing what the annotation workflow would be so we can use the same descriptions from, like, from everyone from like the curator to to the developer. So, because, because, yeah. It seems like a lot of times we're just re explaining what we want, but in different terms because everyone uses a different vocabulary. So. All right, so this is my presentation I guess I can, I can learn a little, a little early so we could possibly go to see if there's any questions I'm not sure so. Yeah, thank you. They're really lovely reflection on, yeah, kind of nice project that got quickly driven by some, some urgent needs. There are a couple of questions here. I'll ask the first one, which is just asking how much work did it take to customize the mirror door annotation form. That's a good question. I'm not the developer who did that so I don't know but I think at this point it is, it is, it is, it is routine, but it still needs some effort so I think this was done by developer data. They, they have like about 30 projects or so usually we're going to turn around in like, in like a day or two, depending on their other workloads so. But no, that's a good question. In fact, I think a lot of these things when I look at it, I keep thinking it seems like there should be more, it should be easier to do something so this is definitely something that I would like to spend some more time on so. Great. And another question is just asking if you can give an example of a complex annotation scenario. So, not sure, unless you're going to interpret the word complex there. I suppose one situation could be, we have, you have complexity in terms of the amount of different types of metadata you would collect. We have, we have a few things where we did we did collect. For the, for the Vietnam newsletters we collected the table of contents, which among other things we had yet, they had people just trans, because there's like, they're like type written in a bad font OCR didn't work so well, we had people transcribe them, and we had the mark that the page number in a table of contents in like in bold, so that way we could machine process it out later like we could keep track of that so. I'm not really sure how us to track with a complex annotation would be. Okay, looks like I can see the questions. I'm going to move on to the next one here. So, so question is, great to see good examples of crowdsourcing, including paid anymore insights and to share about your experience. Yes, this is actually a very, very good thing and there's very and I think I think them being paid was was good for many reasons for it was good for them because they're definitely spending our time on this. It's so good as an incentive to actually get people to actually work on it like we noticed, we have we have noticed a drop off. This last, this last April as people start coming back on campus the number of annotators was going was going down so I do think that going forward like we will want to keep, we want to keep like a paid element or at least have people who will work on this, and that was a good thing I like I definitely like the crowd sourcing is kind of fun being bringing people in as you know the challenges making sure everything's a load seem common denominator. Another thing that we were looking at that we didn't do was trying was like because for annotations annotations have been great. Another thing we're looking at is using as something like the universe for things that aren't quite the level of diversity, but I but no it's been great, but like I said the biggest thing is, is I would do with crowdsourcing is communication, making sure everyone feels like they're like they're part of something bigger, so. That's, that's wonderful. I'm afraid I'm going to, I'm going to call it there, but that's the great news is that we, you know, you, we can kind of keep that chat going in the hoover platform and if you're willing to answer the questions that are there. We can start the next session now so we can start the next session I'm going to say a big thank you. Thank you to Don for for this presentation and provoking a lot of those good questions thanks everybody for joining us. Yeah, and we hope we'll help you see it at future sessions later this week. Thanks so much.


to let people know is that Victoria Sands is incredibly graciously, she's come across a great high-value wristband for AFFES this weekend and since she is heading back to DC this is so if anybody is planning to be here this weekend and would like a wristband so you can do your pub crawling as we call it, just let me know and I'd be happy to give it to you. All right so this morning I am very happy to introduce to you VJ Matthews whose presence has been virtually here all week as you know with Thayer Rogers has been live streaming our events through HowlRound and video documenting other things so we're all and most of us were already familiar with HowlRound before coming here so VJ is co-founder and cultural strategist of HowlRound which is based at Emerson College and prior to that I believe was the coordinator of the new plate of development for the NEA. Yes all right so he's going to be talking to us this morning thank you. I was going to ask you all how many of you were familiar with HowlRound before but I think yeah so many. Yeah so I want to the way I'll structure this is I'm going to give us some background to HowlRound as an organization and then this new project that we have the world theater map and then the second part of that will be actually if you have your computers to look at the website together and so at the beginning I'll be you know talking and then we'll be then we'll get into conversation and actually getting onto our computers. Okay so HowlRound Theater Commons you know I'm a co-founder it's based at Emerson College in Boston we've been in existence for about seven years but we've been at Emerson College for since 2012 and it's a free and open platform for theater makers worldwide and our mission is to amplify progressive and disruptive ideas about the theater and to connect diverse practitioners around the world to conversation and information and ideas and on average we have about 50,000 people who participate in our platform per month and some of the tools that we use for this kind of community building include an online journal where theater makers share ideas opinions and practice and and then also a live streaming network where any theater organization can live stream and our role is to help develop their capacity connection and so it's it's seen as a shared infrastructure like all of our projects a shared infrastructure for the field to use and our role is as facilitators for that and we also host in-person convenings that exemplify progressive values and urgent issues in the field and the new project that I'd like to share with you all today is called the world theater map and but before we get there I just want to mention kind of our our reason for being in the conceptual frame within which we work so when we created HowlRound seven years ago it was a direct response to research that suggested that artists were increasingly decentered or disempowered from theater making within our institutional theater field and then we wanted to also figure out what can we do with existing internet tools to help destabilize and de-center these entrenched power structures and these behaviors that we saw that were kind of counter to artists and also and change and change narratives around what theater making and theater culture is in the U.S. and so we founded HowlRound to push back on U.S. institutional theater culture that we think is very much a modern culture and lacking many types of diversity and which is to the detriment of our overall general theater culture. Too many voices have been left off of our stages and not given resources and not represented inside of our institutions and not recognized for that substantial contribution to to the past and present theater and and we also as a field have a terrible tendency to adopt commercial behaviors and commercial structures to to make our theater even within the not profit structure which was set up to be a public good like like museums like libraries and now or you know for a long time the nonprofit sector has been almost indistinguishable from commercial theater and so we we set about to create a a group of tools that could amplify voices and issues that were chronically underrepresented and under heard and unheard in the theater here so but instead of doing this in a kind of conventional top-down way that a that a maybe online organization or media organization would do instead of being top down and hierarchical having kind of an editor-in-chief or like an artistic director model we we took our cue from the structure of the internet and instead adopted the principles of the commons as the way that we had organized ourselves and we didn't seek out directly commissioning people but instead we created and developed invitations for the theater community to participate and contribute and co-own and co-manage this emerging knowledge commons for the field and and this invitation to the field and it still stands as is around a set of progressive values such as equity and inclusivity and accessibility and diverse aesthetics generosity and collaboration all of these values which we think are very much embedded in commons based peer production which is the way that we make our information our conversation our knowledge and to give give a definition of the way that we're defining commons is it's a social structure that invites open participation around shared values and another definition is that a commons is a form of wealth that we create together that a community co-creates and which is shared broadly without barriers and is co-managed by this community in order to benefit this community and the most well-known example of a knowledge commons is the online encyclopedia wikipedia and which absolutely uses commons based peer production as its way of producing its its content its knowledge and also open source software projects use commons based peer production and in terms of a model I I think how around is closest to an open source software project in that there is a a a small group of people who are working for an institution for a non-profit organization who have set kind of an agenda a set or has proposed a set of values and is inviting a community to create around that set of values and so our roles don't take on a conventional traditional role as being editors but as rather community organizers or facilitators enablers of other people's expression and enablers of other people's agendas and yeah so we'll move on to the world world theater map world theater map it's very simple it's a user-generated directory of the global theater community and it's a digital commons that's free and open to all and it's very similar to wikipedia anyone can add information about any piece on the world theater map the purpose of the world theater map is to make the global theater community visible to itself to facilitate conversation knowledge sharing movement building and we originally had another project that was that we had developed for about five years called the new play map which was just focused on playwriting and and and the us and the purpose of that map which we started I believe in 2009 even before the start started how around the purpose of what that was to actually reveal the way that new work would get developed and produced within the united states that there was this narrative of new work emerging being born in new york city or in major theater capitals and then getting maybe a new york times review and then have a life and that wasn't actually the actual accurate story and so this map was a way to change that narrative for into enable the community of new play developers and producers and playwrights to actually list the histories of of new work and that we could see it visibly and actually see it and also a continuing issue or problem or need that practitioners want is the need to connect to each other and also to connect to more knowledge about the field and and so the map is is hoping to fill fill this fill this need and also a tool like the map can help bridge various kinds of silos that our world theater community are in and help facilitate connection so the map is for literally everyone everyone in broadly what we're defining as the theater field and that means all types of theater makers and companies and institutions you know actors technicians directors playwrights scholars festivals networks producers universities cultural centers and any person can edit any bit of information on the map and it doesn't have to be your own so here i'll go i'll go into this and then in the second part we can we can review some of this so i will i will jump over now to the actual website for the world theater map so here's the the homepage just that world theater map dot war and it's also in English Spanish and French and they all have their own URLs to get there and and then up here you see the actually what's in the database what's in this directory so far so you this you see this many people organizations shows festivals and locations maybe half of this directory came from the previous project the new plane map and the rest is is data that people have inputted since for the past in the past uh we're in the 16th 17th month now of it being public and then here is just one visualization of what's actually in this directory so this this globe you're showing the upcoming events that means what's happening within this short time frame today and and then just to give users an idea of what else can be visualized in the directories here we have people who have identified as female theater makers around the world and and so that's it's very simple visualization and and many other kinds of searches that the directory can be put on this globe as well as listed out and and then here we have the latest posts both live streaming and journal articles from how around and and then again the the numbers for the the directory then the main the heart of the the map is this directory search so i will show you first up an artist profile and i'm currently connected to edu realm so let's see so i'll search for there's these three tabs up here profiles which is both artists and organizations or people and organizations and shows which means titles of shows and festivals so i'll search for this one playwright here okay so aditi is a playwright in from minneapolis and and this i'm not sure if either she wrote this bio or not but again it could be edited by anybody here shows her kind of home base location contact information and then the shows that where she's either playwright or some kind of a primary creator of of these specific titles and then i'll click into this play by aditi and and then you get an overall summary of it and then an actual history of its performances and development work in progress here and then these can be even further clicked in to get information about who are the particular artists who who worked on this this one wasn't filled out so i'll give you an example of that well come back to that so each one of these performance or workshops could be filled out by you know who was the designer for it or who was the director of it and and then this is laid out chronologically and what's what's interesting is that this entire history of this play doesn't have to be known by one person where if for example you were the playwright center and you produced a work work in progress or there was a work in progress at the playwright center they could just or someone could just put this information in and associate it with this play and then the system itself would compile all of these events so it's in a way it's taking a bunch of distributed information that could be isolated and siled from each other to create this coherent chronological story of it okay and then now let's look for an organization so this is a a theater a producer and presenter in Moscow and here they have they're they're about they're all their contact information their social media and and also all the shows that somehow they've helped to support or produce and then i'll just click into this one title human use of human beings and and then there again you can get a synopsis of this particular work of theater and then it shows you that it just had this one performance on these dates and then i will i click on the wrong one sorry i will click into that particular event and then here we'll be able to see this example of the specific artists who were or participants on this particular event that happened July 18th through the 20th and then all of these links link out to each of the profiles of all these people okay and then then another thing about this is networks and memberships and associations which is another way for the various groups of people to be to be given one space one centralized space to find each other usually the speed of this is its instance maybe i should get on to another network here okay so this is uh this is a network based in in australia and they're able to as a network they're able to have add organizations that belong to the network so that they appear on their profile and vice versa for example playwriting australia is able to add themselves as a as associated with the internet work new south wales okay and then i think probably the most exciting part of the directory here is actually searching by by interest so when when someone is filling out a profile of either an organization or a person or a show um they're they are able to select multiple interests so for example let's say we wanted to look for a director interested in climate change then we get this list um and then also if you want you want it visualized you can get it visualized like that and then also you can share these specific searches if you find something very interesting you could just tweet or share and then people will be taken to this to this list or to this globe and and then you can do multiple things like i'm looking for let's say organize some kind of organization a festival for example interested in climate change or a festival that's interested in the african diaspora and um or everyone interested in artist rights and safety and then let's say and circuits and then so you can mix and match and you can have multiple selections with with all of these searches and let's try one more here okay so educator scholar morocco and um okay so that's uh yeah the people and organization search you can also search by cities if you know a specific city all the different kinds of organizations out there as well as different categories of roles that people take within the theater field and and then as well as you could search by a postal code for organizations and people then here's a now we'll jump over to the show search so this is titles of works and for example if you're looking for a show that's where the let one of the languages or a language is in arabic you would get that listing as well as a map of it and then you can do multiple languages if you wanted and i'm just curious is there any sign language yet okay yeah so two that have sign language um and then and then you could also search by a show countries country of origin so let's see shows that originate from argentina and it gives you both events as well as the overall show page and then um then i'll jump over to um festivals tab which is you can go back in time if you wanted to to you know as far as this calendar will take you which i'm not sure i can think i think it will go back years and years um or you can search in the future so let's let's say august from august first on on these are the the festivals that have been listed and yeah so that's that's kind of a an overview of what the map is able to do the in terms of adding information you're just given these two choices you can add a person organization festival or you can add a show event and that means you know either a performance or some kind of development work on a show and you can list it there so um what we've been doing in terms of launching this so it was launched as a beta website in january 2017 and um and what we have been doing is collecting feedback from the community of users through various means through a user forum which is over here and as well as you know people emailing us um we conducted surveys and we also engaged a cohort of of um ambassadors from 29 different countries 29 ambassadors from 20 different countries to help to be kind of a cohort of people um to give us feedback um based on the way that their local communities um were interacting with the map and um and so we we've been working with them for about eight months now and and some of the findings that we've had is that the way people are using the map currently is um I'll just give you some anecdotes so mapping rural theater in India to make those theaters visible to um to urban theater communities and um and then another ambassador was talking about how people in her community for example mapping all the women directors in Ireland to counter the dominant narrative that there aren't in or or or that there are very few of them and um and then another use is finding other theater makers and organizations who share certain values or aesthetics or interests for things like a conference or festival opportunities to connect and discover okay so I think we can now um go back to this here we can now kind of open it up to questions and then we can get on get on our computers to actually do some more things one simple question I have so um what's the mechanism for if you uh if there are errors once you've entered something or you notice a problem I see how you can add new names and things that are missions but yeah so for you so um for example if if there's some kind of error here you just click edit and then um and then you could just you know add something edit here you could even change the title if it needed a title change maybe from good to just bold and um you could add additional primary creators to this and and then you can and what the best thing I mean one of the great things is this can it's iterative the editing process so um someone could add additional um subject categories or tags to make it more findable um and then yeah so that's one way to edit and then you can also for example subscribe if you wanted to if you're interested in just getting any updates to this particular page you could just hit subscribe and then you get an email notification sent to your email telling you what the new change was whenever there is a change so I mean that's that has various uses where it's just to keep up with get informed and another is if you're interested in kind of protecting the integrity of the information there you could always see what's being changed how do you resolve disputes in that instance um no not yet it's been it's been mostly um no disputes yet there's but I'm looking forward to that do you have a mechanism for that no not yet and um another thing that can be done is just deleting because there's also in terms of um some of the data gets duplicated or there's some kind of technical errors if someone's connected to the internet and then it goes out then some maybe a duplicate maybe created then anyone can just hit delete and then we're notified that there was a delete request then we check it over and then if it's good then we can delete it more questions so I'll add on to that um but since you don't have a mechanism yet for solving disputes I was just wondering because I love your idea of progressive principles and equality do you do you have a plan like what you could need us for like locking things down or like and how does that does that conflict with like the values of the organization I so I just get to like what we would like to do I guess yeah um yeah we're not sure yet we um it would like what I what I think we'd do is once we have a dispute a first one we'd kind of in a way pilot it out and then ask the community what we do with it um and uh so I'm looking forward to that but in terms of locking something um we I don't think we're gonna do that because it's um the kind of one of the the core principles here is that we're collectively creating this this very always permanently incomplete data set together for various purposes and one of them is not to be comprehensive um and um and yeah so and I think in terms of that kind of dispute and potential for um vandalism you know we when we were running the new play map we didn't have any instance that we know of vandalism and and that was up in public for about five years the only vandalism actually came from just software from spam spam bots but people weren't doing it which was interesting because I think um there's there's something about the structure of for the sometimes things have changed in the past since the election um but there's something about the generosity of commons based peer production that um I think really brings out the best or best intentions in people now to the election and other things we have seen definitely um out of nowhere or out of that um trolling you know commenting on our journal um that we had never seen before which is different and different interesting phenomena do you have a way for users to access um like the raw data that you're creating oh yeah that's a really important um and what we think is pretty promising or we're really excited about it so the entire code base is available on github and um for free anyone can pull it take it adapt it and we'd love collaborators to make it improve it or just do something completely different with it and then there's also an api everything is available all the data except for um people's emails and passwords um and which no one actually needs because you just need an email and password in order to be an editor and that's the only thing and so um yeah but everything else every piece of it is completely available also we're very interested in in the potential or possibility of of collaboration that way that would be that would be excellent because really what we're up to here is um in a way this is a first draft and it's a it's a call for collaboration and participation for the community out there to um to work with this or work with us or do something or make it better I guess a similar question with the previous this is all really interesting to me and I'm not that familiar with how around mostly being a dance person rather than a theater person um but as I as I look at the map and think about who's submitting things to this map and so forth um maybe you already kind of answered this in a way and saying that it's you know not kind of locked down but I'm curious about the monitoring or curation or whatever of those people that add um right now as we're moving through this I mean mama mea pops up but there's a lot of things that are uh things that I would I would typically choose to go out and see because they're in a certain range or a certain demographic of things that I that I agree with that is non-commercial um and so I'm just wondering I mean is there any intention or is it already in place that commercial theater is not supposed to be using this tool I also just think about the way in which uh theater events typically get advertised in the world through events pages and whatever and the nonprofit stuff the experimental stuff tends to get drowned out by the people that have money inevitably so I can just imagine how this starts in a really good place but eventually just becomes completely overwhelmed by the kind of theater making that has resources and money and whatever even though this doesn't cost anything just by sheer volume I could see it being overrun by things that make it then difficult to get to the things that you want to see you want to participate in want to know about and so on so forth which it now currently seems to be doing really well yeah it's so I mean it's been around now for 16 17 months so far and um I think that it's I think it's obvious it's maybe obvious to maybe commercial theater that this won't be an effective tool for marketing and um and so um and then also in terms of just kind of the principles of digital commons um more can actually be more um that the only space where there's maybe scarcity is just in this kind of one little tiny segment of the directory which is this visualization of what's happening today right that could be crowded out that's just you know that's a limited real estate there um but generally you know say there were thousands of uh commercial projects on here they would just appear in particular searches in the directory yeah so for the moment it's not I don't think it's an issue um but it's like that's a it brings up the good point about um you know scarcity and kind of the and the curation or putting barriers around around things and um and yeah so for the most part for you know an information project like this it's it's oftentimes not necessary to put um you know just to put restrictions so we haven't even defined theater here and there's nowhere where you find a definition of theater and you know theater means different things in different contexts um so yeah that's another another aspect to it two things um I'm curious about how you get this out to theater groups me people who are doing performances to other invitations in out because I don't know about theater and world map you know how do you disseminate um or invite people in and then also um is there commentary for after seeing the show I mean not so much reviews but just sort of an interactivity with people who looked it up on there went to it and have something to say about it yeah there's not that feature of commenting on on a show um however in terms of letting people know about it it's so we have this this core part of world theater map ambassadors I mean just quickly pull up where they're from so you see the list here and um they they have been doing a lot of um you know going to festivals and conferences and talking presenting the map um in their various communities um and then um then also we as a step at how around have gone to a few of festivals and conferences to present it um but that's yeah and we're not sure how else to get it out except just through this kind of thing this kind of forum and word of mouth it's only uh media performance so anything connected with multi modals and things is very it's largely european but it has a large um asian base as well but he just started with a email just inviting people who he met and then it you know to so anyway it's just and it's wonderful because it connects you to those people you know I get just get an email once a month and then I go to the website I think maybe just a quick question first of all this is fantastic just thank you and the community for creating it I do have a question about I'm sort of I'm on the tab what kind of organization is this and I'm wondering if there's a gap there or a purposeful sort of lack of space for uh colleges universities and other kinds of educational institutions and organizations yeah so um that's a tricky thing about coming up with these tags and the way that they were developed oh I I stopped it so that we see a lot of you see a lot of slashes here right um because what kind of the principle of tagging is that there's a certain kind of um point where tags no longer be useful are useful when you have a proliferation with them right so um we really worked hard to chop this town drastically with a huge amount of feedback from a bunch of test users and people from all over so and also this interest list is pretty it's small but at the same time big but I think we're at maybe in the sweet spot of not being too big not being too small that was a very funny answer to a very dumb question here's why I would like to thank you as well and how well in general as as well it's something I've used a lot as a playwright I used to the new play map and I I teach an MFA playwriting program so it's been really helpful finding connections and people and new plays um and this new project is really exciting the way it expands all of that my question um is I wonder if you have thought at all I know it's just beginning but I wonder if you've thought at all about connecting to other um theater maps like the theater times that Magda Romanska does it also at Emerson you know or these other instant these other websites that are trying to like create sort of a global uh knowledge base of what's happening and you know it would just be really I don't know how you would do it because I don't program but you know like if you could both look at like what are the plays in Poland on the new play map and then what are the articles about plays in Poland and the theater times or you know I wonder if there's any room for collaborations like that yeah um that's yeah that's a constant question we have um that you know ultimately this this is a call for collaboration and um figuring out how to technically connect with a bunch of different other data projects is very very difficult and um and requires a huge amount of resources of time money capacity personnel and um so that that's one you know obstacle there um the way that we've been the way that we conceived of this particular project is that that this is a place that points out to other sites to other websites organizations people shows and where it's not really housing the content but being a centralized place to point out to it so the ideal thing is that things are on Wikipedia or things are on organizational websites and they they get listed here and then people click off to them and then that's why in terms of this API it's a flow of information out from this map from this database as opposed to a flow of information in that makes it manageable and I just want to point out one thing about for example um um links to articles or giving more context to the data that's here that you may not have seen yet so the confrontations festival which used to be a fantastic festival in Lublin Poland um is so it's listed here this is just their organization profile or for this particular festival 2016 and then um you automatically get it's imperfect but you get um a little more context you get written work or or video content attached to this profile automatically and so there were live streams from this festival and then you could click over and actually see this a show from this particular festival okay then um then another thing I'll just point out another example of that because that that's interesting see um I'll go back to that original playwright we were talking about where it uh the system automatically pulls posts where she's mentioned or that she's authored on her own uh so you know pulling pulling other sources you know things off of her own um we thought about that it's that would be very complicated and then also the selection of what what we choose would be very difficult but um so my first is just very um pragmatic related to that can you is there a way to pull content from other places can you post um links to other if you had other websites or video um or like a website to a theater company or particular artist or does it have to be showing up on howl around in order to um connect it so that posts show up here no um no that's not possible however um for example let's see you could let's let's look for a show on an actual show profile you could link to various other organizations and websites um or more information just through a your uh and um yeah and I also just want to again like other people mentioned to say how great I think this is as a resource and I do plan to make use of it it's terrific um but my other question is actually a little more generally about that the howl around website um so uh for example the the latinx theater commons um is this so um is is this something that you want to continue to build at howl around more generally so so this is totally independent of the map is there an interest in creating more sort of specific community commons within the howl around um yeah so this um so in a way incubating other commons exactly other networks and communities definitely at this the latinx theater commons was really special in that the the people who came together were incredibly motivated incredibly passionate about it and um and very focused and so um and so we and we were lucky to be able to um you know to help um foster that and um you know support that work so yeah um in a way to kind of you know how how around is is modular in in that these modules come from the community and it's we we um kind of feel feel where the energy is coming from and we go towards it and then we help amplify it it's less of a direction of it's a less of a you know top-down thing where we're trying to plan out um what are we going to do next or what what are we interested in next this is just a comment um that I'd like to say out loud while I have everyone's here um my head is currently buzzing with lots of uses for the world theater map in the classroom and I know that we're going to be talking about pedagogy today and probably quite a bit next week and um this is an invitation I'd love to chat with anyone in this room about potential assignments or projects based on this because I have a lot of you know eight faked ideas right now yeah this is great thank you in terms of uh and I wanted to just quickly point that out but in terms of um scholars researchers educators using the map one of the things that I've heard is that you know for example there's there's someone who's um researching a francophone african theater and um and those information sources are very scattered and um and so she sees the map as a potential has a lot of potential we're actually like having you know making a list of theaters and artists who come from um the certain francophone countries um just to you know have that visualization um and and then to be able to share that with other people so that's one use so I'm just pulling up the low visualization of educative scholars so far only 224 so get to it yeah so yeah I have a more technical you might be getting into this next but in the searches um I see the default is it it's uh doing or's can you do ands to restrict categories or not to like if you're interested in everything but commercial theater could you say not commercial theater oh uh yeah um so let's see there was this was a big discussion with our web developers about you know a weeks long thing about what we could do and it's yeah it is and and that's because it's pulling from I actually don't understand how it works but it's um there's something big and complicated that makes it impossible to limit it through um to limit to do an or versus an in so let's see so this for example yeah we see the numbers increasing right right but then let's see if we add one country here to to this okay so this is so it is a combination so it used to be 777 and then when I added Argentina it limited right so it's so there's a kind of interplay between the fields and a horror within but conceivably let's say 10 years from now if this gets you know exponentially populated it might be very difficult if say you're searching you know two categories in you know North America yeah if you get 1500 returns without the capacity or the mechanism to limit those in some kind of meaningful way yeah yeah that's it that's an issue um because it's right now you're building it as a as an accumulative exercise yes and that's yeah that brings up a thing interesting thing is that um we are we're building a new how around com website which we're hoping to launch in september or october and um so we have seven years of incredible content from the community writing and video thousands and thousands of pieces um and we have lost track of the incredible richness of this database of this archive and so what we're what we're going to do a feature is create creating a way for people to make lists or a mixtape of uh i got that term from from you last night um of um of posts that people could use in various ways you know their favorite or they're teaching a a course and it's a listing of articles and video content and so and also you know that falls really squarely in the idea of a community managed resource where all of us um become docents you know of of this archive because it's it's really impossible for us to keep up with what's going on and so yeah in this case of the ever expanding database um that's maybe one possible solution to kind of help that issue of things being too big to fight do you have a like a librarian or someone no in for no no we don't that would be great to have that would be that is unfortunate right there's a whole field of study that is about managing how information gets stored and categorized and how metadata gets developed and how to make that comprehensive and sustainable yeah we we would need help yeah and yeah i really recommend that especially seeing this in world theater that i i i think how i'm going to benefit enormously from having uh such a great elaborate information science working with us in tla for example this one um one possible committee by five people excited about working off with them with us first of all right it's got a great idea about getting in touch with tla thanks because this is what we love and what we do really well um so different ways of thinking about organizing that database and allowing various kinds of mining of the database so a mixed tapestry is a really cool metaphor for that so is a syllabus right and if you're working with educators who think well how might you one what might be some educational paths through this database or into it or around this database um or even providing a more abstract concept of what are some ways of thinking about the database and how you might shape it into some kind of a lesson plan or a semester or or even a multi semester so the 50 thousand viewers, participants, people that come in to the world theater map or to how around in general all the platforms together so tv journal map do you have any further information about like the world theater map like how people are behaving like what who they are coming and what they're interested in so they're um they're about i think now um 1800 people who have created accounts and half of them are editing actively editing um and that means um you know creating content or editing already existing content and then um it's about 70 percent of those users are outside of the us of those editors are outside of the us um which is which i think is pretty good because the other project was all us and so we weren't expecting that much um adoption from outside the us um and since you know we have a legacy of being focused on the us um and you know a community that kind of came from that um it's been kind of a slow slow development to have more non-us people and then other things yeah so that's that's what we know about the map users is is that it's about half half our editing um further moving on my criteria about uh the police room or riffing on what i do for stories through uh would be maybe even in finding a place where people could post syllabi and um the an assignment sheets fully so that i mean they could come up on profiles for plays and things like that so oh i want to sign this play what syllabi also includes play what people have assigned to have created assignments for that for not what people but what assignments have been created and like kind of linking through that way too which is kind of more stuff but um over the last several days we've heard from a a lot of people about you know how challenging it is to develop their dh projects and put them online and how long their web platforms last online which is not very long um so i'm just curious in this instance uh i mean you talked a little bit earlier um about where you're hosted and so forth but i'm just trying to get an idea of how many people like what is the labor behind the scenes that manages this thing in its entirety and i would imagine you hope to keep this going in perpetuity but what are the you don't have to get into specifics i'm just trying to get a journal sense since we're all interested in our own projects and this seems to be a project that is successful and up and running and it hasn't collapsed in on itself yet um and let's hope that it never does who will what does it take to make this happen and keep it running so we we've been working with a web development company it's actually just two people um called maswood um and they're based in manila they used to be part of um a worker cooperative uh web development company called quilted and based in oakland and um and it was one of the pioneering worker cooperative um and then that worker cooperative actually disbanded um and then um one of the developers then just started this other firm so we've been working with them for many many years now 10 10 years i think um on mapping as well as our previous website um so it's very close relationship that way um and then so not not a huge team at all it's really just one two people and then in terms of um us as administrators it's um there's not apart from the kind of um just communicating about it and managing it in terms of that um it's just you know our very small team of four people at how around um but among various other projects that we're doing and um yeah so it's not it doesn't require that many resources however you know these technology projects are very tricky in terms of complexity when um or just you know bugs there's non-stop bugs that that happen or that just suddenly um so um and then also we're very willing to end this project after maybe a year or so if if there's if there's not real fire underneath it and uh and i think we're not not afraid to actually just completely end it um and which i think is important because there's a lot of learning from from this um that we could you know that other people can take and a lot of learning for ourselves for future projects so we don't know it's still i mean we we see it up here still help data all these questions which they're all fabulous by the way um coming from our history i'm i'm definitely gonna figure out a way to use this in my class because this is phenomenal my question is and this is of course one of those logistical issues that i'm sure probably gets thrown out all the time is the the language components great i mean you have these three major languages but i'm like my favorite especially these in middle eastern art and i'm just thinking about the language components there and also the access to information um and if there's any way like any of those of like iraq i'm looking at um sarama barks website the theater that the one iranian theater that's represented just wondering if there's any ways to disseminate or maybe provide language access if the if there's no if say theater professors or rights etc can't act can't speak these languages like spanish or english yeah um this is kind of comes up against our this is kind of a question of our capacity yeah as well um it's it was pretty pretty intense to get it into these three languages and um you know the dream the original hope was that it would be um definitely in arabic as another language um and then yeah and then i mean hopefully that is absolutely multilingual but then you know us kind of our first time doing a website in multiple languages we learned that it's a lot more difficult than what we got um and so and just to briefly quickly talk about how this language stuff works is so for example if you are on the french website and you added information you do it you can do it completely in french and then the system itself will automatically the software will do a software translation it takes a google um translate api and translates that french into english and then there'll be a little notice whenever you visit that profile saying this and when you see it in english it says this was translated from french by not a human and please improve it um and uh which is i think a great kind of like in between solution that we have for people just being able to see this in french or just being able to do it in spanish and then to have their stuff actually accessible in english though they may not speak english um so that's yeah that's one thing but like the next language definitely if we have the resources would be um arabic um then apart from that um the only thing at the moment is just you know using a google translate little browser extension to have bizarre translations super insane you're like in japanese or arabic turn out so i feel like what you're doing is is up and i know that you're a theater person because i got a huge chance to speak with you um during dinner i feel like what you're doing is really maybe there's other people doing it but it still strikes me as a rather unique space that you're carving out for yourself so i had a question for you it's maybe like a career trajectory question how do you how did you conceive of yourself as a cultural strategist how does it fit in with sort of how did you come to this place how what satisfies you personally in this work and yeah where you have another life as a career like as a theater artist how did you how do you make space or did you have to give it up or what do you think is going to happen in the future and oh yeah for yourself yeah so um so that title okay well that's um that's about you know the kind of a role that i think also that we take all all of us hara is um one of the primary goals is to change culture and um and that we can actively do and so that's where that kind of title comes from and um and then especially through the systems and models in which we operate and so that's why we're we're big fans and big believers in in common space peer production um and that in a way as a as a social model that um that really enables and brings out certain behaviors in all of us and because if we continue to live in in just kind of our dominant um society's model we adopt those behaviors of scarcity of competition and um yeah and kind of one of the big lessons that i've had personally over the years with hara is that um that your personal intentions um often get can get and will get um trumped or your personal values can get trumped by the systems and organizations and institutions in which you live and um you know the the systems i live in sometimes can bring out the worst in and um and so whereas alternative models like common space models um can do other good that can bring out good behaviors and so i'm kind of you know i feel not like we don't we're not in complete control of ourselves does anyone want to is there a show in there that you want to add from and see it and that's another question so some of them don't have links so that's maybe you can hear at it and just add in oh absolutely yeah you see something that's in complete um because i'll wish i just i just looked this up like a minute ago it's one of the theaters and back dad's coming back after not being there for obvious reasons um so it's one of those things like can you add like links to news articles say the theater was destroyed it is in the process of being rebuilt yeah that might be something that's interesting for people to know like in the next few years they'll start running shows again and start using it as an open venue yeah so what you could do like let's just pretend this was the theater company um those links that could definitely go in um in this about box right and then um and then of course the organizational website or maybe even better just link to the organizational website and then that would be nicely organized um and then you see all these other links that you can add in here as well as adding um links that are not one of these other ones like social media are you um so you're like collecting data on who's using it where they're from um do you have the ability to to collect information of who's visiting it even if they haven't signed up with an account yeah so we're using um google analytics i may not be able to get in right now but um yeah so we're able to see the non non logged in people and where where they're coming from and it's about the same as um as the people who are who have accounts what do you mean the same so like same distribution where it's um likes um the percentage of people outside the us and let me see if I can maybe pull up um quickly like the country spread of the users well I know that the the majority the country that has the most users is um I believe it's Argentina users who have accounts and um and the top in the top five or so countries are um spanish-speaking latin american countries so anything else any other ideas for what to put up there that came up this is just bugging me so yeah it ends in 2050 I don't know if that's because that to be absurd someone with a sense of humor actually this is uh this is that original production that's not that theater yeah it's running continuously yeah since 57 and I think they have they have no plans of stopping so I think it's completely legitimate so they have to put an ending uh yes in order to be an event that's happening you have to have an ending yeah so they just put it away yeah let's see if they build up further in terms of dates and and stuff so you know this is very much also in you know historical archive it could you could put in events and shows that happened you know 20 years ago and um so you'll find if you dig in there especially I think some us-based plays you'll see some old stuff so right at some point so there's you know there's this way you land on this page and it's and it's upcoming events right there's kind of this this very kind of presentist or future projecting feeling to it but at some point it becomes more and more of a historical archive right as as you know as many shows kind of probably more I would imagine at some point are in this kind of historical archive as are coming up right and then at some point you also start to have this maybe archive of makers right we don't live forever and so it starts to become this so I guess I'm I'm thinking about I'm not really sure what the question is exactly but there's something interesting here about how this right now feels very present but at some point becomes very much about the past as much as anything else and so I'm just curious about does that at some point shift how what this is and and how you deal with it yeah yeah you know in terms of this this visualization like part of this is also you see I think we would all feel bad if this was the homepage like because in but the trade-off is that if we don't have a globe that we would just be looking at this because the core of it is just this search is the directory right and that would be pretty boring but the problem is with us having a globe in a way we're communicating the wrong thing that it's about events happening today where it's really not that that's just one slipper of the directory so that's a that's a bit of a struggle in terms of just the visual design because like say this was the work theater directory I would die yeah so that's that's something has anyone gone into pull out some some of the data and make other kinds of visualizations no not yet I'm just thinking about like other ways of yeah I'm not that you should read your homepage but I'm just thinking of like you know making a map that like was also like correlated to a timeline right and so if you sort of you know I don't know starting in 1957 let's say and as you sort of slide forward you could watch things pop up in different spaces absolutely yeah and so we're we're really hoping someone would do that and what they would do is just pull on this API and then they could just for example query one particular set of data to then make a completely new website web app or mobile mobile app and you know what what what would be an example is like I don't know someone just interested in um like like you said a chronology of events or climate change theater artists in the UK and make something around that or alerts you know wouldn't be too much just like you do it so like basically it would you know pin you every morning with like everything that's happening today in the world or in a particular you could craft it to a particular city um something that has come up a lot in this already is this idea of um that any particular interface and I think this is and if I can make my own totally non quantitative um survey of the things we've discussed particularly applications that have any kind of animation or using something to animate themselves much like the globe which is which is very cool and very smooth and sleek um tend to break down very fast in the online realm and so um I have at some point all that might be left might be the search and then not the search anymore and so um have you thought about sort of a a legacy for this I know you said if it's not succeeding you want to take it down but um assuming it is successful and which I think it will be and continues to grow then have you thought about what happens as the technology changes and if no one is concerned enough about the project in 10 years to maintain the the interface the actual yeah so um I mean what you see here is actually just kind of one way I mean the data is structured in a very specific way and but then um and then that data can live on um in this repository and can be revived you know if if this site doesn't exist anymore um it can be revived and then reworked um yeah so there's that and then also we could actually kind of totally refactor um or create a new expression for this website based on the underlying data because this is actually just one one subjective expression of the database though at the same time the database we we designed it in a specific way and maybe it's the wrong way um and maybe we'll discover that um so I think we're near our time yeah so that was fantastic thank you so much all right so we'll meet back here at 10.40


If we go to GitLab to the Study Emnist Digits repository to the project, we can see the files involved and there's a readme right up front that gives a good overview of what's going on. The instructions for getting this up and going are in the readme here in the project. The very first thing that you'll need to do is to get Cottonwood installed and running on your machine. If you go to your command line, clone the repository, pip install it, and then make sure to check out the V28 branch. That is the version that was used to develop and run this case study. Cottonwood is not backward compatible, meaning that with any new version there might be changes that break how it used to work. So we have to make sure each time we use it to get the right version. So this is a version 28 project. And if you don't have them already, installing Cottonwood will also install NumPy, Numba, and Matplotlib, some other packages that you'll need to be able to run this. Another thing it will install is the Emnist Digits dataset. So luckily there's a convenient Python library for downloading these digits pre-processed. And then within Cottonwood, we've wrapped them inside of a convenient data layer so we can pull them out one at a time, one from the training set during training and one at a time from the test set during testing. There are 50,000 training examples and 10,000 test examples. So 5,000 each from each of the 10 digits for training and 1,000 each for testing. Then to be able to train a new model from scratch, you can run the train script. When you're done, you can run the test script to evaluate it. Also, it's set up so that during training it saves out the model in its current state periodically. So you can actually in another window run the test script at any time during training to get a snapshot of how the model is performing on the test data at that point.