The floor may not be in the robot. In the summer before I went off to graduate school, I was trying to stack as much money as I possibly could. This included working full-time, taking up odd jobs on Craigslist like helping people move and tutoring high school students. One day while browsing Craigslist, I came across an ad for work as a junior animator slash video editor. It paid $20 an hour, so I instantly applied. I had passing familiarity with animation programs, because my friend and I had spent years trying to design a simple video game. And my video editing was quite good, because I had run a popular YouTube channel when I was younger. I got the job. It was weirder than I expected. The company was in an on-descript business complex in Irvine, and every employee had an electronic badge that unlocked doors. Certain levels of employees could unlock certain doors. Being at the bottom tier, I could only unlock the entrance, the door to the room I worked in, and the conference room where we'd have weekly meetings. I never saw any other rooms in the building, and never spoke with anyone who worked in them. There were seven animators including me. We sat in a row of cubicles in our own small room. Our job was to edit cartoon knockoffs of popular children's characters, typically Spider-Man, Elsa, SpongeBob, My Little Pony, etc. We worked on one or two videos per week, and basically we just created cartoon objects and settings. The work was surprisingly simple. There was very little real animation required. The job paid so much that I hardly paid attention to how strange it was. The company divided our labor in such a way that none of us animators ever saw a video in its entirety. We each worked on a few seconds of it, and often, the project would be taken away from us and transferred to another department before we were finished. The rules were odd. The animators and I were not allowed to speak to each other under any circumstance. We were not permitted to exchange names or introduce ourselves. Speaking, or looking at another person's computer, was a terminatable offense. No two people were allowed in the break room at the same time, and no cell phones were permitted inside the building. Ever. The room was strange too. It was blue. Everything was blue. The walls, the chairs, the keyboards, the door. A blue air freshener was taped to the wall of each workstation, but it didn't smell like anything. There was one object that was red, a telephone. It rang every so often, but we were not allowed to answer it. I was instructed to stand up from my chair and stretch each time it rang, but over time, I noticed that the other employees had been instructed to do other things. One of them took deep, slow breaths. One of them put his head down on his desk. Two of them left the room and returned. One swirled around in his chair. One coughed. I noticed a few other weird things about the company during my short time there. It wasn't unusual to see employees crying as they made their way through the halls. Any time I spotted one of them crying, they always tried to hide it. Some of them couldn't. On a few occasions I saw a child wandering through the halls looking for someone, or maybe for a bathroom. When I brought this up to my supervisor, he told me, It's bring your kid to work day for the department upstairs. He told me that three times in two months. Things started to get really uncomfortable around the two month mark. One day, when I checked my company email account for the weekly breathing workload assignment, there was an email titled, Lullaby. Inside was a link to a short, low resolution video of a young girl asleep in a bed. She gabbled in what I believe was Russian or Ukrainian, and occasionally fidgeted or brought her hands up defensively to protect her face. It was clear that she was having a nightmare. Behind her, on the bed post, was a blue air freshener, much like the one next to me in my cubicle. Whimsical vote of ill music played in the background. I examined the recipients and sender of the email, and found that it had been sent from inside the company to several employees on a list. I forwarded the email to my boss, and asked him what the deal was, and he quickly responded that it was a joke from our partners overseas, and that I had been mistakenly added to the recipient list. He told me to ignore it and keep up the excellent work, and that my review would be coming up, with the possibility of a raise. More than $20 an hour. I guess my memory is for sale, because I quickly forgot about the video. Only a few days later, when I returned to the office after a holiday weekend, there was another email waiting for me, titled, Be Brave Spidey. I was reluctant to open it, and now I wish I hadn't. Inside was a link to a Russian language website. When I clicked it, I saw a video of a real kid, probably four or five years old, dressed as Spider-Man. The boy sat in what looked like a child's bedroom. His mask was pulled down, and his costume sleeve was pulled up. The boy screamed and cried, as an adult man wearing a Hulk costume gave him three different injections with a long needle. Offscreen, another person hurled stuffed animals at the kid, hitting him in the head with them, and even once hitting the needle as it stuck into his arm, causing the kid to wail even louder. By the end of the short clip, the boy was shaking and merely catatonic. The Hulk man laughed and danced around him almost ritually. Cheerful kid's music played the entire time. As far as I could tell, the video was not acted. What I saw was a real medical procedure, and real terror. Horrified, I emailed my boss, demanding an explanation. I received none after about an hour. Normally he replies within minutes or even seconds, so I left my cubicle and stormed down the hall to knock on his office door.  I was so angry and freaked out that I didn't care if I interrupted him. I badged the electronic lock and cracked the door open. The conference room was dark, but I could see about 15 men sitting inside at the far end of the wall. Most of them were dressed nicer than me, so I knew that they were senior employees who worked upstairs. A video played on a large screen at the other end of the room, and even though I couldn't see it from my angle, I recognized the sounds. They were watching the same horrific video I'd seen an hour before.   One of the men in the room occasionally replied in Russian. I left work early that day, too freaked out to return to my station. By the time I got home I had a missed call from my boss, and a voicemail summarily terminating me, stating that the project was complete, and that unfortunately our entire team was no longer needed.  I didn't plan on going back anyway. I spent the rest of the summer doing odd jobs, and trying to forget that company.  A few weeks later, I visited my brother and his wife at their home in Southern California. My niece Katie was five years old at the time, and could already operate electronics better than I can. She's got an iPad, and spent a bunch of time showing me photos she'd taken of birds and insects and people. She's also got Netflix and YouTube, and watches those regularly. One night during my visit, my brother and I were on the couch watching one of the Hobbit movies. Katie was lying prone on the floor nearby, watching a cartoon on her iPad. When I leaned over and asked what she was watching, I immediately recognized the cheaply animated characters. It was a video I myself had edited. I recognized the ringing red phone, which I had designed after the phone in our office. I recognized the glass bottle the characters drank from. And I recognized the way the joits and jaws moved, all things I had worked on at one point during my brief stint at that company. But I had never seen a full video. This one was about five minutes long. It featured two cartoon kids dressed up in Elsa and Spider-Man costumes, stealing their father's beer and getting drunk. Then, one of the kids trips and falls, smashing his face into a desk and splitting his skull open. Blood sprays everywhere.  Another video played, then another, and another, all products of my company, some of which I'd worked on. Every video featured recognizable children's characters from Disney and Marvel and other big brands, but something weird, or violent, or sexual, to place in them. I pulled Katie away from the iPad, and put Finding Nemo on the TV for all of us to watch. Before I returned home, I warned my brother about what I had seen, and advised him to keep her off YouTube for a bit. It wasn't until I returned home and started digging around on YouTube, that the true scope of these fucked up videos came to light. I found several channels with child-oriented names like, Silly Hero Phone, all of which produce videos exactly like the ones I'd worked on. They all specifically target children using familiar characters, and they all link to more legitimate cartoons via the recommended videos algorithm. The more I watched, the deeper the rabbit's skull seemed to go. These videos are constantly removed, renamed, and re-uploaded, over and over and over. After watching about a hundred of these videos, I found that they all shared certain similarities, and can be divided into recurring themes. By intergalactic no-sleep law, I'm not allowed to link the videos or mention the YouTube channel names, but if you want to find these videos for yourself, simply type Elsegate into YouTube, and you will see for yourself. Warning, the cartoon videos are disturbing, and the live-action ones are outright depraved. I consider some of them to be actual child abuse. The themes I've identified are as follows. One, some of the videos show characters stealing alcohol and hurting each other. One shows child versions of Mickey Mouse getting drunk on their dad's beer, and then one of them splits his head open. This same video has been reskinned over and over with Elsa and Spider-Man, Paw Patrol, and Minions. Getting drunk and hurting yourself is ubiquitous in these videos. Also, burning yourself on a stove or getting sucked into an escalator are common. Accidental injury is the driving plot device.  It works with Spider-Man, Hulk, etc. Two, the phobia of spiders and insects is another common theme. I found a video showing Minions covering themselves in disgusting looking bugs. The end of the video depicts a man drinking a bottle of urine, which I'll discuss below. Another video shows Elsa, Spider-Man, and the Hulk all being swarmed by insects. Sometimes they require hospitalization and surgery because of the bugs. The characters always react with horror to bugs, and the bugs always injure them. Search terms include Mickey insects or Elsa insects gross. Three, drinking from toilets, eating poop, drinking urine, and smearing feces on people's faces is another theme, commonly portrayed in these videos. Many of them are live action, with real actors dressed in costumes that target the attention of children. In one video, Spider-Man and Elsa drink from toilets, and also find insects in one. In another, Venom buries Elsa alive and shits on her head. Another shows the Joker feeding excrement to Elsa and Spider-Man. Any of the character names with the word poop or toilet will return these videos. For extreme medical violence and the phobia of sharp objects is yet another theme you'll find in these videos. Children cutting each other's fingers off with razors, doctors forcing needles into children's arms, eyes, and rectums, and gory surgery are all present. In one, Hulk crushes Elsa's bones and she requires injections. In another, Hulk gets needles shoved into his face, and has his eyes pulled out with tweezers. In that same video, Spider-Man throws sand in a child's eye, and the child requires injections in said eye. Spider-Man later gets sick from eating bad food, and requires needles to be shoved into his body in multiple places. Search terms include hulk eye injection, Elsa's surgery, or Spider-Man slash Elsa's sick. Five, pregnancy is frequently depicted as a curable illness. Unsurprisingly, the cure is an abortion injected directly into the woman's stomach. The worst video I found depicts tummy aches, illness, and pregnancy in a very blended way, all of which require the use of needles to cure.   Search terms include Elsa pregnant surgery and Elsa pregnant injection. Really any of these cartoon names with pregnant works. Six, the helplessness of children to protect themselves from adults is a popular theme, especially in the live-action videos.  Search terms include bad hulk superhero battle. It gets worse and worse the more you follow the video trail. There are also tons of videos of toddler-aged girls being kidnapped and tied down by adults men, depicted in a playful manner. Many of the men are wearing frightening Halloween masks. The children are often crying, and are not having fun at all. Some appear in pain. Many of these have been reported or taken down by YouTube, that now the channel has converted all video titles to Russian, and they cannot be searched in English. This is the sickest channel I found, and the point where I completely stopped watching. Seven, sexualization of children and depiction of pregnant children as a good thing. Many of the Elsa gate videos depict children in an arguably sexual light. The most popular channel with this kind of content stars to young Asian girls, and has 3 million subscribers. Many of the videos depict butt-shaking, playing doctor, and fake vomiting. Others show girls and even boys celebrating their own pregnancies. I won't even provide search terms for these. Just don't. It took me a while, and a bit of research, to pick up on the purpose of these videos.  But when I started to see how they all mimic each other and build on each other, I realized that they must have a grand purpose. The fact that there are thousands of these videos, but they all cover the same seven topics, screams conditioning. The creators of these videos are banking on the probability that if kids watch enough of the videos, they'll be saturated with 2 or 3 ideas.  The fact that violence and sex are such recurrent themes, tells me that the creators want to normalize them.  Maybe even curious about them. The comments in the videos reveal that a lot of the viewers are adults, and fetishists, perverts. They really really enjoy the videos of kids being kidnapped and tied up. They beg for more, and offer to support via crowdfunding. In short, these videos are designed to groom children, and to satisfy perverts. After digesting all this information, I contacted my brother, who had some terrifying news for me. Apparently, he and his wife had received several phone calls from people asking for me. When my brother asked who they were, they always hung up. He said, They always have an accent. Worse, a man actually tried to pick Katie up from kindergarten by claiming he was me. He gave the office my full name and told them he was her uncle, here to pick Katie up for a doctor's appointment. When the receptionist said she was going to call Katie's parents for verification, the man took off running. He didn't even get into a car. He ran out of the parking lot. I began receiving text messages from very long numbers. The texts always contained links to YouTube videos. I always deleted them and blocked the numbers. By the time I was packing up and preparing to move, the texts had stopped, but my brother told me that Katie came home with an air freshener in her coat, and couldn't remember how it had gotten there. He sent me a photo of it, and I recognized it as the same type from my office. He said it had no odor. Things settled down for a while. My first year of grad school blindsided me, and I forgot all about the strange incidents. But over the summer between my first and second year, something else happened that reignited my old fears. I worked part-time at the university library. I always took the night shift because I could relax and work on grant applications, and didn't have to deal with many students. But one night, an older man checked out a stack of medical books at my counter. He looked and smelled like a tenured professor, so I thought nothing of it when he struck up a conversation and asked me if I'd had my flu shot yet. I told him I had, and he smiled and turned to leave. But then at the door, he turned back to me and called out. And has Katie had all of her vaccinations? By the time I recovered from the shock of his question, the man had disappeared into the dark outside. He left the books by the door.


Alright. Welcome everyone. Thanks for coming to my session. My name is Ildi Kovancza. I just recently joined the OpenStack Foundation. My role is Ecosystem Technical Lead. And as part of this job role, one of my responsibilities is to try to keep the ecosystem healthy because if you just look at these numbers, our community is quite large. So even regarding people, we have many countries involved, so we have many cultures. It is a great mixture of people, mindsets and ways of thinking. We have many many companies around and well the codebase is just growing and growing and growing. So how to deal with all these numbers because these are nice and large and we like to keep them growing as much as we can or keeping them stable or keep them growing. But how to operate in an environment which is kind of this dynamic at this size. So as OpenStack itself is kind of a software package, an open source cloud platform, we can look at it from multiple perspectives. One of these is the business view. So if we have in the room people who are taking care of a company's business, then you might be interested in the user surveys that we are continuously running because from a business perspective it is really important to have the latest data to know how the market looks like, how the adoption of our platform is going. It is really important for us as the foundation and as the community to know how successful this software package that we are developing. And on the other hand for those companies who are using it and building their business on top of it, it is also a very important information for them. So as you can see on the slide we are reaching out to more than 2,500 users in the timeframe of one year. But I still need to remind you that this is not a comprehensive market study. So the user surveys are similarly kind of going on community with a community mindset. So they are with voluntary basis. Therefore when you find information regarding which projects are mostly used and which new and upcoming projects people are interested at, those are good numbers. But you need to know that those numbers should not drive your business and or technical decisions. And on the other hand OpenStack itself is not just the software package but all those more than 60,000 people who are working on it. So it is very important that we are all aware of that how the community operates and how we as individuals within the community are operating. So when you are joining us and start to participate then you start to recognize that it is a little bit chaotic and also there are so many moving parts that those ways how you might use to track that how efficient you are or you were with your work and those deadlines that you set up as targets and how you collaborated with your team within your own company. Those things are just almost the same but not anymore because of all these moving parts and we are having people coming in and leaving. So it's really not a stable environment therefore we cannot handle it from metrics perspective either. And one very important thing right at the beginning all the numbers that you can find as publicly available data these numbers are not for just increasing them and kind of playing metrics game with them. So it's really more for your information and for you to be able to improve and also to have the community to improve by being a good community citizen. So what you can find and what we provide is really just a basic set of metrics. So we as the foundation or the community itself we are not really combining any of any of the data that is available. We are we are trying to give you a package from which you can build your own system by which you can track yourself or how the community is doing. And on the other hand I would like to also draw attention to the fact that no matter how simple the metrics are and how simple and straightforward the definition of a metric is it still depends on that who looks at that number and who tries to interpret it and what they are trying to use it for. Because it will it will not mean the same thing when it is used in a different context. And also what we what we feel important is that the data that we have is that can be collected automatically. So what you see online it is usually really the latest and greatest values. And as I see still not a that good mixture of women and men in this room. It is it is quite new activity. This is the first diagram that personally myself I saw this morning the women of OpenStack group in a in a collaboration with They started a diversity program with with collecting metrics and kind of trying to check the tendency in this area and see how we are performing to to deal with different types of diversity like gender diversity. So as you can see on the figure we are we are doing better. So I was hoping to see more women in the room. But if if you you or your company are interested in this number or participating in the activity please reach out either to me or to the women of OpenStack group or just or just follow these these activities because the data is really interesting. And it still shows that we are on on on track and on on the journey. But we still have kind of homework to do and we have room to improve further. So what tools we are we are having that offers you data. We have basically two publicly available. One is tech analytics. I would guess that many of you at least heard about it already or maybe using it already. And the other one is a bit of job based activity dashboard. You on both of these you can find different metrics. I try to just rather paste screenshots here to to give you an idea that basically what we have. The community view is really more about the numbers of of collaboration and contribution. So in this sense we have for example comets we have we have emails we have lines of code which personally I totally hate. We have reviews and in the other tool you can see that we we try to follow that how the tracking systems are used and operating and how the IRS IRC which is the tax based communication channel for us how busy that is and how much people are active and responsive on that media. So when you when you look into into these these metrics I I thought to try to talk a little bit more about those which are more commonly used and which are sometimes also misused. Personally I have experienced from the past when when when people kind of thought that if if a number is is growing rapidly that's just good and that will keep their managers happy or the manager thought that that it will just keep him or her happy if if the team operates like this but it's not really only not not really true. I think in most cases it's it's not true especially if you're really just looking to one simple number. For example if you just type into the browser that Stakelytics.com then the first metric that that you will see is the code reviews because all the all the code changes are reviewed by by the project teams and it is kind of a very important number from the from the sense of to see that that how much the code is double triple checked before before it gets part of of the software package that we are offering and it is also something that for example developers are rather do always tomorrow or maybe next week or start next month or if their managers as done for so here I would like to draw attention to this one because it is really important to do valuable reviews and and don't be afraid of numbers like like growing the number of minus ones which means that I think you should work more on this code because those are the most valuable reviews in most cases and just kind of blindly giving plus ones which means that you you agree with you agree with the code change you don't have the power to to merge it but but you think it's good without any comments it's just not the beneficial for the project teams for the community for the the quality of of the code base so for example when you as a developer or if you're a manager of whom the team starts to to work upstream and work work in open source it is really important to to avoid let's say practices like what if we plus one 10 or 20 patches every single day so I I saw earlier like I don't know having 400 reviews in in 10 days or something like that so you can see that that something is really wrong there and the unfortunate thing with these cases is that in many times it's not really about just to to to grow the number but they are kind of doing it by mistake and they they don't really think about that that what what this really means and on the other hand when when you look at it from community perspective it really has a kind of bad interpretation because you can be considered as a as a rather bad than any kind of a good community citizen and this is something that you would really really like to avoid so cold reviews are the things that that you need to do in a clever basis so for example when you as a developer look at your numbers because sometimes you're just checking that that how you're operating and progressing within the community if you see that those numbers are too high then you might think about that that that the review and the other activity balances maybe not okay especially if you grew the number too quickly and you and you gave too many minus ones that that usually means that you need to check back on on all those things later so you you can easily overload yourself so these numbers can can be used in a way of finding the balance in your in your own work and find find a good way to to do the everyday tasks when you're when you're trying to be part of a developer team or or if you're already part of it. We have lovely numbers like commits and lines of code and these are kind of favorite ones and I think it's kind of usually handled as a competition at least people many times look at it like who has the most commits who has the most number of lines changed in the code to to clarify the two words if any of you wouldn't know commits means a chunk of code that or documentation changed that that got merged and got part of the software package that we are offering lines of code means the number of lines of code or documentation that got changed and especially when we are trying to rationalizing things this can be a high number either or either the changes consist of many comets and each commit has a smaller lines of code or or maybe a few larger changes regarding these things what you would usually use these things for for example when when you're trying to make sure that that your strategy is followed by by the activities so when for example you're working for a company and and there's a product that that you're developing then you know that which projects you need and most probably those are the projects where you would like to see more and more comets and or lines of code in these metrics because if you just focus on being part of the community by kind of growing the numbers overall without checking where where you're active and where you're not active this usually means that you get distracted and and the efforts that you're you're putting in the community contribution and collaboration might be kind of a wasted effort because one important thing about open source and how we are working together is that it is really important that that you you have influence within that team and and project within that team and project within that team and you have influence within that team and and project where where you have where you have business or where you have interest either either personally because there's really no other way to kind of influence the direction where where the projects and the codes life is going so the only ways to to be active at those places which is really important for you and your business as opposed to just randomly trying to generate large number of changes here and there it is also also personal experience that that sometimes people just just misinterpret this this whole thing and trying to use it as kind of more a marketing opportunity rather than something really really strategical work we have one interesting metric I I think I figured it out what it really means a year ago and I'm I'm in in OpenStack for more than three years now the person day effort number actually means it counts the days when you had any sort of off stream activities so for example you you sent out a mail you uploaded a new part set for review or you you you made a code review this can be really useful for you just to check that how much time and how much effort you put into the upstream and open source work I don't really think again that just using this one number it will be that much meaning for you but but using it as kind of part of a group of numbers it can be a really good indicator just to compare the effort that you put into the job and and the outcome that you have by the end of the day beyond this we we are also tracking other things like like blueprints and bug reports these are these are more the values of of how busy a project is and and how the maintenance work of the project is is going usually I would say you as an individual I don't really think that you will use it much but from from the project perspective where where you're working in this is kind of again an interesting and somewhat important set of numbers maybe you if you if you try to use it just to track your own work it can be an interesting number that was the ratio between your submitted blueprints and the implemented blueprints because if the numbers are let's say you're submitting many many blueprints and none of them gets to the implemented stage then this is this is a feedback that you you you don't do something good in a good way and this this can be for for multiple reasons but most probably the the root cause of the reasons are the communication because one of one of the things that that are really really important in a community like OpenStack is communicating that what you would like to do how you would like to solve your problems and and if someone has a comment on on your idea or on your code change then how you agree on the best solution with which you can move forward so if you're not really successful with with bringing the new feature ideas through you might revisit and rethink the way how you're describing your ideas how you're trying to discuss your ideas with with other people within each project and how you're handling the whole process from introducing the idea until it gets implemented and fully functional. Regarding the tools one of the I think most useful thing is that in Stack Analytics you you can find a people view or well person view so you can you can find the activity reports for each and every individual who contributes to the code base of OpenStack and many times the question comes up that okay I got stuck with something and I don't really know the solution I try to reach out to people on IRC but I wasn't really successful I also have problems with with the time zones because the people are from all around the globe so it's really hard to to ensure that that when you're awake and active you will find many people around but actually in this activity view you can find a diagram with these smaller and larger dots which is the UTC hour of when the person is active so this is mine based on this as a homework you can you can figure out that what time zone I might be in although maybe it's a little bit misleading because I mostly stay up late at night as opposed to get up early in the morning so maybe you will not be able to locate the the exact time zone but but you will be really close and also if if you're interested in in in the project and the area that that I'm working on and you find overlapping with your active hours and mine you can even reach out to me on IRC directly because you will have a better chance to to get response so for example if you're if you're struggling with this you can you can use this graph to to find this data which is really really useful because many people are struggling with how to do the communication effectively and how to deal with the time zone difference which is really really inconvenient beyond this you can also find all the numbers that a person produced and also some information like launch but ID and github and get ID and also that which projects the person participates as a core reviewer or core developer these are also important information if you would like to reach out to the person and also to identify that that on what the person is working on so you you find a common ground for your discussions. The other tool that we have and I mentioned is is the one which is based on Biturgya which is the activity.oblastag.org. I think it is kind of more of a lonely site because most of the people at least who I know are using Stakelytics. This one is more more about an overview like thing as opposed to go into details like providing for project metrics so you can see the overall activities regarding commits for example you can also find the data which is for companies but you will not find that which OpenStack project produced the different chunks of the data and you can also find information here for example about IRC which we don't have on Stakelytics or at least not yet and it can be also kind of an interesting and the important metric in the sense of you will see that who are the people who are kind of our people persons who will more likely answer you or will be available on IRC. I would not recommend still to start the person on the very top of the list but at least you have an idea that if you need information then who might be your friend and who might help you with the process. Okay so as I mentioned in the beginning it's really important that what mindset you're looking the numbers at because it will have a totally different meaning for you. For example when you're more in the management layer and you're more interested in the business view and product strategy view of the development process then you will be interested in both seeing some numbers growing as a tendency or being stable and or growing and on the other hand you also need to ensure that all the effort that your team is putting into the open source and upstream work is focused on all those areas which are important for your product strategy and it does not necessarily have to be just all those projects that you are using today but if you're planning to bring some more projects into your production environment or into your product you might start to let's say invest into those areas and build a team and have influence within those projects which you will need in the future because then by the time you will actually get there that you need box fixed, you need to introduce new features, you will already have the stable ground under your feet and you can kind of ensure way better that you can keep your timelines and you will be successful with all the contributions and product plans that you have. I think it's kind of true for multiple aspects that it's usually better to try to put together a matrix package and kind of combining all these numbers that you find available as opposed to looking at them as single values because if you just want to see a very high level view on whether something is moving or not then you can use one number but if you would like to also ensure that the process that you are having internally and the ways of working that you are trying to follow with the community where you are working then you need to kind of compare that how much effort went into the work and how much outcome you have and how successful and how productive the team is. When you are an individual personally myself I was way more interested in these numbers when I was a newcomer and when I started my journey with OpenStack because it was a really good way just to see that how I am as an individual succeeding or not succeeding in the community even if I was contributing on behalf of a company. So it can even be a kind of a self-motivation thing when you are seeing that you have comets merged, you have your reviews ongoing and you start to see all the positive aspects of your community involvement. And here what you usually try to really focus more on is that how you are doing all your things are efficient enough. So for example you can track your personal day effort along with your reviews, the comets and the comets that you had and kind of seeing the combination of these things as an overall number that you would like to kind of stabilize and or grow and you can see by this way that actually the work what you are having has a certain outcome kind of in the timeframe how you plan to get all your work done. And basically this was the information I wanted to share with you and we have quite some time for questions that I am happy to answer if you have any. Can you come up to the mic just so that because we are recording. Hello. Can this matrix be used for improving the code quality and improving the education of the newcomers of the community? Regarding code quality I would say that if the review numbers in the balance of minus ones and plus ones is kind of compared to the project size is a good set of numbers and it is not going down but more up. That is something that means that people are kind of actively reviewing the code. And I think it is more for you as an individual who tries to participate and who tries to ensure that where you are working will be stable and something that is easy to maintain. The review number is the one that you can concentrate on. Beyond this I think we do not track too many things which would explicitly tell you things about the code quality although I know about activities where we have data for example from universities and they are working on code quality improvements based on that data. I think in most cases they also make the data public so it is available. But so far we as a community we do not really track those and by data I mean for example code complexity or code duplication and these sort of things. We do not have these available as a community but we have teams and people around the community who are trying to keep an eye on this aspect as well. And I can really just encourage you that if you are interested in these areas and if it is something that you are already doing then please join and provide us information because it is really important from maintainability and efficient code development point of view to address these aspects as well. Regarding newcomers I would say that we are offering trainings and we are more trying to teach them how to use these numbers better and try not to misread them. This is what we have mostly in connection with onboarding new people and using these numbers. Any further questions, yes? Hi, thank you for the presentation. Just to mention for the audience we are part of the activity board but we haven't had the proper support for the last year or something. We finished let's say the specific relation with the foundation with that product so that's maybe the reason because there might be, you may miss some information there, said all of this. The question is my personal perception having a specific metrics like having top people contributing in terms of commits and so on, it's gamification but once you have a metric people may try to cheat on that metric so instead of having individuals metrics that maybe is as you say you should try to improve by yourself having your metrics but maybe from the foundation perspective you should look for some general trends and have to improve performance or bottlenecks but probably not going for individuals so I would like to know your opinion about this. You mean not track individuals but also try to... I mean tracking individuals the problem is that if you say this is the metric we are using, it can be better than others if you keep adding commits or smaller commits you know so how to fix this? That's a really good question and to be honest I don't think we have the answer for it. This is a discussion that pops up every now and then. I think it was a half year ago when I read a quite long myth read about it and I think it was about cold reviews particularly and personally my view is more on this that it would be better to let's say provide some education along with the metrics because I really don't think that you can correlate these metrics and do anything with them so that people will never ever do these games with it or they just don't try to game it but if we can provide let's say examples that in what way it is really useful and also kind of let them know that we try to follow these numbers and we try to reach out to people I mean on community level and community-wide not really directly as the foundation but we are reaching out to people who are doing these patterns and who are following these patterns and I think it's kind of better to provide the information before rather than go after them and talk to them that for whatever reason they are doing what they are doing it is wrong and it is recognized by the individuals in the community and it has a negative response on it so beyond that I don't really know how to fix this but it is definitely an interesting thing so kind of partially the reasoning behind this talk is also to share information and try to spread the word and give people ideas that if they put effort into using these numbers the right way then they can be way more successful in the community than if they just really try to compete by number by number Any further questions? Hi, I work on a community team and we have goals to be number X and number Y and reviews and such like that and this encourages poor behavior like quantity over quality as you mentioned before and we see things as like drive-by reviews drive-by commits and even like co-authorship what kind of goals would you recommend for a team instead of those? I would rather say that so the number of commits is at least something that gets merged so if you're using the number of commits then that activity is still something that was useful for the community but for example if you're doing this with reviews or patch sets because we are even tracking those so how many versions of a patch you're uploading that can really easily end up in behaviors like you're just rebasing everything in every second day because that will just improve or increase these numbers what we used to use when I was part of a team a team of contributors I think we multiplied the number of person-day effort the person-day effort number the number of negative reviews and the commits and we checked this overall number and it was kind of a feedback for us that how we are operating and for example if you don't do reviews and you don't try to really thoroughly go through a patch and see whether it is good or not if you never ever had a comment to any patch then there's something wrong so when that number is zero then your overall number will be zero so you can have let's say 2,000 commits but your overall number will still be zero so that's kind of a feedback that what you're doing is not the best community citizenship behavior that you can do and it's kind of true for the commits as well because you might be really successful with reviewing which is really important but also if you're part of a development team then your goal is also to get your code changes let it be a bug fix or a feature part of a feature into the code base so something else that I pay attention to in metrics is disagreements and reviews would you consider using those as part of your goals do you know what disagreements are in terms of stack-alytics it's when a core contributor will minus one after you plus one or vice versa yeah so I wouldn't recommend to put there a minus one just because you want to get that number up but if you have disagreements I usually consider that as a positive sign in the sense of no matter who plus two or plus one a patch you went through it and if you found even the smallest thing that does not look okay then you still put a minus one on it so in general having that number larger than zero is definitely positive but if it grows too rapidly then I think it should be an indication that you might not just kind of disagreed by for a reason but just to get that number up so kind of game the system again I think in both ways but anyway our time is four minutes we are four minutes over so thank you everyone for attending and happy participation and summit


What you doing? Huh? Hmm.  I'm Rick. Put up your window this guy Instagram Twitter for juicy content see your pin. Thank you. There would be sorts on patreon Follow us for sure to count subscribe and hit the like button. I just stole his ID and Today, we're doing a movie of you And it's a late to move review one. Yeah, one of the later movie reviews. We've ever done it most likely Yeah, we don't normally go to the late showing now. We don't I'm a sleepy boy.  Yeah Ladies come from oh, yeah Today we're doing a movie of you of the new just came out ljp Dost film Say it it is called They said it enough times. I was trying to get it down. Yeah, it's no way. Well Malakota. I valley button What she said she's she's a bongo. What does she know? Yeah, honestly, why what are you doing trying to Guys probably know more than she does. Yeah. Yeah, settle down back there. Anyways Obviously ljp's new film with the mega star Mohan Lall and it I do want to thank the Distribute I think their distribution production company production company.  Yes It's an easy drive if it's always gonna be a 25 minute drive But you're never gonna get any surprises. No, and so they they added a showtime For us and not just us we there were other people in the theater. Yeah, that was that we're with us But it was actually a good size house. Yeah, but I do want to say thank you to them Very very much paid anything. That was just a very they were very kind and they reached out specifically To do this and so I do want to thank them They because that we don't always get that and sometimes we either don't get them or we have to go far far away So yeah, thank you so much, but anyways this film obviously It's start LGP. Sorry Mohanlal. It just came out. So we're going to start with non-spoilers So to not spoil your little behinds here but It's in theaters now. I'd how the distribution is in India. I don't know obviously in Kerala. I'm sure it's wide In all that and I think it's pretty wide in North America So Rick your initial non-spoiler thoughts of this film. It's so hard to do this with non-spoilers It's very very difficult. So And it's it's just So to see it or to skip it It's gonna depend on Who you are yeah, if you are From I Think if you know a lot of what is being represented in this Which I can't say that because then it's spoilers. Yeah, you're because there's so much in this that is completely foreign to Us to me Symbolically whatever it may be So the cinematography is gonna be what you expect. It's gonna be off the charts Incredible cinematography just like you saw in the trailer and more There's a blending of genres I don't want to say you kind of get the gist of it from the trailer But I don't want to talk about the blending of genres because that's a spoiler for me I This has nothing I think to do with the film and has to do with my I don't be interested to see what you have to say my ignorance to Who's being talked about what the masks represent in certain places what certain things represent There's so much representative and some symbolic that is beyond it's kind of like Trying to introduce It would be kind of like it trying to introduce Shakespeare to someone who's just starting to learn English. Yeah, they would be like I I don't know I guess I I don't know. That's how I felt a lot of the time. Yeah. Yeah So it's a complex for you very complex for me about from a film making from well That's what I mean from the technical side of it I can appreciate the technical side of it, but because I didn't get a lot of the symbolism Yeah, it got I got lost and I didn't know if the reason I was lost was because I'm not picking up on what's being represented Or is the scope of what's being represented getting away from LJP, which I can't imagine that's the case So I put the blame on the fact that so much of what he's conveying is Completely new and I've never heard of it. I don't know what's being represented There are some aspects of it that are very Western, but I don't want to talk about that because that's kind of a spoiler Yeah, I like that part of it But as a whole is a film for a lot of it. I just kind of sat there and thought I Don't know what's going on. I Really for the most of the part I did most of most of the film. I did not I did not kind of comprehend what's going on This film is very It's a lot of things this film is a lot of things one would be Shakespearean a little bit Like a play a little bit. Yeah, it's a lot like a play It's in some spots a little bit of Tarantino sometimes a lot of it a Tarantino Lot of symbolism Obviously from stuff that obviously the same I didn't always get the the references going on because a lot of his films in the past that we've seen are very Christian symbolic behind it and so we get a lot of those references yeah, the worst Christian symbolisms in this as well But I do feel with the nun. I don't I don't think it's a spoiler, but a lot of Hindu symbolisms in this So obviously we are not as verse in that aspect And I think I regional regional maybe as well. Yeah I do think this is by far at least from this ones that we've seen his most stylistic Film is and his biggest in scope which we fell from the trailer. Yeah, I Think a lot of that is stylistic though the bigness So like, you know Tarantino is very stylistic, but a lot of LGP's films are Small nuance right this From the ones we saw I know there's many that we haven't seen yet of LGP So maybe yeah, he has stuff that that that we don't know of yet that it is very stylistic What I'm laughing about is for me. This is his tenant In terms of I just didn't understand what was going on a lot of the time gotcha does that make sense? Yeah I did I wasn't as confused about tenors as you were though Yeah, I still But yeah the The the scope of it and what he was trying to say and I'll get into some of that in spoiler I don't think I'm not gonna try to guess what he was saying with the full thing No, this would be an easy film to say, you know what he's saying, but you really don't know Yeah, yeah, I'd be ever interested with you guys who are Malayalees and well-versed and yeah Indian culture Malayalam culture all the stuff if he has references to other films of his in this Film itself, but man, I I would see this film just for the cinematography of it Good Lord Is it beautiful? You would expect I would expect that from him and a lot of it is very symbolic Symbolism as well And stuff that once again, I didn't pick up on But I'm like that means something right. It's clearly mean something clearly. I wish I knew what it meant And but a lot of it's extremely pretty Sanjay Libansali level at at times in terms of what he was doing with with certain shots It just could have absolutely been a play and the way it's thought it's it's very slow-moving in parts Like in the way it goes it goes by chapter kind of yeah, yeah And but and again Pacing is hard to critique for me when I'm not tracking with the film because then I don't know if what I'm sensing with it For example, if I feel like a film is dragging because I don't know if that's because of the flow of the direction and the editing Or if it's because I'm just not Correct, you know what I mean absolutely. Yeah, but of the stuff that I did I did enjoy a lot of it Like I said, I I'm okay with directors doing what they want to do and Me not getting it fully If I can just watch their vision on screen and be like I don't fully know You know what it what what just happened, right? I think I have kind of some know what this remind You haven't seen this film, but this is gonna be for those of you who have seen what I'm about to reference You haven't seen the boy in the Heron. No, not yet. We will I felt the exact same way watching this as I did The boy in the Heron. I think that might be because you're your first. Yeah, but I felt the same way of Mirzaki film this I can tell that this is probably great, but I don't know why and there's a ton of stuff in this that I just I'm watching thinking what the heck is going on. What does that mean? But I left with the sense of I can tell this was somebody who knows what they're doing. I Just wish I knew what they were doing Yeah, yeah, I get it. Yeah Man, I thought Mohanlal did a very good job. His presence on screen was really cool And not like it looked like a Raj Nekanth cool. That's not that's not what the film is very different Really it's not bad at all. Yeah. No, this is a LGP film through and through the star of Every LGP film I've ever seen is LGP and that's no different In this, you know, obviously this one has I'd say more of a star than a typical Lilo yeah film But he's still I think the star of the film I think if we do we're Molly Ali or something we'd be able to just dissect it better right than two American white guys. Yeah Then that's that's happened before I think and it happened a little bit with his last one even Because they were there were element not as much no, but there were elements of it that were very Tamil very Malayalam References and in certain things. It was a ton. We missed but that last one had a broader appeal in terms of it having a universal understanding Throughout more more so then then this one. I think a lot of people internationally would be able to appreciate the technical aspects of It and the beauty of the cinematography Yeah, but as far as story is concerned, I think there's it's it's it's I'd be surprised To find out how many people who haven't been exposed at all to India, especially this region of India Would be able to track with the story. I would be I'd be really really shocking to me. You were that lost Yeah, I was really really lost. Oh, okay. Yeah When it gets well again, yeah, we'll talk about I can tell you when I didn't get lost We'll talk it talk about that and in the spoilers, but overall in terms of like what I see I would definitely go see it especially if you're for Malayali and if you like Lilo LGP I can't say his full name Because You might be even smarter than we are in the realm of understanding of the film But I think if you want his style it is very different than his normal style actually very different But also still has a lot of elements of what makes a LGP film an LGP film, which is thinking I think that has to do with most of his films is like what is what are the undercurrents of this? This one has a lot of those And also even more than any other one I've seen because there's so much going on Mohanla is not going to give you anything to gripe about in this film The There's actually a ton of quite a bit of songs in in in the film Compared to what are normally in yeah, the LGP film true The cinematography you could go in literally just for that And I thought does the sound design was very unique very loud. Yeah, I think we don't know I don't know if that's the I think it's on purpose Yeah, it might be on purpose in spots or it could be the theater. Yeah, who knows That specifically but very unique In in how he decided to do it but overall I Enjoyed it. I think I you might enjoy it even more though If you if you're a Mali Ali or even just an Indian that knows the references better than I Don't know.  Okay, so Rick. What were you gonna say earlier? Just so I appreciate the genre mixing because this is a Western That I got this is definitely a Western in the in the whole scope of it and he gives a lot of shout-outs to the to the Western and He Mahanlal is the Clint Eastwood character who kicks everybody's ass and he's the loner and then he befriends somebody It's very outlaw Josie Wales and then the person that he was caring for gets killed So I got that at the end when they started to actually put exposition in the text Yeah, and say here's what's going on, but I still didn't get I Didn't get the big jumping foot in the ground and he's on the other opposite side of the hill And he's gonna face himself in his final battle, which is against himself I guess or another avatar of who he is. I that that I didn't get so I appreciated the blend of both what felt like a Sanjali, Libansali Indian film and a Clint Eastwood Western and But but I so much of the first 70% of the film was him just kicking ass and taking names and It was so comical sometimes with the wire work that I thought that was a stylist's choice. Yeah, I thought We're go. Are we farcical now? Where are we going in and out of farce because I thought at some points you're being really grounded and you're being really Honest in what you're conveying here in a way that you want us to feel a sense of reality and now it's becoming Hidden tiger crouching dragon or is it reverse crouching? I forget With the wire work So, yeah, that's I think it was absolutely supposed to be comical blend absolutely similar to how Me and also different to how Quentin does in Django in a lot of that the The fight scenes specifically were clearly Done stylistically in that way You know, I almost want to say they wanted to do it almost like it was a play how it would be done in the play How would look in a play? And I that the humor was absolutely there and supposed to be there on on those parts So that I 100% agree the one of my favorite parts though was God they he blended so many genres right Like that one song that came on where the the white people started to dance. Oh, yeah, I was like, oh my word Yeah, this is this is so outlandish and that's why I was like that's why the entire thing It was supposed to have the kind of no doubt that yes, it was serious at times, but it was also farcical It was also that it was obvious to me. Well, some of the time really obvious I know I think there was a lot of those really obvious moments but the I Want I maybe I'm just reading into it, but I think that whole sequence where The the kind of big battle sequence of the film with the army there with the army and her on the big gun Rotating cannons. I like that. Yeah, the big gun an animal. Yeah, I did. I I thought that was almost a Answer to not really answer like he's trying to do it, but no almost. He's poking fun. Yeah India's obsession right now Everybody having big mass guns. Yes, and and yes, I thought that whole sequence specifically It felt to me like it was almost a Pointing up a mirror like you see how ridiculous we're looking right now. Yeah Let's call it. We don't need a circular cannon gun and he kind of he does kind of throw in I'm not gonna say it's Masala, but he throws in all of the kind of things that you see in different Indian films So you get the big Bollywood numbers. Yep, you get the Ridiculous white people.  I see he's he's the bad guy. Yeah, I think the bad guy I think you guys can tell me if you read that in as well I felt like it was a little not that that was a whole point of the film, but one of the under count under Current themes of it was Almost a call out of the big like can we can we Pull it down right some guys cuz this is what we're looking like right. We don't need an oscillating cannon Right, just categorically hilarious. Yeah That wouldn't work yeah, but The whole song with their dance and all crazy Everywhere The beginning started off with that same comedy and in the end And not really the end of most of it in general I don't I can't tell you specifically what all the Hindu references were if it's even that It would part a lot of them were like the end was that was Shiva, right? Yeah I'm not sure it was blue The destroyer right returning Yeah, yeah, maybe it was a different avatar of him. I mean there's thousands in blue with dreadlocks. That's Well, I well, I think Mohanlal. I don't know if he's gonna be trying to be Hanuman as well Or maybe just a different app. That's what I'm just reading stupidly sent to it That's what that's what I'm trying to tell you all those are just like from what I know of the Whatever that film was the anime film that I saw was right Was it cool? I didn't see it, but you did the anime film from the 1990s. Yes. Thank you Rahmavayan that that yes, thank you That's basically what I know of all those gods So I was like, yes, I think the masks were the the demoned guy That Scythe was playing in that. I don't know. Was it was the demon with the heads? I know didn't mention them Who do you know who those masks were? That's what I thought I thought it was supposed to be like reflecting not specifically him but a demoned that they everybody in the crowd was like Anyways deceiving that's what I would read into it, but once again very stupid You guys could tell if though if my assumptions and any of that is true Because I don't know that was those were the parts that I'm like, yeah, I don't know I'm not familiar with anything. Yeah, because of that because of so much of it was symbolic I was looking for and couldn't find out Why anybody for the first half of the film in particular I couldn't really figure out why anybody was doing what they were doing And I thought okay, maybe that's the point Is that the point of the and then I just I had so many questions and couldn't track what was going on For so much of it that I just spent a lot of the film just lost Yeah, I like I like sometimes when when films just kind of Take you on a ride regardless of if you're Like relating to character specifically or I mean in LGP's famously done it and then the film he got sent to the Oscars We talked about that It's very much like when we came out of the boy in the hair And we were standing on the street about to cross it and a lady who was in the theater with us We didn't know it until she looked at us and she saw that we were in the theater and she said What was that? And we said we're still trying to figure it out. Do you know she said no I can't figure out what what that was all about. So Yeah, I haven't seen it yet.  No, when I see it. I will you will Well, Stephanie will and then I'll say whatever hard thing she says weird as I like it.  Oh, like We have obviously give us our aha moments, but I do want to once again. Thank the production nerd distribution house I don't know specifically because It's the first time it's happened to us. And so I do want to thank you guys But let us know what you thought about the film and anything else we need to know down below


Hi everyone, thanks for joining us. Governor Tim Walls is proposing a $49 billion budget that includes a 20 cent hike in the state's gasoline tax, along with significant increases in spending on education and health care while granting tax relief for many Minnesotans. His proposal announced today includes a $523 million increase in state funding for local school districts, more aid to cities and counties, subsidies to cut premiums on the state health insurance exchange by 20 percent, and $70 million to ensure that all households in rural Minnesota have high-speed internet access. The gas tax increase for roads and bridge improvements would free up general fund revenues that would then be redirected to education. As I traveled mile after mile and enjoyed countless conversations across the state, it became abundantly clear to me that there's three top priorities for the people of Minnesota, education, health care, and community prosperity. Minnesotans want to bring down the cost of health care, provide a quality education to their children and grandchildren, and ensure that communities across the state are prospering. The budget that I'm unveiling today will make significant strides in achieving these priorities. I've often said that a budget is far more than a fiscal document, it's a moral document. This budget reflects the morals and values of the people of Minnesota. This is the budget that Minnesotans voted for in historic numbers in November. Republican legislative leaders say their caucuses won't accept the tax increases that the governor has proposed given the state's projected budget surplus. House Minority Leader Kirk Dautz says that with a $1.5 billion surplus, it's absurd to propose raising taxes by more than $3 billion. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


question and answer at the end as well. So in the next 15 minutes, we're going to guide you through the process of taking a scholarly approach to your virtual world educational activities. And as we all know, virtual worlds are continuing to be an expanding field and many of the studies on virtual environments that have been done over the past 15 years have been fairly basic and exploratory. So we all have a great opportunity to contribute to the field of virtual world education by evaluating our virtual projects thoroughly and disseminating that knowledge through publishing and presenting. And so we're going to cover various project types, creating learning objectives, the potential types of settings you might want to use, the study design approaches, including a little bit about IRB approval and data collection as well as some tips for publication. And so in my next slide, I'm showing some data that I'm sure you're all fairly familiar with, but it just illustrates that there are huge opportunities for networking, collaboration, and scholarship with a worldwide network of learners and educators using virtual environments. Likewise, there are opportunities for publication and a recent review of virtual world related healthcare studies show that there are publications in the literature from educators around the world. And potentially those could be our collaborators. So stating the obvious, open simulator community conference and other virtual conferences are great opportunities to disseminate this work. And many of us are doing that, but as educators who want to not only share what we've done, but help others succeed, we need to take that next step to put more of the information out about what we are doing and how we're doing it so that others can replicate that work. And to do that, we really need to publish it. So first of all, I would like to talk about the different project types and take a quick poll from the audience. First of all, as you think about your project and what you might be working on right now, take a look at the various categories that I have up on my slide. And I have practice and simulation. I have games, distance learning, collaboration. I also have role play, informal learning, visualization, dramatization. Those are all various categories of work that virtual projects tend to fit into. So what category does your project fit into? And if you could type that in the local chat. And if you'd like, you could say briefly what kinds of learners you're working with and what you're doing so that we can tailor our discussion, but you don't have to. You could just type in one of those categories as something that you're interested in doing or have thought about doing. Okay, I have some distance learning from David and practice and simulation from Max. Hi, Max. Good to have you here in the audience. I believe we have one slide that shows some of your work down the line here or two. Nova visualization. I have good representation from the Vibe group today. All right, very good. So, and at the end, we do have a short handouts on a note card that will help to guide you as you start to think about project development in any of these areas. And maybe you're planning to start a new project, maybe or try to venture into an area that you haven't ventured into in the past. This will guide you step by step into how to storyboard and conceptualize the things that you might need for your new project. Because we wish we had time to go into that today, but we have so many other things to cover. So we have here a collaboration from 12th Night, Ellie Brewster, Gender Studies. Yes, hi, Ellie. Role play for literacy. Everything but role play and distance learning, okay? A bit of all of those altogether. Okay, excellent. So we have a little bit of everything. Very good. Well, that's wonderful. And so next, Carolyn will talk more about formulating good learning objectives that will help you to measure outcomes in each of these areas. I imagine most of you know about learning outcomes, but I want to go over it a little bit more. And sometimes in education, we almost forget that. Just imagine that you're going on a trip somewhere. Where are you going to go? How are you going to get there? And how will you know when you're there? So in education, where you're going, that's your outcome. How are you going to get there? That's your lesson plan or your project design. And how will you know when you arrive? That's your assessment. Now, many of you are in universities perhaps, and you may know that universities now require, accreditation requires that you have accessible outcomes. And so this has become, while in the past, that wasn't something in a lot of higher education situations. Now it's become top of the list of what we need to do. I'm missing a slide. I'm missing a couple of slides somehow. This is not good. Okay, there's two different kinds of assessment. There's formative assessment, which is what you do while they're learning. So this could be quizzes or... I'll just pass you... Okay, yeah. I don't know what happened to my slides, they were in there. That's not working either, Rachel. Okay, well, so what else? Maybe go off of the Google Doc, just to talk through, and I'll get your slides up here in just a second. Sure, thank you. Okay, to make a learning outcome, first you need to decide what you really want them to know. A lot of times what we assess is not what we really wanted them to know. Also, most of you are probably somewhat familiar with Loom's taxonomy. Is anybody not familiar? Just type no or in in local chat and see, so I can see. If you're not familiar, type an in in there. I'm just curious. Okay, so everyone is familiar. One of the things about virtual worlds is it's a great way to work your way up the ladder of higher level thinking. Thank you, JJ. So this is a modified Loom's taxonomy, and a lot of times what we assess is remembering. Those definitions, just memorizing those. That is the lowest level of learning. Virtual worlds are a great way to get up to the higher levels of learning. Analyzing, evaluating, creating. And so here is a great list. If you can click next for me. Okay. Thank you. Terms, should I read those to you? Probably not. But you can see that there are a lot of different ways to, different kinds of learning objectives that you can have. Students can formulate. They can calculate. They can categorize. They don't have to just memorize or sort or quote. Okay, so here's some examples. Next slide. Of some learning objectives that Rachel and I have used. And the first one I actually used this past week with my students who are going to be elementary teachers. The student will be able to create a tour through the cell explaining the organelles and their functions. And since my students are going to be K8 educators, their requirement was they had to be able to explain it to a third grader. So they couldn't use any language, a third grader and an 80 year old. So they couldn't use any language they wouldn't understand. The student will analyze a scenario and evaluate whether the environment presented is a healthy one. The student will design and create a virtual habitat demonstrating the requirements needed to maintain life in a closed environment. Okay, next slide. So here's some possible outcome categories. There's concept acquisition, which is just the content. What do you want them to learn of this unit that you're teaching? Learner satisfaction and motivation. Learner behavior in the virtual world or outside the virtual world, which one of my grad students is finding quite interesting. Performance in the workplace and on work outcomes following the virtual world experiences. So did they learn in the virtual world what they needed to know? And so here we have some learning strategies, some things you could do. And this kind of goes back to the slide that Rachel showed. Active learning strategies, the problem-based, case-based learning, collaborative learning, experiential learning, skill acquisition activities, active discussion, discussion during a lecture, back chat, which is really, I find, very powerful technology. And then there's that passive learning strategies that we often rely on, the lecture and the passive reading. Virtual worlds are perfect for those active learning strategies. They don't set themselves up quite so well for the passive. Rachel, I'll pass this on to you to talk about designing modules. OK. Well, now that we have learning objectives, we can move on to the next step of defining what kind of environment we'll use. And I'm going to show some environments that I've used for our learners. But certainly one thing to keep in mind is that functionality is really important in virtual environments. It can be really beautiful and very detailed, but if it's not functional and if the learners can't learn from it, then it's not achieved the purpose. So this slide shows the south side neighborhood of a bill that we call SPH Places. And it's part of a public health simulation for medical and public health students on the social economic determinants of health. And it's being used by our students for that purpose. But a very different bill is the East Africa Traveler Simulation, which many of you have visited, and I've been so thrilled to host a few tours on that, on the Vibe Grid, where travelers, people that are interested in global health, can learn about working with limited resources, basics of cross-cultural interaction, traveler safety. And we've created this more for medical students and residents who are traveling to global health sites, but certainly anyone who has an interest in East Africa could certainly visit it. And we're currently building a simulation for the West Africa region, which includes cultural factors like burial practices that have been responsible for Ebola transmission, as well as personal protective equipment for healthcare workers. And that's currently on the Moses Grid, and it's nearly ready for beta testing. But if you're already on Moses or have access to Moses, you can certainly hop in at any time. And of course, the environment really needs to be tailored to what the learners are doing, like I said before. So let's talk a little bit about what learners do in these environments. So the first example I'm going to give is the Biochemistry and Genetics Labs on Genome Island, UMACS, on Vibe. And this environment is specifically tailored to this purpose, and it's been hugely successful for students that are learning these concepts, and they can go and practice at any time. The next slide shows a giant cell that's also on Genome, and learners here can understand cell physiology by walking inside a cell. So the environment itself is the cell itself. So you don't have to have a large spread out simulation. All you need is just this one cell that the learners can use. The next slide shows an activity on Biome, which Chloe has created that allows students to carry out a forest diversity study. And this is patterned after what they would do in the physical world, but it's been recreated in Biome. And then I'm going to switch back to the medical field again and say that health professional students, and this does not just mean medical students, it can be nurses or occupational therapists or pharmacists, but they can practice patient interview skills. And there's a lot of data, a fairly significant amount of data that shows that you can teach empathy, you can teach how to deliver bad news, you can teach motivational interviewing to try to get people to see how certain health habits should be adopted, as well as cross-cultural communication. And the military has done a fair amount on cross-cultural communication, because they kind of had to. And the next slide, again, switches back to public health. And we've used the STH Places Southside neighborhood to allow public health students to do a community needs assessment. And so they go around the neighborhood chatting with community members and learn a little bit more about the community so that they can think about the kinds of interventions they might want to develop to make it a healthy community. So I borrowed this slide on gamification from the VWBPE talk by Hughes, because I really think it demonstrates the elements needed to make virtual world activities engaging. You have to think about the mechanics, you have to think about the reward and what behaviors you want your learners to exhibit and how you're going to measure those behaviors. And so some of the things that we've done have tried to get at some of those elements. So in the East Africa Traveler Activity, we have a student-square health tracker that shows their exposure to diseases. And one of the motivating factors of the activity is to stay healthy. And if you do get exposed to a disease-causing agent to try to get rid of it by treating yourself as quickly as possible, look through your manual, figure out what you need to take and use it so that you can stay as healthy as possible and your scores tracked in that way. And we did do kind of an adaptation of that for the public health bill as well. It was modified to include a stress tracker. And so I want to give a shout-out to Kira Komarov and Wizardry and Steamworks, because she did a lot of the scripting behind this. But these kinds of trackers can be used for assessment. And while this is one way to assess learners by tracking their progress and looking at a score, there are other ways to do that as well. And I'll let Carolyn share those with you. Okay, now we're in the right spot. Assessment is how you know that you got there. And in traditional classes, probably your classes, that was homework in the test. But now we know there's a lot of ways to assess, a lot of better ways to assess. What's important to know for our presentation is you have to know, in order to know that your students learn, you have to assess. For scholarship, which we're going to talk about, you have to have assessment. So assessment is essential in virtual world activities, just like it is in any learning activity. Okay, next slide. Formative assessment is what you do when it happens. So how do you know that the student is learning while they're learning? In the virtual world, that could be the observation of student behavior. It could be questioning the students. It could be practice quizzes. It could be a health meter that's working as they go. Summative assessment is the kind of assessment you do at the end, the test. But in virtual worlds, that could be a post-test. It could be a concept map. It could be a presentation or a project. Or it could be done in-world, an in-world presentation or an in-world project. Okay, next slide. Okay. That's yours. Yes.  One of our modules is designed in this way. It's the module on the impact of environmental factors on health. And our learning objective was to have the students discuss the distinguishing qualities of healthy neighborhoods. And so we selected an environment that had quality low-cost housing, clean parks, safe streets, sidewalks that allowed lots of social interaction, a new clinic, a new public library and community center. All sorts of services like employment services and childcare and community gardens. Spaces that allowed social interaction. And that we consider to be a healthy environment. So we let loose our students into this healthy neighborhood. And we had them do a scavenger hunt to find all these healthy places. How did we assess that? Well, we had the students take a picture of their avatar next to a healthy place and upload it into the course management system for their instructor to review. But they also had to annotate that photograph and say why it was a healthy neighborhood. The other thing that we had them do was wear the health tracker and so their health score increased with the healthy spots that they found. And that was recorded on a scoreboard in the environment. So we had two measures. One was qualitative and the other was quantitative. And as you might have observed, this can all be done without the instructor actually being in the environment with the students, which was a really key part, I think, to the success of this activity. The instructor of this course, I don't think he ever logged in, to be honest. But he was content with getting these materials back from the students as part of an assessment, uploaded into on course, which is our course management system that he's familiar with and he was able to assign them a grade. And this model has been used successfully with other activities as well. And I do think that one way of getting more courses done in virtual environments and more instructors to buy in to the use of virtual environments is not necessarily mandating that the instructor has to get in there and get their hands dirty because they may not have time to do that. So moving on, now we're going to discuss the different approaches to study design. And I want you to hold on tight here because we're going to go on a wild ride, but it's going to be fun, I promise. So my first question for the audience is how many of you or has anyone done a qualitative study before? And feel free to type yes in the chat. Anyone done a qualitative study? Focus groups, interviews, open-ended questions that students answer. Okay. Thank you, Chloe. One yes. Okay. So there are several basic approaches to qualitative studies. Okay, one more. Hi, Connie. The list is not exhaustive, but the focus groups, interviews, and open-ended questions, observations would be another. But there are certain types of approaches to qualitative analysis as well. And phenomenology is really that lived experience. Basically what we do with that is we ask the subjects certain questions to try to understand what their lives are like, and then you try to conceptualize that lived experience in your analysis. Grounded theory is another approach, which actually tries to develop a theory behind whatever phenomenon you're observing. It's kind of interesting that I've done some grounded theory studies. And then ethnography is actually one of the approaches that's very common in virtual spaces where there's a lot of observation. So you're trying to understand the life, the culture, what people do just by watching them and interacting with them in a sort of removed fashion. So the good news for qualitative researchers is that chat logs collect your data. And so if you use chat, which a lot of us do, then you kind of have an automated data collection process that you can just go through and review. Okay. Alrighty. So let's move on to observational studies. And I've kind of just done a cross-sectional study. Right now I asked you a question, how many folks have done a qualitative project in the past and some people answered yes. And so I could assume that anyone that didn't answer yes has answered no. And so that's my cross-sectional study. And I've just performed research. That is fabulous. However, there's a downside to the cross-sectional study. What if you decided to do a qualitative project tomorrow maybe as a result of what you learned today? Then I have not captured that. But a cohort study would because basically with a cohort study I'm following you over time and I'm watching to see what you do or if you develop a disease or whatever it is and from the medical side it would be disease development but from the educational side it may be some outcome or the other that I want to understand a little bit better. Or a case control study actually compares individuals that may have been exposed to a condition and intervention, whatever it is, or that have a disease in the medical field with those that have not. And so I'm trying to understand the differences between these two groups. Cohort studies are generally prospective meaning that I'm going forward in time. Case control studies are generally retrospective meaning that I had to look back to see oh you had this disease and or this intervention. Actually I should not be using disease because we're talking about education here. But for example we had some students in the global health activity they went to Kenya and that was their intervention. And now we're following, we're going back to them about 10 years later to say how are you different from a group of control students that did not go to Kenya. That's a case control study. So that would be an educational example. And observational studies are good because they're fairly easy to do. They don't require a lot of funding. You can study things that kind of happened in the past. But where you really start to get serious with research is with your experimental study designs. And I only mentioned this first one to say that a lot of virtual world articles use this approach but it's definitely not recommended. The one shot case study where there's no control group and it's really difficult to justify the results because they might have really just happened by chance. So this is the what not to do part of research. Don't just say, well A, got this treatment and then we did this measure and that means it's great because it really might not be. All right, so let's move on to slightly stronger study designs. The two group post-test comparison is a design in which you have this a group that gets the intervention and a group that doesn't and then you do your measure after that. Now the main advantage of this is if you randomly assign the participants to each group that randomization controls for your demographics and for any changes over time. And because you didn't have a pre-test there's no... the pre-test doesn't influence your results. So this is a pretty decent design. I mean it does have its downsides but I think it's a good one to try. The next one, again, is a... another design but it's not... it just has one group and it has the pre-test and the post-test and a lot of people will do this if you don't have a large group of students and you just want to do a quick study with a pre-test, you do your intervention and you do the post-test. It's really difficult to generalize these results without a control group though and so it's a fairly weak study design even though it's very popular. If you can do something a little bit stronger than this then it will be better received. So here we have two groups. We have an experimental group and we have a control group and again we have a pre-test and we have a treatment or an intervention and then we have a post-test. So great that we have a control group poor that we have no randomization. So while the groups may be different at the end of the study it may be that they were different to begin with. So a lot of times when we're doing virtual world studies we'll have a group of students that are fairly motivated maybe they'll volunteer for the study and so they may be a little different from the controls that didn't volunteer and so that intrinsic difference may account for your study outcome and there's no way to really tease that out unless you randomize. So our next example is a randomly selected experimental group and that's compared with a randomly selected control and basically you have your pre-test your treatment and your post-test and the key advantage here again is randomization so any differences that appear in the post-test will be really the result of whatever you did educationally with them rather than a possible difference between the two groups to start with. This is a classic type of experimental design but it has one downfall and that's the possible effect of the pre-test so the pre-test might in some way influence what the subjects do on the post-test because they've already been sensitized to the things that you're interested in. So the next design which looks like we may have lost one slide I'll talk you through it is basically a design in which you have exactly what you have on this slide but you have another group that does not have the pre-test so you have an un-pre-tested experimental group and an un-pre-tested control group and with the help of your statistician you can sort out the effect of the pre-test. Now you have to have quite a few students to do this though so you may not be able to do that with your first foray into collecting data but you should be able to do that as you get more and more students. So next we're going to talk just a little bit about IRBs and so IRBs are not our favorite group of people to work with but any data collection on human subjects that is going to be used for generalizable knowledge is considered research and so if you're working with human subjects which our students are human you do need to generally go through an IRB and so institutional review boards may small attempts at humor may not really have much experience dealing with virtual projects so they do have questions often on identity like how do you know that this person is actually who he or she thinks she is how do you know that they are the age that they say they are what about privacy on the internet well we know that doesn't exist and how are you going to incentivize so Connie might actually be able to tell us a little bit about his experience or his or her experience about passing through the IRB process but here are some good research practices to consider and if you include some of these elements in your IRB application it's not a guarantee that they will like it but there's a good chance that they might look upon you favorably and so identifying yourself as a researcher is always a good practice you don't just land in an environment and start collecting data without telling people who you are and what you're doing next setting up a consent room to provide an opportunity to identify yourself and allow others to potential subjects to ask you questions is a good idea and that can be clearly demarcated in your region marking areas with active surveillance and chat logging with the fact that that's going on is good practice and just be aware that there might be the risk of duplicate participation especially depending on your incentive try to tailor your compensation to the specific context that you're in so if you're in second life use Linden's don't say we're going to send you a check it's kind of silly but some people have written IRB applications like that and then read the terms of service I know we hate to do this but try to figure out which ones apply to the research and be sure that the relevant information privacy risks are included in your research materials that you're distributing to your subjects and be aware that any changes in terms of support may affect the participants and you might have to go back to them and update the material so very good alright thank you Cody so next we're going to very briefly touch on the after data piece so Heroin has had some experience with this as well and so we can both speak on the challenges of publication but there are two approaches traditional and non-traditional publications journals so peer reviewed journals with an impact factor and searchable databases like ESCO are kind of the gold standard a lot of virtual world educators have written books which have been hugely helpful for others coming up through the ranks and there are non-traditional approaches like creating web based educational information as well as educational software or video and YouTube is just filled with those but a short list of potential journals that you might publish in is listed on this slide and many educational journals now include research on innovative technologies for teaching and learning so it's not all inclusive but these are some of your best bets and I am going to go ahead and put a little plug in for the journal virtual studies and full disclosure I'm one of the section editors for that journal but it is the primary publication for the virtual best practices and education proceedings and what I'm going to talk about on this next slide is actually from my experience with this group and so just for us to know that journals are looking for papers that speak of the ability of virtual spaces to really transform learning or transformed individual it might be one or the other and any studies using an environment where the medium is not face-to-face can be considered virtual studies or virtual or acceptable for a journal like this so and they accept lots of different kinds of papers and papers from really any field of study and I know there's been a call for papers out for a little bit so if you have a paper you've been sitting on or maybe you're smarting from rejection from other venues and you want to give us a try feel free to do that right Caroline I'm going to pass this back to you because I know a lot of our audience must be thinking about well what kinds of resources do you need to actually be able to do all of this I don't know how much time we have so I'll go through this very quickly but to do all of this you're going to need some resources and it may or may not cost money but you need the basic technology you need computers that run the viewers which today is much easier than it was when I started in 2007 you may need help with scripting, building, curriculum design or basic skills you may need money we'll talk about quickly and server support which vibe virtual islands for a better education does provide some of that and you're going to need time and I can't help you with that but it's something you're going to have to find the time for learning, building, designing okay so where do you get this help a mentor find someone who's done it before my mentor happens to be sitting in the audience it's Max I found her right away and she helped me a lot as I got started experienced builders people that have done it before get them either to teach you or to build for you or for your students we Rachel and I can provide assistance and advice we can provide we have already built web page tutorials how to get on virtual worlds how to get your students on and we can provide workshops in a number of different aspects I think we're kind of running out of time money you're going to need money for a server or server space to pay for scripting or building services if you need that to pay for consulting to pay for basic yes and I've used student help which sometimes is free projects in their class or inexpensive so I think we're out of time actually I think we are actually done this is just about the last slide and so we'll throw the floor open for questions at this point and we have our emails up there in case you would like to contact us later time yes please do thank you Carolyn and Dr. Umoran for a terrific presentation this concludes the first day of the second annual open simulator community conference 2014 though the conference programming has ended for the day little field grid is hosting a social event this evening at 8 p.m. PST at the speakeasy dance club we encourage our attendees in world and on the web to visit little field for fun and post conference discussions you can hyper grid teleport to the event with your conference avatar or for more information see the conference program at conference.opensimulator.org we'd like to thank our audience in world and on the web and our speaker staff and volunteers for a terrific first day of the conference we'll begin tomorrow at 7 a.m. pacific time keynote regions with an exciting keynote address from Steve LaValley from Oculus Rift we'll talk about the race to bring virtual reality to a mainstream audience we hope to see you there