Okay, I guess it's time to start. All right. Well, thank you. I appreciate the science circle inviting me to give a talk about some of the activities I've been doing with students in Second Life. My name's Kurt Winkleman. I am a chemistry professor at Florida Tech down in Melbourne, Florida, which is on the east coast halfway between Miami and Jacksonville. I have to say most of the time I give this talk to a bunch of chemists and I have to explain what Second Life is. And so I'm pleased to be able to cut out those slides from this presentation. So this maybe we'll have more time to talk about some other stuff. If you have questions, please just post them in the local chat and I will... And so I'll try and keep track of that as we go along. But if I miss your question, just remind me at the end and I'll be happy to scroll back and look at it. So actually I'm curious, are there any chemists here? All right. So hopefully I'm not going to get too much into the chemistry. Oh, very cool. High school, excellent. I'm not going to get too much into the chemistry, but if I throw out some term that you're not familiar with, just let me know and I'll be happy to explain it. Most of what I'm talking about really is the on the education research side of things. Okay, so let's... Well, let's... Good question. What is chemistry? Watch Breaking Bad. They pretty much explained everything. Okay, so let's talk about the design and effectiveness of laboratory experiments in the Second Life. So I am... Let's see. Okay, just having a little trouble getting the slides advanced. There we go. So I want to first start with introducing my collaborator, Dr. Wendy Keeney Kinnicut, who's Julia Traxibar in Second Life. She actually is now retired, but she was a Texas A&M University faculty member. And the collaborator on this project, she has actually done a lot more work in Second Life students than I have. And in fact, this project deals with her students, not actually not mine. But she's a chemical education researcher who's been very active in Second Life. So as you may or may not know, depending on, I guess, how much experience you have or familiarity you have with educational uses of Second Life, Second Life is already used in many disciplines, especially in nursing and veterinary education. However, when we started this project a few years ago, we found that there were no published accounts of chemistry lab experiments conducted in Second Life. Now, I know of people in Second Life who actually do have students performing chemistry experiments, but there was nothing published at the time. And actually, there's still not enough published in the education literature. So we wanted to really explore this and see whether or not we could actually do some find measurable changes in student learning by doing experiments in Second Life. Students learn as well in Second Life as they do in real world activities and they enjoy their virtual experiences. There is, as far as science and engineering goes, there is a lot of content in Second Life. It's more along the lines of providing a space for tutoring or displaying some three-dimensional objects. Now, for a virtual lab in Second Life, they offer some real advantages. One is there's obviously unlimited options for activities. There can be assessment that's done in real time so that instead of a student taking a quiz at the end of a lab to for us to determine whether they learned anything, we can just actually track their activity during the experiment, look at a database of what student, the student actions that they performed in the virtual lab, and then we can just, you know, from that determine whether they did the experiment correctly and whether they learned something. Now, you can do a lot of that in, say, a virtual reality environment as well, but the advantage of Second Life is there's no special technology needed. There's a low cost of use. Schools have computer labs full of computers that often aren't used very much because now students have their own laptops and so we always thought, you know, Second Life was kind of a special place in that it was freely available and didn't require students to have a certain quality of phone or, you know, headsets or anything like that. And of course, there's a lot, you know, Second Life's been around a long time so it's a fairly stable place to develop activities. Of course, there is the cost of developing those activities and, you know, the learning curve for users is a little bit steep so we have to address that whenever we introduce new students into Second Life. So, as I said, there was no prior published evaluation of student learning or attitudes for lab experiments conducted in Second Life which replace real world experiments. In terms of virtual labs, there are a lot of publications or published articles talking about chemistry labs done in a virtual environment augmenting or being like a pre-lab activity before going into the real lab. And of course, you know, virtual pre-lab activities are beneficial as well but we really wanted to look at actually replacing chemistry experiments that were conducted in a real lab. So, Wendy and I got together and we replaced two general chemistry two experiments at Texas A&M with Second Life versions. Now, Gen Chem 2 is a freshman class, you know, mostly first year students. It's obviously the second semester of an introductory chemistry sequence. Now, we did two experiments. The first is what we call a titration, which is a very careful way of determining the concentration of some chemical in solution. In this case, they already had a laboratory experiment developed where they tested the salinity or salt content of a water sample. The idea for that experiment was that these were water samples taken from an estuary in Texas and so they were determining the salt content. They weren't actual water samples but that was sort of the background lab. And the second experiment was collecting gas over water to determine the molar mass. Molar mass is a big idea in freshman chemistry and so this technique is fairly common. So, we found two lab experiments that we could replace the Second Life versions and the lab experiments were also ones that would be very commonly not just at Texas A&M but at just about any other university. So, we have some research questions. Sorry, I have to pause here. I'm having, on my screen, these slides are not rendering very well. Is anybody else having trouble? I see them. Okay. Oh, good. All right, great. Well, I'm not sure what, that's fine. I've got, I know what I'm talking about so I don't need to see them. So, we have research questions. We wanted to compare student performance in the Second Life lab experiment to their performance of the same experiment conducted in the real lab. So, our four questions were do they, how does this virtual experience affect their attitudes about chemistry, if at all? How much do they learn? And not just learning in terms of textbook content knowledge but also their hands-on skills. This third research question addresses a very common concern, complaint criticism whatever that I get from other chemists and that is that you can't teach hands-on lab techniques in a virtual world, especially one like Second Life where it's driven. And I'm not sure that that's actually the case but we wanted to find out. So, we wanted to know if they can learn kinesthetic skills from the Second Life experiments and do they enjoy the experiments as well? Because that's obviously important. It's an important component of learning. So, I'm going to first talk about developing and implementing these experiments. And so, the first thing we had to do was take our real world experiments that we knew how to do and convert them into a Second Life version. And so, we had some programmers, well, I'm sorry, before we could send anything to the programmers, we had to perform the experiments and rewrite them in a way that nonchemist computer programmers or software programmers could understand. So, there was a little bit of translation involved there. We performed the experiments and recorded lots of video and images so that everything was scientifically accurate as possible. And so, we also wanted to take into account that students obviously, you know, they sometimes make mistakes in the lab and we wanted our experiments to incorporate those possibilities. So, we actually spent a lot of time making mistakes intentionally and then, you know, putting that into the program itself. These particular lab experiments are, there's a fair amount of calculations that students would have to perform in order to, you might say, get the right answer or answer questions about the experiment. And so, a lot of the outcomes of the students' actions can be mathematically modeled. So, if they add a certain number of grams of one chemical, then they're going to get a particular result. If they add a different amount of chemical or they wait a different amount of time for their chemical to react, that all plays into the outcome of the experiment, which is mathematically modeled. We use professional programmers and I'll, actually, they're included on the acknowledgments at the end. So, if you're interested in finding out who did this work, I'll be happy to share that. The developers had experience working with educators, which we found important because educators can be difficult to work with. So, they had some, they understood where we were coming from in terms of, you know, dealing with students and things like that. So, the developers were actually great. We did run into some implementation challenges. There's a need for full support from the department, the academic department that's offering this, in our case chemistry, and the IT department. The students and lab instructor buy-in is also important. We found this out kind of the hard way.  And if you don't fulfill those expectations, they can kind of get upset and, you know, that's when they start complaining whether things are too hard. It's just different than they expect. So, we really worked hard to explain to them that they were not just guinea pigs in this research experiment or this research project, but, you know, that we were also trying to give them an experience, you know, working in a virtual world. This is, you know, something they're going to probably be down the road when they get careers in science or engineering. And so, you know, that actually helped with getting them on. And obviously, we don't want to create work for them, and we want to provide the training that they need. Yeah, okay, so there's some comments about, there's some comments about, you know, safety and how students experience, what they experience in the lab. And it's true, they can't smell chemicals, although we really don't encourage that anyway. In a lot of cases, if they perform the experiment safely, they shouldn't smell anything. As far as holding, you know, a beaker in your hand and feeling it react, that's true. You know, there's some things that we have to give up when we go to a virtual environment, but I think there's also going to be some advantages as well. And yeah, the buy-in, yeah, the buy-in is important. But yeah, you're right, if you have days when students can't meet in the lab, then a virtual lab is really good. It also relieves some overcrowding from existing labs. Yeah, I think they enjoyed not having to worry about setting themselves on fire. Actually, that really was a serious, I think, an important point that the students raised, actually. And so we'll get into that. So our design of the lab experiments in part was guided by the existing lab environment. On the left is an image of the Texas A&M chemistry lab in the real world, right up overlapping with the virtual version of the lab. We wanted students to feel comfortable like they recognized the virtual world they were entering into. And so we wanted to make it as realistic as possible, the idea being that we wanted them to be immersed. On the right is a picture of the lab building, which turned out to be it was not essential to them conducting the experiment, but it was essential for them to feel immersed when they would go to, when they would log in and we sent them the station to teleport, to teleport in. This is where they would, this is where they would show up. And so then they'd have to walk into the building. They'd walk past a cafe and some offices that were empty, but they looked like faculty offices. And then they'd go down a hallway. It was set up like a real academic building. And then they'd enter this virtual, this chemistry lab. The actual experiments were conducted on lab platforms that were in the sky that they couldn't see from the ground. But as far as their entering into the virtual world, we wanted to make it as realistic as possible. Okay. So let's see. We, okay. So when we designed the experiments, one of the challenges that we, we had two major challenges. One was that in order, I mean, there's a lot of lab, lab equipment that students use that has very, very fine markings.  Second life is, is limited in how much you can zoom in. So you can see on the picture on the right here that we have lab equipment like this balance and a tub of water that is significantly larger than the avatars. And so that was just one way that we had to, one way we felt was necessary to make the equipment big enough for the students to easily use. Now it seems kind of comical and, you know, it is kind of funny looking. But it really wasn't that big of a distraction because the students would often zoom in and look at it from a really first person point. And at that point, when they're just looking at the lab equipment, it all seems relatively the right size. Yeah, they shrunk themselves. That's funny. So the other disadvantage of second life that we had to deal with was the, you know, a lot of it had to be menu driven. And we would have liked to have more sort of free freedom for the students to do things. But often, as you can see here, they would click on an object and they would get a series of options, things they could do with that object. And, you know, there's only, you know, often there's only a lot of, there's only a limited number of things you can do with an object. So the list was fairly comprehensive. But still, it took away from some of the realism. But we did have some advantages between things in second life that took away from the realism, but that was a benefit. Like this little button down here is a reset button. In the middle of their experiment, if they messed up, they just click that. Everything goes back to where it started and they can start over. In a real lab, they would spend 15 minutes cleaning things, getting things rearranged and back in order. And that's kind of a waste of time we felt like. If they want to start over, let's give them as much time as they can to actually do the experiment. And of course, there's chat. And so that was nice. The students, some of them use headphones, some of them use local chat so that they were able to communicate. They didn't actually have to be sitting next to each other. And as the picture on the right points out, these were, these students were working with lab partners, which is an important part of chemistry lab experiments or experiments in general. You know, learning is a social activity. And so we wanted them to be social in the virtual world. And we found that, in fact, they really did kind of behave the same way in the virtual world. All right. So let's see. And other things. Other things we did to help them was they, you can't really see it. It's kind of in the back here. But there is a button to call for help. The picture on the right is from the viewpoint of one of our, one of the teaching assistants. What you see here at the bottom is a heads up display, which allows the instructor to teleport to any lab platform immediately. So when the student clicks the call button, you might say, which is up here, then one of these buttons lights up, and the instructor can teleport there and walk with the students, just like if they raise their hand in a classroom. We also had a feature where, and this is Wendy, back behind our one way mirror, the back of the lab area was a one way wall or window, I guess. So we could stand behind that and we could view the students doing their lab experiment. They wouldn't see us and that enabled us to really kind of watch them without them feeling like we were watching them. We wanted the students to be able to do their work. But we also wanted to be able to monitor what they were doing. And of course with the oversized lab equipment, they could zoom in and see some of the details for some of the equipment. I'm trying to read some of that, some of the chat here as well. Yes, okay. So wanting to see explosions. So when we started this project, we really, we had to think about what type of experiments we wanted them to do. In our case, we chose these experiments because they were ones that were being done at Texas A&M at that time. And we wanted to have a good comparison. If we weren't conducting the research study, we would just create the most fantastic type of experiments that would be entertaining and educational, but also unusual and things that they can't do in a lab. So yeah, working with things that could explode would be really fun and can be educational too. But because of the research design, we actually had to we had to tone it down and make the experiments kind of normal. Okay, so our experiment features, we tried to emulate real life as much as possible. Well, to some extent, we made these are the choices we were allowed to make by designing the virtual experiments is how realistic we wanted. Measuring things are not always exact in a chemistry lab and we did want students to have that experience. So dispensing chemicals, they wanted a certain volume of solutions. They would have to be very careful in how they dispense that it was automatically appealed. There was always some random error. We could have had a pre-lab experiment, I'm sorry, a pre-lab quiz in Second Life, but we chose not to again to make it as as equivalent to the real world as possible. The students did their quizzes on. And as I said before, students make mistakes and that affects their results. That's an interesting idea that the virtual experiments encouraged them to encourage people to do the real thing. Hadn't thought about that. Okay. So when we had developed the experiments and then we implemented them, we had students perform a pre-lab, some pre-lab activities if they were in the virtual lab group, they viewed some tutorials that created an avatar and they did some mandatory training. At first, the training was optional, but we realized pretty quick that they needed to have some basic understanding about how to walk around and if you see things in Second Life. So the training turned into something that was mandatory. So they did all that the week before they started the first Second Life experiment. They met in the computer lab to do the Second Life experiment. So they were all meeting as a group, just like they wouldn't meet as a group for the regular class. They work with a lab partner and we had a lab instructor, a TA there as usual. The TA was in the room, but also in world. Okay. So again, our four research questions, we had a control group, we had a virtual lab group, and so we were able to compare outcomes of the two for each of these four research questions. For our assessments, we use what's called the attitudes towards subject chemistry survey, which is a well-known survey for attitudes about chemistry. Pretty close lab quizzes and lab report grades, major content knowledge. We had a lab practicum for the experiment where they collected gas over water. They had to actually assemble a half a dozen little things together in order to take a measurement or collect some data. So we performed the lab practicum and then we had opinion surveys after the activities were complete to get their attitudes about the virtual experience itself. Okay. So as I mentioned, this is a general chemistry two course. The lecture class was separate from the lab class. So that didn't have much of an effect on what they did. The lecture didn't have much effect on what they did in the lab. It were randomly assigned to the two groups, and the teaching assistants, the lab instructors were also randomly assigned, but each teaching assistant taught one second life group and one control group as well. We had about 140 students in each of the groups. Okay. Our population, as I say, this is a first year course for most students. They were similar in just about every respect that we measured. Typical young college age, mostly female Caucasian, and a vast majority were in science, engineering, agriculture, or sorry, veterinary science. So we had a, what I would say is a fairly typical traditional group of college students, at least for a large public university in the U.S. Okay. So we started, we'll start with these student attitudes. The ASCII survey covers the subject of chemistry in general, not just their attitudes about the lab, although that is a part of it. It's 20 survey questions asking them about anxiety, interest, emotional, sorry, emotional satisfaction, intellectual accessibility, things like that. It's a, it's a, it's like a what they call Likert survey. So the students get two, there's two statements, one on the left, one on the right, one says, I like chemistry. The other on the other side says, I don't like chemistry. And then the students choose on that, on a seven point scale, which where their answer lies on that continuum. We didn't see much effect of the virtual labs, really of any of the labs on student attitudes. Their initial attitudes were typical compared to what's reported in the literature for other large chemistry classes. Both groups showed similar pre and post scores on this survey, meaning that neither the real world labs or the virtual labs had much of any impact. And there was also no impact on, no effect of gender on outcomes. So I get, I should mention the students, they performed these two virtual experiments, but that was among a group of like 12 experiments that they performed throughout the semester. So they had two weeks in second life, and then they had two, 10 weeks in the regular lab. The control group did all 12 real world experiments. Okay, so then, sorry, get my slides out of order here. So content knowledge gains, this is where we did start to see some impact of second life. Of course, both we would expect and we were pleased to see that both groups learned something during their respective lab experiments. So the quizzes they took at the end of lab showed that they had learned something compared to their initial scores on the pre lab quizzes. But on one of the quizzes, the second life group made even greater gains, and they made equivalent gains on the other quiz. The quizzes were multiple choice randomized questions, but basically covered the same content. The second life group performed better on lab reports using the same rubrics. Now the lab report results were interesting because students wrote the lab reports throughout the week, mostly probably the night before they do. They wrote their lab reports outside of second life. So even the students might have spent a couple hours in second life doing their virtual experiment, but the data they collected there was of higher quality, so to speak. And so that helped them write a better lab report. They perhaps understood what they were doing a little better in the second life experiment so that they could write a better lab experiment. And both groups achieved similar results for the other lab report. And just to remind you, the grading was done by TAs that supervised each one class of students in each group. So the same TA was grading a control group and a second life group. Okay, so now laboratory practicum. This is what, personally this is the part I was most interested in, because I took a lot of heat from colleagues about students not being able to learn hands-on skills. So one of the experiments involved them collecting gas over water. The gas was actually butane and you can see the butane is in the cigarette lighter. The idea here is that they weigh the lighter before and after the experiment. They release some of the butane during the experiment and so they get a change in weight of the lighter. The butane goes from the cigarette lighter through a piece of tubing that you can see here up into a cylinder called graduated cylinder that has been immersed in this tub of water and filled with water and then held upside down. So as the butane goes into the graduated cylinder it displaces some water and so they get a volume of butane collected in the graduated cylinder. They know the mass of butane based on the change in mass of the lighter that they recorded and from that they can do the calculations to figure out the molar mass of butane. This is kind of a standard chemistry experiment. But to actually do this experiment they have to connect to tubing. They've got to put the graduated cylinder into the bucket of water. They have to burst the graduated cylinder so that they have to fill the graduated cylinder with water. They have to invert it. They have tubing up into it. It's a multi-step assembly process in order for them to in order for them to actually gather a single data point. Okay so what did we find? Well we looked for the laboratory practicum. We looked at several steps. Now we performed the practicum the following week after they after everybody did the lab experiment where they collected gas over water. The next week we had them come in it while they were doing the next experiment that week. We called up each student individually. So while they worked as a team as a pair of students during the actual experiment we called them up to do the practicum individually. We looked at the overall practicum score but that overall practicum score was the combination of being able to connect the tubing, arrange the cylinder, collect the gas, adjust the height of the cylinder which is a I forgot to mention but they have to they have to adjust the height or position of the graduated cylinder after they put butane gas into it so that the water level inside and outside of the the tube are the same. It's an actually an important point in terms of the chemistry but it's just a detail that they had to to they had to perform in order to get the the correct answer. They had to read the the volume of an upside down graduated cylinder which is you know not necessarily as sounds. They had to record that volume. They had to write the number the correct way. Significant figures started and then a part that all of those items were part of the overall practicum score. We also asked them how confident are you in your measurement? Do you think your measurement did you think you did this correctly? And so okay so some of the results most of the results actually show within within a 95 degree 95 percent confidence the results were the same. The overall score between the control and experimental groups the students doing the real-world lab and the students doing the virtual lab were the same. They were able to do these hands-on skills just as much overall and some of the skills like connecting the tubing or collecting the gas they they did do equally but there were some notable differences. So arranging the cylinder the control group did a better job actually positioning the cylinder the graduated cylinder in the tub of water. They did that better but the second life group recorded the volume better. So overall they got the same score but they second life in some cases gave them help them learn skills better in other cases not as well as the real world. But what was I think most interesting is the degree of confidence that the students had. The second life group were less confident performing the practicum in the real world. So that kind of points to a maybe a challenge for educators who design virtual experiences that students can they can learn just as much. Obviously they learned just as much. We saw that in the previous slides. It just as well in the lab quizzes if not better. But they may not feel as though they learned as much simply because of you know a bias against a virtual experience versus the real world experience. And we've we've actually seen this in a pilot study I did with some high school students as well. So it's it's not unique to this particular to this particular project. Sorry just trying to keep up with the chat. Okay so whoops there we go. Okay so the last research question was about their opinions. So all the students in both groups answered a 10 question survey about their real world experiment experience. You know what they did in the real world lab. All the students did at least 10 experiments in the real world lab. Again it was a what we call a semantic differential survey. You know two questions. I liked chemistry lab. I did not like labs and it was a five point scale between those two. They both had similar both groups had similar views of their real world lab experience in terms of how difficult the experiments were, how enjoyable the time and effort and how much critical thinking were required. All that was both groups had that the same experience which was good. Then the second life group took a second survey. It was the same survey except we swapped out real world experiments with second life experiments. And the second life group said they really liked the second life experiments.  They did say the procedures were easy to perform and I think and then later on we asked them some follow-up questions and that was it was basically because it was menu-driven and they were able to just click things as opposed to you know picking up an object in the lab and wondering you know what do I do with it. The menu-driven procedures sort of focus them on you know what they had to do. But despite that they thought that the experiments still required critical thinking and you know once they had their data from the experiment they still had to analyze it the exact same way. The experiments in second life didn't spit out an answer for them just allowed them to record a measurement in their lab notebook. So analyzing the data writing the lab reports that was the same experience. We then asked them to compare the real world and second life experiments. They said that the second life experiments took less time. That interestingly wasn't the case. The second life experiments took just as long but the students were I think more they used their time more productively. They weren't waiting around in the lab room you know trying to find pieces of equipment. Everything in the second life lab was sort of laid out for them and so it was sort of a very I think a more straightforward experience for the student. In the virtual lab they saw all the equipment they need it's all there. They weren't distracted by other students. Where do I find this chemical and so forth. They thought the lab reports were easier to write simply because they felt like they had better data. They thought each type required the same amount of critical thinking in terms of getting the right answers and although they liked the second life experiments and saw the advantages they did not necessarily want exclusively second life experiments in the future. They really wanted a mixture which I thought was kind of a I don't know maybe a mature response even though they thought they were some nice features about the second life they didn't want to give up the real world experience. So we asked them in some focus groups to give us more information about how they felt about second life. They said it was less stressful being in a virtual as chemists and educators you know were very comfortable doing the learning activities and being in that learning environment the students can find it very stressful and I sometimes forget about that so being able to do a chemistry experiment in a virtual lab you don't have to worry about safety they don't have to worry about spilling things and all their peers looking at them like butts. They don't have to worry about breaking stuff and having to pay for the lab equipment you know I mean we charge students if they break a beaker you know it's a couple bucks so they don't they don't have to worry about dealing with actual chemicals. Now maybe you want them to gain those skills and certainly if they're a chemistry major they should gain those skills they should they should overcome their you know apprehension about being in the lab but we have a lot of students who just take one semester of or one or two semesters of chemistry. Do they need to worry about that? I don't know maybe I think it's a question that educators have to ask themselves but a virtual environment can eliminate some of that stress. The lab environment is less distracting in second life it's more organized again they don't have to wait around and to to get their chemicals and things like that and they did again as I mentioned the procedures being menu driven because they felt that was a big help. Okay we also had a focus group for just the teaching assistants the lab instructors and this was also some surprising feedback we got from them. Initially because we didn't do a good job of getting buy-in from them they felt like they were on the verge of being replaced and so they they were initially very much against our second life lab experiments. We realized that we had to reassure them that that was certainly not our intention or it really wasn't even going to be possible that they could ever be replaced but they didn't know that. They did say that the second life experience experiments were better when the labs were overcrowded and there was insufficient equipment so students had to you know even lab pairs lab partners had to share equipment with other lab partners in other groups and that was that can be kind of a hassle and although the students cited the the fact that in second life you don't have to wait around in line to get stuff and that's true the students thought that was an advantage the TA's actually pointed out that these students actually can learn stuff while they're waiting in line they talk to each other so as they're waiting in line they ask you know how did you do that part of the experiment or you know they would they would ask each other questions so that they could better understand what they were if you take away that time that they're forced to wait in line then you take away that opportunity for them to to chat with each other so we thought that was actually an interesting interesting point it's kind of a making the lab experience more efficient is sort of a double-edged sword okay so overall the outcomes they had similar attitudes about chemistry at the end of the semester they the second life group slightly outperformed the control group but they at least did as well um on the two experiments they performed the practicum equally well but there were different aspects of the practicum that each group better than the and the real world group had a greater confidence in their ability um both groups had similar views about the the real world experiments the second life group thought these the second life experiments were a little easier because of the menus um they like them more than the set the real world experiments but they really wanted a they would prefer a mix of the two so what did we learn um I think from this we were we were confident second life experiments are effective substitutes for real world experiments they learn just as much they gain the the kinesthetic skills they have favorable favorable attitudes about their experience in the in second life um yeah educators who have worked in second life already know training is is mandatory um but we need to be cognizant of the feelings of the the graduate students if we have graduate students um or other colleagues who are teaching the labs we don't want them to feel threatened by uh it phased out by by a virtual environment um the the students felt like the experiments were were shorter but they actually weren't and they can gain there's some benefit to having them wait around and chat with each other um so this last slide I think this is my last slide um is one that I share with like I say mostly chemists who who aren't familiar with second life um you know some of the advantages of of virtual worlds in lab education um and I'm sure a lot of you already know this but I always like to point out that you know when NASA and all the the airlines the military and medical schools all currently use lots of virtual education maybe just maybe if our astronauts and military and doctors and and airline pilots if they can learn their trade by in a virtual environment maybe it's okay that our freshmen learn a little bit of chemistry in a virtual environment um in a real world lab students learn less than we and that reminds me I with the practicum all the students did the practicum individually when we called up some of the students who had done the experiment in the real world they told us they didn't know how to do it said well of course you know how to do it you did it last week and they said oh no I didn't do that I didn't assemble anything I was in charge of recording the data my lab partner did all the assembly so even in a real you know in a real world environment where the students have an opportunity to gain hands-on experience manipulating lab equipment they don't always take advantage of it um and so you know they they they still might not learn those those real world skills that we want them to and of course virtual worlds provide access to can provide access to a quality lab experience for online students mostly who are most of them are female deployed military students with disabilities and students in underfunded schools um let's see and ah my last slide is acknowledgments um my two developers were zandy mars and random coal um and so I would highly recommend them um some really high quality development uh student activities um i co-pi on this uh dr wendy enie kinakutt our assessment team at texas a and m we uh lennie bernice was an undergrad at the time who helped um it was fun to be able to tell a student hey go into the lab do these experiments and make all the mistakes you can just record what the results are so that we can send it to the developers and then we had some nsf funding for this and i thank you for yeah go aggies um yeah so i want to thank you for your time and uh i'll be more than happy to answer any questions and i'm sorry if i missed any in the chat um you guys were very active so um we can keep up with everything but yeah let me know if you have any questions okay so let's see how much resistance did i get from students um and yes we did we did force them in um if if a student had a real problem um i mean if they just had a i don't know ideological objection to doing a virtual lab we would have let them switch to a real world lab section um but we didn't necessarily advertise that opportunity so we uh you know we just they didn't complain the the students really didn't complain a whole lot um trying to think if there was any yeah i think because it was because it was only a a two week um set of activities they didn't uh you know and and and they were still they were still graded the exact same way so they didn't um i guess they they weren't required to do a whole lot of extra work and they were still graded in the the traditional way with a lab quiz and lab report um we i actually i was asked uh earlier about doing like a field trip and that's something that we can arrange um the the our location it was posted in the the local chat um to be honest i'm not that will take you to um what's called 12th man island which is one of the uh um one of the islands that that uh Texas A&M and and that that wendy actually still maintains half of it is her island um and half of it is ours that we use for this project i don't know if i think we had to whitelist people to get we had to whitelist the students to get into um to get into the the second life lab just so we didn't have any grieving um problems i'm not sure if that's still in place so if if you if you want to go there and check it out by all means you know try it um but uh if you may you may have some some trouble getting in if not um if you do have problems um let me know and and i can i can show you around or i think we're going to organize a field trip here pretty soon uh let's see let me i'm trying to scroll up here um okay so let's see um there are i um let's see Texas A&M they no longer offer the virtual lab experiments um to their students and if like i say at Florida Tech we have a a traditional chemistry lab um for where all the labs are done in the in a real world environment um there are some schools that do completely virtual labs um i know ucf does completely virtual um biology labs in in i think they're in second life um and there's a school in Alaska Prince William Prince William Sound Community College or Prince William Sound College I think it is now their chemistry lab is completely done in second life um again it's it's you know the advantage well it has advantages um and the the downside like well let's say the downside there are advantages obviously the advantages to doing it um and the i guess the there's they don't have a concern that their chemistry majors aren't getting a full lab experience because if you're a chemistry major you're doing lab classes for four years if one of your lab classes is virtual it's not going to make probably not going to make a whole lot of difference um so that was kind of a long-winded answer um this this is the second life is the only environment that we've looked at um we are in the process of well we're we're actually searching for funding to develop some more live experiments that we are going to develop in unity that way they can be students can engage with them on a desktop computer just like they do with second life or in a a VR headset um let's see uh as far as being used for math i'm not i'm not sure what i'm sure people do i mean there'd be a lot of actually neat things you could do with showing graphs and um things like that in in mathematics um and i know like wendy has done a lot of work with students learning structures of molecules which are three-dimensional you know she shows some three-dimensional models in second life and that really helps um and she also conducts tutoring and office hours in in second life so yeah it is i think it is very effective uh let's see how much time we well we had the question is how much time did we have to have to develop the experiments it's hard it's hard for me to say because we gave the developers like a full academic year and that was just based on the scheduling of when we got the funding which was at the end of the summer um we told them you know have the stuff ready in 12 months and they had it done you know by the summer and we pilot we beta tested it ourselves during the summer we probably could have gotten away with it getting done in maybe half that amount of time um but to be honest i'm not really sure i don't think developers will work full time just on our project so had they worked full time i'm sure they could could have been done in a few months um i'm okay the question is what are some other successful uses of second life for a real-world university um education especially in humanities and social sciences there there are a lot of um i off the top of my head i'm not aware of of specific people or or schools but i think there's it's more commonly used in humanities and social sciences um because you can send students to say a historical time period and they can you know go into a region of second life where developed a you know 17th century france um island and students can walk around and chat with people see the architectures you know and and learn about that time period in sort of environment um and i know there's also a lot of there's some interesting uses um for a second life in uh in psychology i i've again i don't know who but the the neatest one i've heard of is to help help people understand what somebody help people understand what somebody with um like uh schizo with schizophrenia experiences they go in in in second life a person actually see things as somebody would see them if they had schizophrenia and and and other problems like that and so it kind of helps people relate and understand what what some people go through um in a little bit better way um let's see biology and astronomy oh interesting okay so i didn't know some of this uh i didn't know about yeah biology i was aware of um i didn't know about astronomy that's interesting being available uh okay so i think if anybody has more questions i'm happy to um hang around in chat and like i say we'll try and organize a a field trip this is exciting um um and i appreciate your time oh thank you again thank you for everybody for uh for coming and thanks to science circle for inviting me this has been this has been a lot of fun um somebody asked about laser pointers the the screen that i got from that i'm using has a laser pointer built in that seems to work at least for me um pretty well let me look up what screen this is it's slide presentation screen version 7.1 oh yeah thank you that would be great um would love to well i don't know if i really want to see myself talk but it'll be it'll be need to to share with other people okay we'll take care everyone and uh i will i will see you around oh i probably want to take this down um i sent you a bunch of stuff and um i ams uh i think we're doing some amazing work if we could talk some time let me know i sent you my email and real life information but i think we're we're doing some similar sort of things oh really oh great yeah okay definitely i'll be in touch take care awesome thanks bye bye


Hello, and welcome to Managing Developer Workflows with the Kubernetes API. My name is Colin Murphy. I'm an engineering manager at Adobe. First thing I'd like to do is cover why we chose to write our own Kubernetes client. Then I'll show you how we did it. As I said, I'm an engineering manager. My current responsibility is the infrastructure for DocumentCloud, both the architecture and the software. Why write your own Kubernetes client? You certainly could just write a bunch of bash scripts with kubectl, throw that into Jenkins, and get 80% of what you could hope to achieve with your own Kubernetes client. Is it really worth all the effort for that extra 20%? For us, the overriding reason is that just like anything at a large company like Adobe, you're never in a green field. You always have lots of other things to consider. First off, all of these applications were already deployed, either on DCOS or another home spun Quintain orchestration system. So we needed to go from those configurations onto Kubernetes in a seamless manner. So the first one is really, we want to provide outages. We need to exert a greater control over what our application teams can do than just saying, here's kubectl, here's some YAML files, go have at it. You do that, you're going to have outages because people just don't understand Kubernetes. And I'll get to that in a second. But you also want to limit the Kubernetes objects you have, potentially apply labels, keep them from interfering with each other. And all this is really just to say, I should point out that it's not a substitution for RBAC, network segmentation, open policy agent, other controls on the control plane side that you'd wanted to implement to protect. So this isn't a replacement for those. It's just a nice supplementary thing. So we also, as I mentioned, it's not a green field. We had an existing deployment system called Moonbeam. It actually made the problem statement a little easier because we knew we just had a SHA. And that's all we had to do with deployment, so a Git repo, a SHA. And that made it kind of easy to start. It was a nice place to start. But we couldn't just use Spinnaker because we already had a bunch of stuff that Spinnaker can do and likewise for other tooling out there. So the other big thing, we want to reduce cognitive load. Application engineers, they have a deep technical knowledge. They understand how to manipulate a PDF or an echo sign agreement or maybe some sort of machine learning. They don't really have time to do Kubernetes. There's no such thing as a person who fully understands everything that's required in a web application. Then we had our site reliability engineers. And really, they had the same story. There's a huge legacy application. It needs constant attention. They don't have time to win this whole other system, Kubernetes. We also have some compliance and tickets. So it's just not a thing that we could just, oh, let's just re-educate hundreds of people on how to do something. Another reason is secret management. We use hashacorp vault at Adobe. And we really can't have, especially some of the clusters with very complex requirements. We're going to have FedRAMP clusters coming online. But we have PCI, HIPAA. We can't always have a connection to some sort of from a cluster to a secrets management. So we really had to find a way to package up all the secrets an application could need within our internal network and push that in a safe way out to Kubernetes cluster in a way that would allow for rollback, make sure there wasn't a way that it would interfere with secrets in the next release, that kind of thing. And then lastly, it's really just not that hard. It's very simple. It's fairly well-documented. If you know Kubernetes, if you're familiar with the concepts of Kubernetes, you can write a Kubernetes client.  You don't have to, of course, write your client in Go. But it's probably the best supported SDK that at least at the time I wrote the client. OK, before we continue, just want to point out that the client has been open sourced. It's on GitHub at Adobe slash ported to Kubernetes. So moving on to the requirements, I don't know if I've already pretty touched on them. The first one being just that we were migrating from another system or two other systems onto a single system. So we had to have the way that secrets were retrieved, have the way that environment variables set to be the same. And then also, the code had to be the same too, that the application code. So we couldn't change anything and how that was formatted. In addition to that, we needed teams to just immediately need teams to migrate right away. Or at least migrate without talking into each and every single one of them and treating them. It's a special little snowflake. So we did a lot of work around automation generating the right files for teams automatically. And that's still true now. So our onboarding system will stub out the directories required, and teams can just go from there. We don't have, as I pointed out, we don't have a team that can just meet with a new service every time there's a new service and figure out what they need, unless they need something new and different that we don't support. One really key thing and a real improvement over our previous systems was this whole idea of segmentation of responsibilities. We allow teams to do a lot within their repository, generating service instances, generating operator instances that allow them to create MySQL databases or DynamoDB or various Azure backing stores, things like that. And that's all their responsibility. And this tool allows them to do it through Kubernetes objects. We'll touch on a little bit, because that does cause some problems. But we certainly don't allow them to do whatever they want. But certainly a lot more freedom than they had before. There's no other. They don't have to learn some other system of cloud formation templates or some internal system we have that generates infrastructure code, that kind of thing. It's all completely in their control. And also, sometimes in a previous system we had, you'd have to go to some other service if you wanted to make a certain kind of change to your service. So another internal web UI, and you'd have to click through it. And it would take some time. And so that was another thing we really wanted to get rid of was anything like that. It's just there's a single deployment system, as I showed a few slides before. And you can make any change to your service using that system. And for change management, that system, we rely on that system to verify that the proper approvals have been made and reviews and things like that. There's a concept. This is a little bit of a stretch. But there's this concept of hermetic builds from the SRE manual that I think hopefully most of you have already built that way, which is no to build dependencies on the build system. So usually this way to do that is to build in Docker. But if you can stretch that a little bit to deployments, you don't want a service to roll back and have that rollback behave differently. Once you roll back a service, you don't want it to now behave differently than where it was when it was at that previous version. So we really want to tie the secrets and the environment variables of a specific get SHA for the service to that deployment in Kubernetes. So every deployment that a client deploys has the SHA labeled and we inject into it config map and secrets, a Kubernetes secret with that SHA. So if we ever roll back, even if that secret was changed somewhere in hashCorp vault, we're still getting those secrets. And we keep up to five. So if you roll back more than five previous deployments, then I guess you could potentially get a secret change. But we really just want to have that if you ever roll back, you know that you're going to a safe place. And then one last thing is we, I might have kind of inferred this, but we deploy to multiple cloud providers and some applications actually deployed to data centers. And we wanted to abstract that away from the service team. We didn't want the service team to say, oh, I need a different domain name because I'm in this, or this is HIPAA compliant, or oh, it's FedRAMP. So I have to make sure my Kubernetes configuration is a little different. So the idea was we're going to have the configuration live in each Kubernetes cluster. Actually, it's a per name space configuration. And we'll have the client as it deploys, read that information and modify the objects as required. It sounds actually kind of complicated. I'm sure we could have gone crazy with it. It's actually not that hard. For the most part, it's really just the Docker registry, whether it's used, whether the cluster is Istio or Contor, and also domain names. But it'll, I'm sure, it'll expand as we go. But it's kind of a nice idea to just have all that configuration, the common configuration for all the applications just live in the name space, saves us a lot of effort. OK, so moving on to how the ported Kubernetes works. We take the SHA, we clone the service repository, and we kind of go from there. We read in a bunch of environment variables and secret references. I'll just show you what those look like in a second. Then we read the cluster configuration from the cluster itself. And then we compare to what we have for an update. And we update the Kubernetes objects as required. So I'll just show you quickly because I couldn't get it into a slide what these configuration files look like. So it's a nested directory structure. We have a file, we try to keep it really dry. So these files will reference each other. We need at least one for every environment for every region. And as I said, they can reference each other. So if there's one that's common for all AWS, there's one that's common for all Azure, it's just all about scalability. That's the name of the game. So it looks a little intimidating. It's a lot of files, but it's fairly straightforward. So in those files, we have four different environment types. So in configuration environment entries, the first is most straightforward, an environment variable, and the value. We also have, as I mentioned many times now, secret references, so vault paths. We also have the ability to override cluster settings. So say you had a different certificate and you had needed a different domain name, TLS cert, or if you didn't want to use the same gateway for Istio gateway or contour ingress, that kind of thing as the other services in your namespace. And it's a constant fight. We're always getting requests for more features. We'll probably have to expand it a little bit, either some kind of annotations structure or something like that, because it gets very complicated with multiple ingress controllers, things like that. So I wanted to show this. This is, I guess, the first code. I was surprised how difficult it is to show code in a presentation. I just want to credit Catherine Cox Boudet. I read her book Concurrency and Go when I got started. She had this concept of pipelines that she talked about that I had never seen in another language. Maybe all of you are really familiar with it. But basically, you're just passing a channel here from in a nested kind of function structure. And it's really efficient and a clever way to, it's not, it is parallel computing, but it's not obvious, embarrassingly parallel. But you're passing through each region individually in a kind of an elegant way. But here, we're basically creating a struct for each region that's going to be deployed to, each cluster, each namespace that's being going to be deployed to. For us, it's typically along regions. And so what we're doing here is we're getting that configuration, we're fetching the secrets, we're reading in the objects, we're copying the objects, doing a deep copy because we're going to modify them. That was a lesson learned, right? You don't, you have a pointer to a struct. If you modify it for one region, it's going to modify it for all regions. So deep copy function method, actually.  The way we handle kube configs is we have just, we have a file that lists, that's common to all of our services and it just lists out all of the kube configs in their locations involved. We use AWS IAM authentication for our EKS clusters we use. Active Directory authentication, Azure Active Directory authentication for our Azure clusters. So it's not really, they're not really secrets, but it's just a, it's, it makes all the security people happy, right? It's giving warm fuzzy feeling. All of our kube configs are involved and they get pulled down at, at deploy time for every service deployment. The next thing I wanted to talk about is structured client client sets versus dynamic. So I really like the, the client sets. I really like the schemes that, you know, you find those for all the Kubernetes objects recently, I think in the last year or so, Istio put out their own client library, they call it. We use the service catalog that has its own. Unfortunately, that newer stuff doesn't, so the Azure, sorry, the Azure service operator and the AWS controller for Kubernetes, they don't provide these. So we have to actually do dynamic. I'll talk about that in a little bit. It's kind of an upcoming challenge for this. But yeah, we just use the universal deserializer, just read in YAML, it's really straightforward, really easy. And then we have a, you know, construct with all the, all of the possible Kubernetes objects that we can apply. And, you know, there's only a finite number, right? As opposed to when we get into custom operators, it really, it really expands. And we have to kind of find a new way to do things. But for right now, this is what works and this is what we do. The nice thing also about the schemes and the type client sets is it's really easy. For instance, we have to always make sure that we, you know, if somebody for some reason puts a number of replicas in their deployment, we typically discourage people from doing that because we want people to use horizontal pod autoscalers. If they haven't horizontal pod autoscaler, we always, we wanna make sure that when they deploy, they don't reset the number of replicas. So just, you know, something like this, really basic. We just, we don't have to transverse a map of strings to interface, right? We just spell out the whole, the path. Actually, so sorry, in this case, it's actually just making sure that the horizontal pod autoscaler references the deployment that is actually being deployed. We don't want somebody to make a horizontal pod autoscaler that modifies someone else's deployment. That, you know, that happens. People copy from one another as much as we tried to automatically stub things out. Just, it's just a reality when you have this many services, right? Hundreds of engineers and, you know, over a hundred services. So, and as we expand this actually out to more people in DoE, we're now going to talk about thousands and, you know, thousands of engineers and potentially thousands of services. So, yeah, just these little kind of little things, really easy to do with the type client set. Cluster settings, so these are, these are when I talked before I mentioned that there's a cluster-specific information. And it's, we've done a kind of a good job limiting it because you could really go crazy here. But these, the real, you know, the kind of the obvious thing. So, registry, we have registries in every region. You know, we don't want an outage say of AWS in you use central one, knocking out our Azure in, you know, Singapore, right? So, we always want, in our deployment strategy, we always have a regional registry. So, that has to change for deployment. And we don't want the, we don't want, that's not the application engineer's problem, right? So, in their deployment.yaml, they don't have to, they kind of just, there's just kind of whatever, you know, my registry, and then the name of their, their Docker image for the, so that just gets overwritten. It doesn't, it doesn't, you know, ported Kubernetes can handle that. Whether the cluster has Istio or not, the domain name of the services in that namespace, the Istio gateway, if there's Istio. So, a minimum number for the HPAs, we typically, you know, it fits in production. You need, you need two, some services need, you know, a lot more than that, but we always want at least some redundancy, sometimes three for some heavier use clusters. And then what, you know, what is the cloud? Is it Azure? Is it AWS? What is the cloud location? This is typically used for the service instances.  And that, that's useful for setting the location and say an Azure service instance, or an, or the Azure service operator, or the AWS equivalents. And then let's see, update. So this is, we, we, this is how the updates happen. It's, you know, it's kind of, if you're familiar with Go, this is kind of probably not that interesting. But if you're not, this, this is, you know, this is basically how we do it. We, there's an update function defined for every type of Kubernetes object because we have those, those type client sets. And basically just make a work, make a weight group. We run that function and then once they're all done, then we, then we, we exit out of this, this, this updater, we, this updater returns the, the, the, the channels with the results. And then it gets called again in a for loop for the, for the next update function. And, and just iterates through. Obviously the, the fact that every namespace can be updated simultaneously. It's a, you know, it's an, it's an embarrassingly parallel operation. So there's nothing, nothing too clever there. Just a, just a fan, fan out, fan in and check the results. Okay. So to wrap up, I just want to go over some of the things we're working on now. The first is we have the AWS controller for Kubernetes and the Azure service operator we're working on supporting those. And they come with this whole mess of CRDs. So it's going to be a common problem. If anybody wants to write their own Kubernetes client, you're going to run into this issue unless you're very specific in what you want to do. If you want to just support, you know, hundreds of engineers, you're going to, you're going to need to, you're going to run into this problem where there's just too many CRDs. And unfortunately the ACK and the ASO, they don't come with client sets or schemes. So you can either generate them yourself, which sounds like not a lot of fun or you can use the dynamic client. As I mentioned previously, the dynamic client has a few drawbacks because you don't have an actual struct that you can really pin down and grab something and compare it to something else. Right, it's a map of a string to an interface, right? But you know, we're going to try to work through that and see what we can do. I mean, right now we're just reading it into an unstructured type of a Kubernetes API, unstructured type, and then using the Kubernetes Go client to apply that. We'll see, not quite done. So I don't really want to show it in action, especially if it turns out to not work at all. And then the other thing is that we have this situation where teams don't want to deploy every object to every namespace, or they want to have a certain type of ingress that they use instead of, you know, instead of just every ingress that's available. So it's kind of how teams can fine tune things. As I mentioned before, we have the ability to override the produce Kubernetes, you know, the region, the registry, those configurations, but that's kind of a little bit too blunt of an instrument. So we're going to work on that. Probably some type of annotation or label that people put in their Kubernetes objects in their k8s directory inside their repository. Looks like that's what we'll do, but you know, still haven't decided yet. So I'd like to thank you all for coming. Thank you for bearing with me with the lighting, which is super inconsistent. I don't have any good, you know, lights. So I'm kind of using the sun and that's kind of fickle. And then also, if you want to reach out, ask me anything on the Kubernetes Slack, I'm there, I don't use Twitter. I try to maintain my sanity, but please, you know, please try it out. You know, as I said, this is all on Git. It's meant to be really a reference for other people to use, but you could, you know, you could use it. We use it here at Adobe for lots of services that are run heavily used. So please take a look. Thanks, bye.